Amazon RDS PostgreSQL: Enabling Innovation with Cloud Managed Databases

transcript

Grant McAlister – Senior Principal Engineer - RDS

March 2017

Amazon RDS for PostgreSQLEnabling Innovation with Cloud Managed Databases

Development to Production

Big Test

DB1 DB2 DB3

DB4 DB5 DB6

Production

DB1DB2

DB1 DB2

Patching

HA & DRBackup & Recovery

ScaleSecurity

Amazon Relational Database Service

RDS is a managed Relational database service that is simple to deploy, easy to scale, reliable and cost-effective

Managed Service

Easy to Scale and Operate

Choice of Database Engines

High Availability

High Performance

Amazon Relational Database Service (RDS)

RDS Version Updates

Now Supporting 9.6

Minor Releases• 9.6.1• 9.5.4 • 9.4.9 • 9.3.14

Extension Support Additions

9.6 bloom & pg_visibility

rds-postgres-extensions-request@amazon.com

9.3 Original - 32

9.3 Current - 35

9.4 Current - 399.5 Current - 44

Future - ???9.6 Current - 46

Availability

Backups and Disaster RecoveryDB Snapshots

• User-driven snapshots of database• Kept until explicitly deleted

Automated Backups• Nightly system snapshots + transaction backup• Enables point-in-time restore to any point in retention period, up to

the last 5 minutes• Max retention period = 35 days

Cross region copy• Change regions• Disaster recovery

Availability – Read and Write – Multi-AZ

Secondary

Physical Synchronous

Replication

Primary

AZ1 AZ2

Application Application

SecondarySecondary Primary

Replication

AZ1 AZ2

SecondarySecondary Primary

Replication

AZ1 AZ2

cname update

Primary Update

Secondary Primary

Replication

AZ1 AZ2

cname update

Read Replicas = Availability

Secondary

Application

Read Replica

SyncReplication

Multi-AZ

Primary

Async Replication

Secondary

Application

Read Replica

SyncReplication

Multi-AZ

Primary

Eventually Consistent Reads

Writes & Consistent

Async Replication

Application

Read Replica

Read ReplicaSecondary Primary

Writes & Consistent

Async Replication

Application

Read Replica

Writes & Consistent

Async Replication

Application

Read Replica

Writes & Consistent

Async Replication

Upgrade

Application

Read Replica

Writes & Consistent

Async Replication

Modify

Cross Region Replicas – DR & Moves

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

AZ1Async Replication

US-EAST-1 EU-WEST-1

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

US-EAST-1 EU-WEST-1

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

New Primary

Async Replication

US-EAST-1 EU-WEST-1

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

Application

New Primary

Async Replication

US-EAST-1 EU-WEST-1

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

Application

New Primary

Async Replication

US-EAST-1 EU-WEST-1

Application

SecondarySync

Read Replica

Application

New Primary

US-EAST-1 EU-WEST-1

Application

SecondarySync

Cross Region Replicas – Reduce Latency

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

US-EAST-1

Cross Region Replicas – Reduce Latency

SecondarySync

Application

Read ReplicaPrimary

AZ1 AZ2

Application

Read Replica

Application

US-EAST-1 EU-WEST-1

AuroraR/W

Aurora R

Application ROApplication

SyncReplication

AZ-1 AZ-2 AZ-3

Storage Storage Storage Storage Storage Storage

Aurora Storage

4/6 sync writes

Aurora PostgreSQL – In Preview

AuroraR/W

Aurora R

SyncReplication

AZ-1 AZ-2 AZ-3

Aurora Storage

4/6 sync writes

Async Invalidation

AuroraR/W

Aurora R

SyncReplication

AZ-1 AZ-2 AZ-3

Aurora Storage

4/6 sync writes

AuroraR/W

Aurora R

Async Invalidation

FAILOVER

Amazon Aurora with PostgreSQL Compatibility

Performance By The NumbersMeasurement Result

PgBench >= 2x faster

SysBench 2x-3x faster

Data Loading 3x faster

Response Time >2x faster

Throughput Jitter >3x more consistent

Throughput at Scale 3x faster

Recovery Speed Up to 85x faster

Patching

Minor Version Patching

• Customer Control Patching• Execute immediately or in Weekly Maintenance Window

• Auto Patching can be enabled • Allows for hands off upgrade in next Maintenance Window

Major version upgrade

Prod9.5

Prod9.6

pg_upgrade

Backup Backup

No PITR

Prod9.5

Test9.5

Restore to a test instance

Prod9.5

Test9.5

pg_upgrade

Prod9.5

Test9.5

Test9.6

pg_upgrade

Prod9.5

Test9.5

Test9.6

pg_upgrade

Application Testing

Prod9.5

Test9.5

Test9.6

pg_upgrade

Application Testing

Prod9.5

Prod9.6

pg_upgrade

Backup Backup

No PITR

Test9.5

Test9.6

pg_upgrade

Application Testing

Security

Forcing SSL on all connections

DB Instance

SnapshotApplication

Log Backups

DB Instance

SnapshotApplication

Log Backups

Security Group

DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

Encryption at Rest

DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

Encryption at Rest

ssl_mode=disable

DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

Encryption at Rest

ssl_mode=disable

DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

Encryption at Rest

ssl_mode=disable

rds.force_ssl=1 (default 0)

DB Instance

SnapshotApplication

HostSSL

Log Backups

Security Group

Encryption at Rest

ssl_mode=disable

rds.force_ssl=1 (default 0)

Unencrypted Snapshot Sharing

DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

DB Instance

Snapshot

Prod Account

Test Account

SnapshotDB Instance

Snapshot

Share with account

DB Instance

Snapshot

Prod Account

Test Account

SnapshotDB Instance

Snapshot

Share with account

Share to Public

Encrypted Snapshot Sharing

DB Instance

Snapshot

Prod Account

Test Account

Encryption at Rest

Default

DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

Default

DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

Default

DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

CustomKey

DB Instance

Snapshot

Prod Account

Test Account

Snapshot

Share with account

Encryption at Rest

CustomKey

Add external account

DB Instance

Snapshot

Prod Account

Test Account

SnapshotDB Instance

Snapshot

Share with account

Encryption at Rest

CustomKey

Add external account

Cross Region Replicas – Encrypted

SecondarySync

Application

Primary

AZ1 AZ2

Application

US-EAST-1

Cross Region Replicas – Encrypted

SecondarySync

Application

Primary

AZ1 AZ2

Application

Read Replica

Application

US-EAST-1 EU-WEST-1

2 Threads 4 Threads 8 Threads 16 Threads 32 Threads 64 Threads0

4,500PG Bench - Read & Write

RegularEncrypted

)Encryption at rest overhead

5 to 10% Overhead on heavy write

HIPAA-eligible service & FedRAMP

• RDS PostgreSQL is now a HIPAA-eligible service• https://aws.amazon.com/compliance/hipaa-compliance/

• FedRAMP in AWS GovCloud (US) region • https://aws.amazon.com/compliance/fedramp/

Data movement

Move data to the same or different database engine Keep your apps running during the migrationStart your first migration in 10 minutes or lessReplicate within, to, or from AWS EC2 or RDS

AWSDatabase Migration

Service(DMS)

CustomerPremises

Application Users

Internet

Start a replication instance

Keep your apps running during the migration

AWS Database Migration Service

CustomerPremises

Application Users

Internet

Connect to source and target databases

Select tables, schemas, or databases

CustomerPremises

Application Users

Internet

Let the AWS Database Migration Service create tables and load data

CustomerPremises

Application Users

Internet

Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync

CustomerPremises

Application Users

Internet

Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in syncSwitch applications over to the target at your convenience

AWS Database Migration Service - PostgreSQL

• Source - on premise or EC2 PostgreSQL (9.4+) RDS (9.4.9+ or 9.5.4+ or 9.6.1)

• Destination can be EC2 or RDS• Initial bulk copy via consistent select• Uses PostgreSQL logical replication support to provide

change data capture

https://aws.amazon.com/dms/

Logical Replication Support• Supported with 9.6.1+, 9.5.4+ and 9.4.9+• Set rds.logical_replication parameter to 1• As user who has rds_replication & rds_superuser role

SELECT * FROM pg_create_logical_replication_slot('test_slot', 'test_decoding');

pg_recvlogical -d postgres --slot test_slot -U master --host $rds_hostname -f - --start

• Added support for Event Triggers

Logical Decoding Space Usage

CloudWatch – Replication Lag

CloudWatch – Slot usage for WAL

Logical Replication Support - Example

RDS Postgres

Logical Replica

Redshift

RDS Postgres

Logical Replica

Redshift

On PremisePostgres

RDS Postgres

Logical Replica

RedshiftEC2 Postgres

On PremisePostgres

RDS Postgres

Logical Replica

On PremisePostgres

EC2 Oracle

RDS Postgres

Logical Replica

On PremisePostgres

EC2 Oracle

CustomLogicalHandler

NoSQL DB

Schema Conversion Tool - SCT

Downloadable tool (Windows, Mac, Linux Desktop)

Source Database Target Database on Amazon RDSMicrosoft SQL Server Amazon Aurora, MySQL, PostgreSQLMySQL PostgreSQLOracle Amazon Aurora, MySQL, PostgreSQLPostgreSQL Amazon Aurora, MySQL

SCT - Analysis

SCT - Detailed

Scale & Performance

Push Button Scaling

Scale nodes vertically up or down• db.t2.micro (1 virtual cores, 1GB)• db.m4.large (2 virtual cores, 8GB) • db.r3.8xlarge(32 virtual cores, 244GB)

• db.r4.16xlarge (64 virtual cores, 488GB) Aurora

Storage Scaling EBS Based - 5GB-6TB & Up to 30,000 IOPSAurora Based – Up to 64 TB

M4 Instance Class – pgbench read only

1 2 4 8 160

db.m3.large db.m4.large

Threads

46% Better Price/Performance

37% TPS Increase

$0.195 $0.182

Enhanced Operating System (OS) metrics

1-60 second granularity

cpuUtilization• guest• irq• system• wait• idl: • user • total • steal • nice

diskIO • writeKbPS• readIOsPS• await • readKbPS• rrqmPS • util • avgQueueLen • tps • readKb • writeKb • avgReqSz • wrqmPS • writeIOsPS

memory • writeback• cached • free • inactive• dirty • mapped • active • total • slab • buffers• pageTable• Hugepages

swap • cached • total • free

tasks• sleeping • zombie • running • stopped • total • blocked

fileSys • used• usedFiles• usedFilePercent• maxFiles • total • usedPercent

loadAverageMinute • fifteen • five • one

uptime

processList• name• cpuTime• parentID• memoryUsedPct• cpuUsedPct• id • rss• vss

Process List

OS metrics

Performance Insights – In Preview

Stats on RAMDISK

• Set rds.pg_stat_ramdisk_size in MB’s• Creates a RAM disk and sets stats_temp_directory to

use it.

• Reduces IOPS

• Good for instances with many tables/indexes and databases.

Burst mode: GP2 and T2

T2 – Amazon EC2 instance with burst capability• Base performance + burst • Earn credits per hour when below base performance• Can store up to 24 hours worth of credits• Amazon CloudWatch metrics to see credits and usage

GP2 – SSD-based Amazon EBS storage• 3 IOPS per GB base performance• Earn credits when usage below base• Burst to 3000+ IOPS

T2 – CPU credits

CPU credit balance

T2 exhaust CPU credits

Burst mode: what’s new

db.t2.large• 60 CPU Initial Credit• 36 CPU Credit earned per hour• Base Performance – 60%• 8 GB RAM• Increased IO bandwidth• Encryption at rest support

Future – t2.xlarge & t2.2xlarge

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

12000100% Read - 20GB data

db.m1.medium + 200GB standarddb.m3.medium + 200G + 2000 IOPSdb.m3.large + 200G + 2000 IOPSdb.t2.medium + 200GB gp2

)Burst mode vs. Classic vs. Provisioned IOPS

$0.10 per hour

$0.58 per hour$0.40 per hour

$0.50 per hour

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

12000100% Read - 20GB data

db.m1.medium + 200GB standarddb.m3.medium + 200G + 2000 IOPSdb.m3.large + 200G + 2000 IOPSdb.t2.medium + 200GB gp2db.t2.medium + 1TB gp2

$0.10 per hour

$0.58 per hour

$0.23 per hour

$0.40 per hour

$0.50 per hour

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

12000100% Read - 20GB data

db.m1.medium + 200GB standarddb.m3.medium + 200G + 2000 IOPSdb.m3.large + 200G + 2000 IOPSdb.t2.medium + 200GB gp2db.t2.medium + 1TB gp2db.t2.large + 1TB gp2

$0.10 per hour

$0.58 per hour

$0.23 per hour

$0.40 per hour

$0.50 per hour

$0.30 per hour

Lessons

Vacuum parameters

Will auto vacuum when• autovacuum_vacuum_threshold +

autovacuum_vacuum_scale_factor * pgclass.reltuples

How hard auto vacuum works • autovacuum_max_workers• autovacuum_nap_time• autovacuum_cost_limit• autovacuum_cost_delay

Transaction IDWrap Around

RDS autovacuum logging (9.4.5+)

log_autovacuum_min_duration = 5000 (i.e. 5 secs)rds.force_autovacuum_logging_level = LOG

…[14638]:ERROR: canceling autovacuum task…[14638]:CONTEXT: automatic vacuum of table "postgres.public.pgbench_tellers"…[14638]:LOG: skipping vacuum of "pgbench_branches" --- lock not available

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Autovacuum

RDS autovacuum visibility(9.3.12, 9.4.7, 9.5.2, 9.6.1)pg_stat_activity

BEFORE usename | query----------+------------------------------------------------------------- rdsadmin | <insufficient privilege> rdsadmin | <insufficient privilege> gtest | SELECT c FROM sbtest27 WHERE id BETWEEN 392582 AND 392582+4 gtest | select usename, query from pg_stat_activity

CloudWatch Metric

AWS Database BLOG

https://aws.amazon.com/blogs/database/

Thank you!

Questions?

Amazon RDS PostgreSQL: Enabling Innovation with Cloud Managed Databases

Data & Analytics