Shootout at the PAAS Corral

EC2EC2RDS

HerokuJosh BerkusJosh Berkus

PostgreSQL Experts Inc.PostgreSQL Experts Inc.pgConf NYC 2015pgConf NYC 2015

DODO

RackspaceRackspaceGCEGCE

https://github.com/manageacloud/

cloud-benchmark-postgres

Thanks!Ruben Rubio Rey

Thanks!

What is The Cloud?

Magic?

Image by Sperlingsmaedchen. Free for non-profit use only.

a bunch of servers

with virtualization

shared storage

… and an API

The Good● Fast deployment

– new servers in minutes, with a script

● Easy scale-out– add replicas in minutes

● Minimize ops staff– no HW wranglers

¢heap(at the low end)

ex$pensive(at the high end)

The Bad● Low system resources

– VMs are small/slow

● Security– more attack vectors

The Ugly● Everything is shared

– Network– IO / Storage– CPU (partly)

Your performance depends on somebody else's peak load.

Sharingis

Not Caring

ephemeral cloud● DR is not optional● your virtual DB server will go away● you need replicas & backup

why Postgres?

transactional DB workout● works CPU● works RAM● works IO● works network IO

at the same time, in parallel

A Six-Shooter of Clouds

EC2EC2RDS HerokuDODORackspaceRackspace GCEGCE ??????

Three from AWS

The GunslingerRoll-Your-Own

The RancherRDS

The DandyHeroku

AWS commons● comprehensive API

– covering all Amazon services– do everything in code

● global distribution– multiple regions/data centers– US, Asia, South America, Europe

The Gunslinger

Roll Your Ownon EC2

Become a Gunslinger

I. Create an EC2 instance

II. Install PostgreSQL on it

III. Configure PostgreSQL

The Gunslinger● Cloud Type: platform● Administration: DIY● HA: DIY● Versions Available: anything● Extensions: all● Extra Features: none● Price: cheap

Instance Types● m3.* general-purpose● c3.* small, CPU-bound DBs $$● r3.* maximize caching $$$● i2.* Data warehousing $$$$

instance tips● Use m3.* if you don't know what to use● Get enough RAM to cache your

database if you can

Storage Types● EBS + Provisioned IOPS

– large size, latency issues– reliable, snapshots– choose “EBS optimized”

● General SSD– better for bursts (about 20% better)– high variability

Storage Types● Instance Storage + SSD option

– low-latency, limited size– risky: data loss, data corruption

PrIOPS != througput

PrIOPS fallacy● not a guarantee, a limit

– but mostly pretty consistent

● each IOP is no more than 8K● random access → each page is an IOP

– no real prefetch

● PrIOPS ~ rows/second

Stuff to Set Up● Backup: WAL-E to S3● Replication: not optional

– in another Availbility Zone

● Monitoring for instance failure● Secure your instance

– SSL– pg_hba.conf

Junior Gunslinger● “small”

“economical”● m3.medium● 1 core

3.75GB RAM● 40GB

+ 1000 PrIOPS

Senior Gunslinger● “large”

“performance”● r3.2xlarge● 8 cores

61GB RAM● 200GB

+ 4000 PrIOPS

Junior Gunslinger Costinstance: $36.50/month

EBS PrIOPS: $105/month

S3 Archive: $5/month

X2 for replica

== $288.00 a month

(+ misc charges)

Prices Subject

to Change

Cheaper Gunslingerinstance: $36.50/month

EBS PrIOPS: $105/month

S3 Archive: $5/month

no replica

== $146.50 a month

(+ misc charges)

Prices Subject

to Change

Senior Gunslingerarchive-only: $760.70/month

with replica: $1509.40/month

(+ misc charges)

Prices Subject

to Change

The RancherRelationalDatabase

Service

“I like AWS but I don't want to do all this

instance management”

DB-a-a-S● “Database as a Service”● No OS● No system config● Just Port 5432

DB-a-a-S++● SEDBA: Somebody Else Is The DBA

– uptime– backups– configuration– updates

● Let your DevOps manage the DBs

DB-a-a-S - -● Limited Options

– versions– extensions– configuration– security setups

● Cost

The Rancher● Cloud Type: database● Administration: mostly automatic● HA: replication, M.A.Z● Versions Available: 9.3, 9.4● Extensions: 2 dozen● Extra Features: none● Price: moderate

M.A.Z.● “Multi-Availability Zone”

– syncronous replication to another node– automated failover– uptime guarantee

● major performance cost

The Dandy

Heroku

“I just want to develop;the cloud should handle

all administration.”

The Dandy● Cloud Type: database● Administration: automatic● HA: auto replication + PITR● Versions Available: 9.3, 9.4, betas● Extensions: 2 dozen● Extra Features: several● Price: expensive

Dandy Bling● git-based instance management

– works really well with Rails/Django

● Dataclips– web-sharable matviews!

● Followers == replicas

Heroku options

● 5 database “sizes”● 3 levels of HA/uptime● that's it

Heroku Sizing● Small

Standard 2: 3.5 GB RAMshared hosting

● LargeStandard 6: 60GB RAMdedicated instance

The Businessman

Rackspace

“I have Rackspaceservers and want to

expand via cloud.”

The Businessman● Cloud Type: platform● Administration: DIY (+ support)● HA: DIY● Versions Available: all● Extensions: all● Extras: hybrid cloud● Price: moderate

Rackspace support● “managed infrastucture” support

required– cost added to instance costs– can bump up to higher levels of support– includes some uptime support

● No help with DBA tasks though

Rackspace storage● primarily instance storage

– could be risky

● block store not available on small instances

Rackspace Sizing● Small

general1-44 cores, 4GB RAM

● Largememory1-608 cores, 60GB RAM

The Drifter

GoogleComputeEngine

“I want an EC2-like platform

much cheaper.”

The Drifter● Cloud Type: platform● Administration: DIY● HA: DIY● Versions Available: all● Extensions: all● Extras: Docker integration● Price: cheap

GCE Sizing● Small

n1-standard-11 core, 4GB RAM

● Largen1-highmem-88 cores, 52GB RAM

The Kid

DigitalOcean

“I want cheap, simple and fast,

and nothing else.”

The Kid● Cloud Type: platform● Administration: DIY● HA: DIY● Versions Available: all● Extensions: all● Extras: nada● Price: OMG cheap!

Kid vulnerability● no block store

– all storage is ephemeral instance storage– nodes completely non-durable

● no storage cloud for backups● no HA features

D.O. Sizing● Small

4GB2 cores, 4GB RAM

● Large48GB16 cores, 48GB RAM

More Clouds

More Clouds● EDB Postgres Plus Cloud● OpenShift● Joyent + Manatee● Azure● ???

the shootout

pgbench

pgbench++● ships with Postgres● microbenchmark

– very simple “bank trade” workload

● fast to set up and run

pgbench--● doesn't do complex queries● pure random data / access● unrealistic balance of work

– too reliant on single-row write speed

● not very tunable

pgbench sizing1. memory read-write (RW):

– 50% of RAM, write transactions

2. memory read-only (RO):– 50% of RAM, read-only queries

3. disk read-write (RW):– 200% of RAM, write transactions

pgbench small●memory RW

● pgbench -i -s 100 --foreign-keys● pgbench -c 4 -T 900

●memory RO● pgbench -i -s 100 --foreign-keys● pgbench -c 4 -T 900 -S

●disk RW● pgbench -i -s 400 --foreign-keys● pgbench -c 4 -T 900

pgbench large●memory RW

● pgbench -i -s 1000 --foreign-keys● pgbench -c 16 -T 900

●memory RO● pgbench -i -s 1000 --foreign-keys● pgbench -c 16 -T 900 -S

●disk RW● pgbench -i -s 7000 --foreign-keys● pgbench -c 16 -T 900

metrics● TPS: transactions-per-second

– measures multiple things

● Load Time: time to build the database– measures INSERT time

other conditions● all Postgres 9.3

– 9.3.5 and 9.3.6

● some CentOS 7, some Ubuntu 14.04● pgbench on same instance size as

Postgres

run many many timesrun many many times

Box PlotBox Plot

495 TPS

587 TPS

1685 TPS

2537 TPS

6156 TPS

50

90

10

Min

Max

0.3X

0.4X

Median

1.7X

4X

50

90

10

Min

Max

0.3X

0.4X

Median

1.7X

4X

50

90

10

Min

Max

0.3X

0.4X

Median

1.7X

4X

50

90

10

Min

Max

when the smoke clears ...

comparability problems

I. Rackspace, DO not as durable

II. Only DBaaS HA is automated

III. Instance sizes not identical

IV. Instance OSes are not identical

work in progress

digitalocean ec2 gce heroku rackspace rds0

100

200

300

400

500

Cost Per Small Node(lower is better)

HA

Non-HA

dolla

rs/m

onth

digitalocean ec2 gce heroku rackspace rds0

500

1000

1500

2000

2500

3000

3500

Cost Per Large Node(lower is better)

HA

Non-HA

dolla

rs/m

onth

digitalocean ec2 gce heroku rackspace rds rds MAZ0

1

2

3

4

5

6

Load Time: Small In-Memory(smaller is better)

median

90.00%

Min

ute

s

digitalocean ec2 gce heroku rackspace rds rds MAZ0

1

2

3

4

5

6

Load Time: Small In-Memory(smaller is better)

median

90.00%

Min

ute

s

Instance Storage

digitalocean ec2 gce heroku rackspace rds rds MAZ0

1

2

3

4

5

6

Load Time: Small In-Memory(smaller is better)

median

90.00%

Min

ute

s

archiving overhead

digitalocean ec2 gce heroku rackspace rds rds HA0

5

10

15

20

25

30

Load Time: Small On-Disk(smaller is better)

median

90.00%

Min

ute

s

digitalocean ec2 gce heroku rackspace rds rds MAZ0

5

10

15

20

25

30

35

Load Time: Large In-Memory(smaller is better)

median

90.00%

Min

ute

s

digitalocean ec2 gce heroku rackspace rds rds MAZ0

0.2

0.4

0.6

0.8

1

1.2

$ Per LoadMBPS: Small In-Memory(smaller is better)

median 90

Dol

lars

digitalocean ec2 gce heroku rackspace rds rds MAZ0

0.2

0.4

0.6

0.8

1

1.2

$ Per LoadMBPS: Small In-Memory(smaller is better)

median 90

Dol

lars

digitalocean ec2 gce heroku rackspace rds rds MAZ0

0.5

1

1.5

2

2.5

3

3.5

4

$ Per LoadMBPS: Large In-Memory(smaller is better)

median 90

Dol

lars

digitalocean ec2 gce heroku rackspace rds rds MAZ0

100

200

300

400

500

600

700

800

900

TPS: Small In-Memory RW(larger is better)

10.00%

median

TP

S

digitalocean ec2 gce heroku rackspace rds rds MAZ0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

TPS: Small In Memory RO(larger is better)

10.00%

median

TP

S

digitalocean ec2 gce heroku rackspace rds rds HA0

100

200

300

400

500

600

700

800

TPS: Small On-Disk RW(larger is better)

10.00%

median

TP

S

digitalocean ec2 gce heroku rackspace rds rds MAZ0

500

1000

1500

2000

2500

TPS: Large In-Memory RW(larger is better)

10.00%

median

TP

S

digitalocean ec2 gce heroku rackspace rds rds MAZ0

5000

10000

15000

20000

25000

30000

35000

40000

TPS: Large In-Memory RO(larger is better)

10.00%

median

TP

S

digitalocean ec2 gce heroku rackspace rds rds MAZ0

200

400

600

800

1000

1200

1400

1600

TPS: Large On-Disk(larger is better)

10.00%

median

TP

S

digitalocean ec2 gce heroku rackspace rds rds MAZ0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

$ Per TPS: Small In Memory RW(smaller is better)

10.00%

median

Dol

lars

digitalocean ec2 gce heroku rackspace rds rds MAZ0

0.5

1

1.5

2

2.5

3

3.5

$ Per TPS: Large In-Memory RW(smaller is better)

10.00%

median

Dol

lars

Who isthis

hombre?

The Gambler

runningwith

scissorsmode

GamblerDB     bgwriter_lru_maxpages = 0

     wal_level = minimal

     fsync = off

     synchronous_commit = off

     full_page_writes = off

     wal_log_hints = off

     wal_buffers = 64MB

digitaloceanec2

gceheroku

rackspacerds

rds MAZscis-rep

scis-ulg

0

100

200

300

400

500

600

700

800

900

TPS: Small In-Memory RW(larger is better)

10.00%

median

TP

S

the trail ahead

More Benchmarks● OLTPBench?

– Wikipedia, Auctionmark, Epinions

● DVDStore?● New benchmark?

– really need something more “webby”

● NoSQLish benchmark?

Better Visualizations● better graphs● automated graph

generation● detailed response

times and time graphs

more shooting● Josh Berkus: josh@pgexperts.com

– www.pgexperts.com

● More Shootouts– www.databasesoup.com– https://github.com/manageacloud/cloud-

benchmark-postgres/– LFNW, pgCon Ottawa

Copyright 2015 PostgreSQL Experts Inc. Released under the Creative Commons Share-Alike 3.0 License. All images and trademarks are the property of their respective owners. Maginificent Seven stills are property of MGM and used hereunder principles of fair use.