Date post: | 14-Apr-2017 |
Category: |
Technology |
Upload: | amazon-web-services |
View: | 904 times |
Download: | 1 times |
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scott WardSolutions Architect
Amazon Aurora for Enterprise Database Applications
Enterprise database requirements ..
Database engine with enterprise class availability, performance, scalability, and security.
Managed services – instant provisioning, push button scaling, automated backups, patching, monitoring, migration.
1
2
Goal: Provide fully managed enterprise class database service without the cost and complexity of commercial database software.
Traditional relational databases
Gradual improvements on decades old design
Accommodate different server and storage hardware
Too complex to tune to achieve optimal performance
Layers of software to mitigate potential points of failure
‘Cloudification’ by virtue of additional layers
High cost, complex and punitive licensing terms Multiple layers of functionality in a monolithic stack.
SQL
Transactions
Caching
Logging
Relational database re:Imagined
We started with a blank sheet of paper and reimagined relational database for the cloud
Amazon Aurora is purpose built for the cloud
Designed from the ground up using AWS technology
Distributed component architecture with built-in redundancy
High-Availability and scale out are part of database core design
Self-healing components designed for resilience
Architected for security and performance
Security
Isolate your data and control access
Virtual Private Cloud (VPC)
1. Define your own private address space in the AWS cloud.
2. Create public and private subnets.
3. Define security groups for your database.
4. Connect via VPN and Direct Connect.
Encryption at rest using Keys you create an manage using KMS
Data, automated backups, snapshots, and replicas in the same cluster all automatically encrypted.
Seamless encryption and decryption, requiring no changes to your application.
SSL encryption in transit
Encryption at rest and transit
Enterprise-class performance
Write performance (console screenshot)
MySQL Sysbench
R3.8XL with 32 cores and 244 GB RAM
4 client machines with 1,000 threads each
MySQL Sysbench
R3.8XL with 32 cores and 244 GB RAM
Single client with 1,000 threads
Read performance (console screenshot)
Writes scale with table count
Tables Amazon Aurora
MySQLI2.8XL
local SSD
MySQLI2.8XL
RAM disk
RDS MySQL30K IOPS
(single AZ)
10 60,000 18,000 22,000 25,000
100 66,000 19,000 24,000 23,000
1,000 64,000 7,000 18,000 8,000
10,000 54,000 4,000 8,000 5,000
Write-only workload1,000 connectionsQuery cache (default on for Amazon Aurora, off for MySQL)
Write scales with number of connections
Connections Amazon Aurora
RDS MySQL30K IOPS
(single AZ)
50 40,000 10,000
500 71,000 21,000
5,000 110,000 13,000
OLTP WorkloadVariable connection count250 tablesQuery cache (default on for Amazon Aurora, off for MySQL)
Less IOs to backend
Effective query caching
Replica managementDo less work
Do it efficiently
Latch free lock management
Adaptive thread pools
Asynchronous commits
Consistent, low-latency writes
AZ 1 AZ 2
PrimaryInstance
StandbyInstance
Amazon Elastic Block Store (EBS)
Amazon S3
EBSmirror
EBS
EBSmirror
AZ 1 AZ 3
PrimaryInstance
Amazon S3
AZ 2
ReplicaInstance
ImprovementsConsistency—tolerance to outliers
Latency—synchronous vs. asynchronous replication
Efficiency—significantly more efficient use of network I/O
Log records
Binlog
Data
Double-write buffer
FRM files, metadata
Type of writes
MySQL with standby Amazon Aurora
async4/6 quorum
PiTR
Sequentialwrite
Sequentialwrite Distributed
writes
Designed for high-availability
Aurora storageHighly available by default
6-way replication across 3 AZs 4 of 6 write quorum
Automatic fallback to 3 of 4 if an Availability Zone (AZ) is unavailable
3 of 6 read quorum
SSD, scale-out, multi-tenant storage Seamless storage scalability Up to 64 TB database size Only pay for what you use
Log-structured storage Many small segments, each with their own redo
logs Log pages used to generate data pages Eliminates chatter between database and storage
SQL
Transactions
AZ 1 AZ 2 AZ 3
Caching
Amazon S3
Lose two copies or an AZ failure without read or write availability impact
Lose three copies without read availability impact
Automatic detection, replication, and repair
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
Read and write availability Read availability
Self-healing, fault-tolerant
Continuous backupSegment snapshot Log records
Recovery Point
Segment 1
Segment 2
Segment 3
Time
Take periodic snapshot of each segments in parallel. Stream the redo logs to S3. Backup happens continuously without performance or availability impact At restore retrieve the appropriate segment snapshots and log streams to storage nodes. Apply log streams to segment snapshots in parallel and asynchronously.
Traditional databasesHave to replay logs since the last checkpoint
Single-threaded in MySQL; requires a large number of disk accesses
Amazon AuroraUnderlying storage replays redo records on demand as part of a disk read
Parallel, distributed, asynchronous
Checkpointed Data Redo Log
Crash at T0 requiresa re-application of theSQL in the redo log sincelast checkpoint
T0 T0
Crash at T0 will result in redologs being applied to each segmenton demand, in parallel, asynchronously
Instant crash recovery
Faster, more predictable failover
AppRunningFailure Detection DNS Propagation
Recovery Recovery
DBFailure
MYSQL
AppRunning
Failure Detection DNS Propagation
Recovery
DBFailure
AURORA WITH MARIADB DRIVER
1 5 - 2 0 s e c
3 - 2 0 s e c
To cause the failure of a component at the database node:ALTER SYSTEM CRASH [{INSTANCE | DISPATCHER | NODE}]
To simulate the failure of disks:ALTER SYSTEM SIMULATE percent_failure DISK failure_type IN [DISK index | NODE index] FOR INTERVAL interval
To simulate the failure of networking:ALTER SYSTEM SIMULATE percent_failure NETWORK failure_type [TO {ALL | read_replica | availability_zone}] FOR INTERVAL interval
To simulate the failure of an Aurora Replica:ALTER SYSTEM SIMULATE percentage_of_failure PERCENTREAD REPLICA FAILURE [TO ALL | TO "replica name"] FOR INTERVAL interval
Simulate failures using SQL
Delivered as a managed service
Databases are hard to manage
RDS platform: managing databases made easy
Schema design Query construction Query optimization
Backup & recoveryIsolation & security Industry compliancePush-button scalingAutomated patchingAdvanced monitoringRoutine maintenance
Amazon RDS takes care of your time-consuming database management tasks, freeing you to focus on your applications and business
You
RDS
Advanced monitoringSingle page dashboard for OS and process diagnostics in AWS console
Customize dashboard with choice of metrics and layout
Add alarms on specific metrics
Metrics egress via Cloud Watch Logs into 3rd party monitoring tools like Graphite etc.
Support for metrics crossover into CloudWatch
Metrics such as load average, detailed CPU utilization, detailed disk IO, and per process provides at a fixed range of granularities ranging from 60 to 1 seconds.
Integrations with tools and dashboards
Migrating to Aurora
MySQL Backup to Aurora
Source MySQL Database Target Aurora
Database
Amazon S3
Move data to the same or different database engine
Keep your apps running during the migration
Start your first migration in 10 minutes or less
Replicate within, to or from AWS EC2 or RDS
AWS Database Migration Service
http://aws.amazon.com/dms
CustomerPremises
Application Users
AWS
InternetVPN
Start a replication instance
Connect to source and target databases
Select tables, schemas or databases
Let the AWS Database Migration Service create tables, load data and keep them in sync
Switch applications over to the target at your convenience
Keep your apps running during the migration
Migrate off Oracle and SQL Server
Move your tables, views, stored procedures and DML to MySQL, MariaDB & Amazon Aurora
Highlight where manual edits are neededAWS Schema Conversion Tool
http://aws.amazon.com/sct
Perfect fit for enterprise 6-way replication across 3
Fail-over in less than 30 secs
Near instant crash recovery
Up to 500K/sec read and 100K/sec write
15 low latency (10ms) Read Replicas
Up to 64 TB DB optimized storage volume
Instant provisioning and deployment
Automated patching and software upgrade
Backup and point-in-time recovery
Compute and storage scaling
Performance and scale
Enterprise class availability
Fully managed service
Many features are unique to Amazon Aurora
Comparing to traditional commercial databases like Oracle• Available only in most expensive database edition (Enterprise Edition)• Failover and Replica – Oracle Active Data guard – Extra $$$ per core• Backup to S3 - Oracle Secure Backup Cloud Module – Extra $$$ per channel• Encryption – Oracle Advanced Security - Extra $$$ per core
Comparing features ..
Don’t be constrained by Licenses, Cost or Capacity
Simple pricingNo licensesNo lock-inPay only for what you use
Discounts44% with a 1-year RI63% with a 3-year RI
vCPU Mem Hourly Price
db.r3.large 2 15.25 $0.29
db.r3.xlarge 4 30.5 $0.58
db.r3.2xlarge 8 61 $1.16
db.r3.4xlarge 16 122 $2.32
db.r3.8xlarge 32 244 $4.64
• Storage consumed, up to 64 TB, is $0.10/GB-month• IOs consumed are billed at $0.20 per million I/O• Prices are for Virginia
Enterprise grade, open source pricing
Cost of Ownership: Aurora vs. Commercial databasesOracle on EC2 Configuration Hourly Cost
Primaryr3.8XL
Standbyr3.8XL
Replicar3.8XL
ReplicaR3.8XL
Storage6TB / 30K PIOP
Storage6TB / 30K PIOP
Storage6TB / 30K PIOP
Storage6TB / 30K PIOP
$2.93/hr
$2.93/hr
$2.93/hr $2.93/hr
$3.75/hr
$3.75/hr $3.75/hr
Instance cost: $11.72/ hrLicense cost: $63.12/hrStorage cost: $15.00 / hr
Total cost: $89.84 / hr
$3.75/hr
Enterprise License
$15.78/hr
Enterprise License
$15.78/hr
Enterprise License
$15.78/hr
Enterprise License
$15.78/hr
Cost of Ownership: Aurora vs. Oracle Aurora Configuration Hourly Cost
Instance cost: $13.92/ hrStorage cost: $5.15 / hr
Total cost: $19.07 / hr
Primaryr3.8XL
Replicar3.8XL
ReplicaR3.8XL
Storage / 6TB
$4.64/ hr $4.64/ hr $4.64/ hr
$5.15 / hr
Storage IOPs assumptions:1. Average IOPs is 50% of Max IOPs2. 50% savings from shipping logs vs. full pages
78.7%Savings
No idle standby instance
Single shared storage volume
No POIPs – pay for use IO
Reduction in overall IOP
Enterprise use cases
Web and mobile
Content management
E-commerce, retail
Internet of Things
Search, advertising
BI and analytics
Games, media
Common Customer Use Cases
Fastest growing service in AWS history1000+ customer after 10 days of launch
Zynga: Online Gaming
Online and Mobile Gaming Company
Challenges running traditional relational database technologies at scale
Instance recovery time Monitor mundane but critical jobs
Latency associated with existing DB storage solution
Aurora benefits: ~9K selects/second during peak for 150GB data
set Reduced operational overhead Simplified scaling for large scale events
Expedia: On-line travel marketplace Real-time business intelligence and analytics on
a growing corpus of on-line travel market place data.
Current SQL server based architecture is too expensive. Performance degrades as data volume grows.
Cassandra with Solr index requires large memory footprint and hundreds of nodes, adding cost.
Aurora benefits: Aurora meets scale and performance
requirements with much lower cost. 25,000 inserts/sec with peak up to 70,000. 30ms
average response time for write and 17ms for read, with 1 month of data.
World’s leading online travel company, with a portfolio that includes 150+ travel sites in
70 countries.
Alfresco: Enterprise Content Management
Needed database to scale without any degradation in performance.
Benefits: Alfresco on Amazon Aurora, scaled to 1 billion
documents with a throughput of 3 million per hour, which is 10 times faster than their MySQL environment.
Provides Enterprise Content Management software built on
open standards.
Alfresco - Benchmark Results
Document load rate 1000 documents per second (with 10 nodes) Load rate was consistent even passing the 1B document Sub-second login times and good responses for other actions
Open Library: 4.5s Page Results: 1s Navigate to Site: 2.3
Aurora indexes used efficiently at 3.2TB No indications of any size-related bottlenecks with 1.1 Billion Documents CPU loads:
Database: 8-10% Alfresco (each of 10 nodes): 25-30%
Insurance claims processing ICSC Provides fully integrated policy management,
claim and billing solutions for property/casualty insurance organizations
For the last 12 years ISCS has used SQL Server & Oracle commercial databases for operational & warehouse data
The cost and maintenance of traditional commercial database has increasingly become the biggest expenditure and maintenance headache
Maintaining its customer SLAs requires complex, difficult-to-manage replication and redundancy across multiple geographic locations
As customer data grows, backup/restore times for its largest data sets have progressed to unacceptable levels.
Aurora benefits
SQL Server backups that once took 5-6 hours daily now happen continuously on Aurora. Snapshots from one customer database (~ 5TB in size) take 5 minutes to make and less than an hour to restore. ISCS can actually test disaster recovery daily if it wanted to.
Data that was once only available “daily, batch” into Redshift can now be migrated continuously using Aurora read-replicas and Change Data Capture (CDC).
Performance at scale is linear since ISCS’s application, like Aurora, is optimized for multiple, concurrent read requests to the database.
Multi-AZ Aurora read-replicas also eliminate the need for additional licenses/deployments of SQL Server.
The cost of a “more capable” deployment on Aurora has proven to be about 70% less than ISCS’s SQL Server deployments.
Amazon Aurora: Earth Networks
Earth Networks process over 25 terabytes of real-time data daily, so need a scalable database that can rapidly grow with expanding data analysis
Benefits: Aurora performance and scalability works well
with their rapid data growth. Moving from SQL Server to Aurora was very easy
Operates world's largest weather and lightning sensor
networks and technology
Want to learn more?
Amazon Aurora home page:https://aws.amazon.com/rds/aurora/start/
• Product Reviews• Whitepapers• Hands on Labs• Customer References
Registration Open Now
Learn more reinvent.awsevents.com
November 28-December 2, 2016 | The Venetian - Las Vegas, NV
Thank you!
Scott WardSolutions Architect