Post on 21-Jan-2018
transcript
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sean Shriver, AWS DynamoDB Solutions Architect
November 2017
Deep Dive: Amazon DynamoDB
Dating Website Serverless IoT
o DAX
o GSIs
o TTL
o Streams
o DAX
Getting Started
o Developer Resources
Amazon DynamoDB
o Foundations
o Tables
o Indexes
o Partitioning
New Features
o TTL
o Tagging
o VPC Endpoints
o Auto Scaling
o DAX
Plan
Dynamo whitepaper
NoSQL foundations
0000 {“Texas”}
0001 {“Illinois”}
0002 {“Oregon”}
T
XW
A
I
L
Key
Column
0000-0000-0000-0001
Game Heroes
Version 3.4
CRC ADE4
Key Value Graph Document Column-family
Dynamo:Amazon’s Highly Available
Key-value
Store
January 2012Fall 2007 June 2009
Meetup235 2nd St
San Francisco
Scaling relational vs. non-relational databases
Traditional SQL NoSQL
DB
DB
Scale up
DB
Host
1
DB
Host
n
DB
Host
2
DB
Host
3
Scale out to many shards
(DynamoDB: partitions)
Scaling NoSQL
- Good sharding (partitioning) scheme affords even
distribution of both data and workload, as they grow
- Key concept: partition key
- Ideal scaling conditions:
1. Partition key is from a high cardinality set (that grows)
2. Requests are evenly spread over the key space
3. Requests are evenly spread over time
What (some) customers store in NoSQL DBs
Market Orders Tokenization
(PHI, Credit Cards)
Chat MessagesUser Profiles
(Mobile)
IoT Sensor Data
(& device status!)
File MetadataSocial Media Feeds
Use case: DataXu’s attribution engine
Meta
Amazon
EMR
JobAmazon
Cloud
Watch
DynamoDBAWS Data
Pipeline
3rd
Party
S3
Buckets
1st
Party
AWS Direct
Connect
Amazon
VPC
Amazon
EC2
Amazon
RDS
Amazon SNS
AWS IAM
“Attribution" is the marketing term for the allocation of credit to individual
advertisements that eventually lead to a desired outcome (e.g., purchase).
Use case: DataXu’s attribution engine
Meta
Amazon
EMR
JobAmazon
Cloud
Watch
DynamoDBAWS Data
Pipeline
3rd
Party
S3
Buckets
1st
Party
AWS Direct
Connect
Amazon
VPC
Amazon
EC2
Amazon
RDS
Amazon SNS
AWS IAM
“Attribution" is the marketing term for the allocation of credit to individual
advertisements that eventually lead to a desired outcame (e.g. purchase).
Technical challenges
Amazon EC2
InstancesEBS Volumes
CloudWatch
MetricsNotifications
Scaling new AZs,
new Regions
Highly available Consistent, single digit
millisecond latency
at any scale
Fully managed
Secure
Integrates with AWS Lambda,
Amazon Redshift, and more.
Amazon DynamoDB
Elastic is the new normal
Write Capacity Units
Read Capacity Units
Consum
ed C
apacity U
nits
>200% increase from baseline>300% increase from baseline
Time
Scaling high-velocity use cases with DynamoDB
Ad Tech Gaming MobileIoT Web
Partition Key
Mandatory
Key-value access pattern
Determines data distribution
Optional
Model 1:N relationships
Enables rich query capabilities
DynamoDB Table
A1
(partition key)
A2
(sort key)
A3 A4 A7
A1
(partition key)
A2
(sort key)
A6 A4 A5
A1
(partition key)
A2
(sort key)
A1
(partition key)
A2
(sort key)
A3 A4 A5
SortKey
Table
Items
10 GB max per
partition key, i.e.
LSIs limit the # of
sort keys!
A1
(partition key)
A3
(sort key)
A2 A4 A5
A1
(partition key)
A4
(sort key)
A2 A3 A5
A1
(partition key)
A5
(sort key)
A2 A3 A4
• Alternate sort key
attribute
• Index is local to a
partition key
Local Secondary Indexes
RCUs/WCUs
provisioned
separately for GSIs
INCLUDE A2
A
L
L
KEYS_ONLY
A3
(partition key)
A1
(table key)
A2 A4 A7
A3
(partition key)
A1
(table key)
A3
(partition key)
A1
(table key)
A2
• Alternate partition
(+sort) key
• Index is across all
table partition keys
• Can be added or
removed anytime
A3
(partition key)
A1
(table key)
A2 A4 A7
A3
(partition key)
A1
(table key)
A2
A3
(partition key)
A1
(table key)
Global Secondary Indexes
Data types
Type DynamoDB Type
String String
Integer, Float Number
Timestamp Number or String
Blob Binary
Boolean Bool
Null Null
List List
SetSet of String, Number,
or Binary
Map Map
Table creation options
PartitionKey, Type:
SortKey, Type:
Provisioned Reads:
Provisioned Writes:
LSI Schema GSI Schema
AttributeName [S,N,B]
AttributeName [S,N,B]
1+
1+
Provisioned Reads: 1+
Provisioned Writes: 1+
TableName
Op
tion
al
Re
qu
ired
CreateTable
String,
Number,
Binary ONLY
Per Second
Unique to
Account and
Region
Optionally:
TTL and Auto Scaling
Provisioned capacity
Provisioned capacity
Read Capacity Unit (RCU)1 RCU returns 4KB of data for strongly
consistent reads, or double the data
at the same cost for eventually
consistent reads
Capacity is per second, rounded up to the next
whole number
Write Capacity Unit (WCU)1 WCU writes 1KB of data, and each
item consumes 1 WCU minimum
Horizontal Sharding
Host 1 Host 99 Host n
~Each new host brings compute, storage and network bandwidth~
CustomerOrdersTable
Cu
sto
merO
rders
Tab
le
OrderId: 1
CustomerId: 1
ASIN: [B00X4WHP5E]
Partitioning
00
55
AA
FF
Hash(1) = 7B
CustomerOrdersTable
OrderId: 2
CustomerId: 4
ASIN: [B00OQVZDJM]
OrderId: 3
CustomerId: 3
ASIN: [B00U3FPN4U]
Partition A
33.33 % Keyspace
33.33 % Provisioned Capacity
Partition B
33.33 % Keyspace
33.33 % Provisioned Capacity
Partition C
33.33 % Keyspace
33.33 % Provisioned Capacity
Hash.MIN = 0
Hash.MAX = FF
Ke
ysp
ace
Hash(2) = 48
Hash(3) = CD
Cu
sto
merO
rders
Tab
le
00
55
AA
FF
Partition A
33.33 % Keyspace
33.33 % Provisioned Capacity
Partition B
33.33 % Keyspace
33.33 % Provisioned Capacity
Partition C
33.33 % Keyspace
33.33 % Provisioned Capacity
Hash.MIN = 0
Hash.MAX = FF
Ke
ysp
ace
Time
Partition A
33.33 % Keyspace
33.33 % Provisioned Capacity
Partition B
33.33 % Keyspace
33.33 % Provisioned Capacity
Partition D
Partition E
16.66 %
16.66 %
16.66 %
16.66 %
Partition split due to partition size00
55
AA
FF
Partition A
33.33 % Keyspace
33.33 % Provisioned Capacity
Partition B
33.33 % Keyspace
33.33 % Provisioned Capacity
Partition C
33.33 % Keyspace
33.33 % Provisioned Capacity
Time
Partition A
Partition C
16.66 %
16.66 %
16.66 %
16.66 %
Partition splits due to capacity increase
16.66 %
16.66 %
16.66 %
16.66 %
16.66 %
16.66 %
16.66 %
16.66 %
Partition B
Partition D
Partition E
Partition F
The desired size of a partition
is 10GB* and when a
partition surpasses this it can
split*=subject to change
Split for partition size
The desired capacity of a
partition is expressed as:
3w + 1r < 3000 *
Where w = WCU & r = RCU*=subject to change
Split for provisioned capacity
Partitioning
Partition A
1000 RCUs
100 WCUs
Partition C
1000 RCUs
100 WCUs
Host A Host C
Availability Zone A
Partition A
1000 RCUs
100 WCUs
Partition C
1000 RCUs
100 WCUs
Host E Host G
Availability Zone B
Partition A
1000 RCUs
100 WCUs
Partition C
1000 RCUs
100 WCUs
Host H Host J
Availability Zone C
CustomerOrdersTable
54:∞00:0 54:∞00:0 54:∞00:0FF:∞AA:0 FF:∞AA:0 FF:∞AA:0
Data is replicated to
three Availability Zones
by design
3-way replication
OrderId: 1
CustomerId: 1
ASIN: [B00X4WHP5E]
Hash(1) = 7B
Partition B
1000 RCUs
100 WCUs
Host B Host F Host I
Partition B
1000 RCUs
100 WCUs
Partition B
1000 RCUs
100 WCUs
A9:∞55:0 A9:∞55:0 A9:∞55:0
Partitioning
DynamoDB Streams
Partition A
Partition B
Partition C
Ordered stream of item
changes
Exactly once, strictly
ordered by key
Highly durable, scalable
24 hour retention
Sub-second latency
Compatible with Kinesis
Client Library
DynamoDB Streams
1
Shards have a lineage and
automatically close after time or
when the associated DynamoDB
partition splits
2
3Updates
KCL
Worker
Amazon
Kinesis Client
Library
Application
KCL
Worker
KCL
Worker
GetRecords
Amazon DynamoDB
TableDynamoDB Streams
Stream
Shards
TTL
job
Time-To-Live (TTL)
Amazon DynamoDB
Table
CustomerActiveOrder
OrderId: 1
CustomerId: 1
MyTTL:
1492641900
DynamoDB Streams
Stream
Amazon Kinesis
Amazon Redshift
An epoch timestamp marking when an
item can be deleted by a background
process, without consuming any
provisioned capacity
Time-To-Live
Removes data that is no longer relevant
Time-To-Live (TTL)
TTL items
identifiable in
DynamoDB Streams
Configuration protected by AWS
Identity and Access Management
(IAM), auditable with AWS
CloudTrail
Eventual deletion,
free to use
Cost allocation tagging
• Track costs: AWS bills broken down by tags
in detailed monthly bills and Cost Explorer
• Flexible: Add customizable tags to tables,
indexes and DAX clusters
Features
Key Benefits
• Transparency: know exactly how much
your DynamoDB resources cost
• Consistent: report of spend across AWS
services
DynamoDB Auto Scaling
Specify: 1) Target capacity in percent 2) Upper and lower bound
DynamoDB Auto Scaling
Amazon
DynamoDB
msμs
DAXYour App
in VPC
Amazon DynamoDB Accelerator (DAX)
DynamoDB in the VPC
Availability Zone #1 Availability Zone #2
Private Subnet Private Subnet
VPC endpoint
web
app
server
security
groupsecurity
group
oMicroseconds latency in-memory cache
oMillions of requests per second
oFully managed, highly available
oRole based access control
oNo IGW or VPC endpoint required
DAX
oDynamoDB-in-the-VPC
o IAM resource policy
restricted
VPC Endpoints
AWS Lambda
security
groupsecurity
group
DAX
web
app
server
DAX
DynamoDB Accelerator (DAX)
Private IP, Client-side
Discovery
Supports AWS Java and NodeJS SDK,
with more AWS SDKs to come
Cluster based, Multi-AZ Separate Query and
Item cache
Elements of even access in NoSQL
1) Time
2) SPACE
DynamoDB key choice
To get the most out of DynamoDB throughput, create tables where the
partition key has a large number of distinct values, and values are requested
fairly uniformly, as randomly as possible.
Amazon DynamoDB Developer Guide
Burst capacity is built-in
0
400
800
1200
1600
Ca
pa
cit
y U
nit
s
Time
Provisioned Consumed
“Save up” unused capacity
Consume saved up capacity
Burst: 300 seconds
(1200 × 300 = 360k CU)
DynamoDB “saves” 300
seconds of unused capacity
per partition
Burst capacity may not be sufficient
0
400
800
1200
1600
Cap
ac
ity U
nit
s
Time
Provisioned Consumed Attempted
Throttled requests
Don’t completely depend on burst capacity… provision sufficient throughput
Burst: 300 seconds
(1200 × 300 = 360k CU)
Hot shards
Host 1 Host 2 Host 3
Hot key!
AZ A AZ B AZ C
Throttling
Occurs if sustained throughput goes beyond provisioned throughput per partition
• Possible causes
• Non-uniform workloads
• Hot keys/hot partitions
• Very large items
• Mixing hot data with cold data
• Remedy: Use TTL or a table per time period
- Disable retries, write your own retry code, and log all throttled or returned
keys
Design Patterns
Online dating website running on AWS
Users have people they like, and conversely
people who like them
Hourly batch job matches users
Data stored in Likes and Matches tables
Dating Website
DESIGN PATTERNS:
DynamoDB Accelerator and GSIs
Schema Design Part 1
GSI_Otheruser_id_other
(Partition key)
user_id_self
(sort key)
Requirements:
1. Get all people I like
2. Get all people that like me
3. Expire likes after 90 days
LIKES|
Likesuser_id_self
(Partition key)
user_id_other
(sort key)
MyTTL
(TTL attribute)
… Attribute N
Schema Design Part 2
Matchesevent_id
(Partition key)
timestamp
(sort key)
UserIdLeft
(GSI left)
UserIdRight
(GSI right)
Attribute N
GSI LeftUserIdLeft
(Partition key)
event_id
(Table key)
timestamp
(Table Key)
UserIdRight
GSI RightUserIdRight
(Partition key)
event_id
(Table key)
timestamp
(Table Key)
UserIdLeft
Requirements:
1.Get my matchesMATCHES|
Table Keys
Matchmaking
LIKES
Requirements:
1.Get all new likes every hour
2.For each like, get the other user’s likes
3.Store matches in matches table
Partition 1
Partition …
Partition NAvailability Zone
Public Subnet
match
making
server
security group
Auto Scaling group
Matchmaking
LIKES
Requirements:
1.Get all new likes every hour
2.For each like, get the other user’s likes
3.Store matches in matches table
Partition 1
Partition …
Partition NAvailability Zone
Public Subnet
match
making
server
security group
Auto Scaling group
THROTTLE!
Matchmaking Requirements:
1.Get all new likes every hour
2.For each like, get the other user’s likes
3.Store matches in matches table
1.Key choice: High key cardinality
2.Uniform access: access is evenly spread over the key-space
3.Time: requests arrive evenly spaced in time
Even Access:
Matchmaking
LIKES
Requirements:
1.Get all new likes every hour
2.For each like, get the other user’s likes
3.Store matches in matches table
Partition 1
Partition …
Partition NAvailability Zone
Public Subnet
match
making
server
security group
Auto Scaling group
0. Write like to like table, then query by user id to warm
cache, then queue for batch processing
security group
DAX
Takeaways:
Keep DAX warm by querying after writing
Use GSIs for many to many relationships
Dating Website
DESIGN PATTERNS:
DynamoDB Accelerator and GSIs
Amazon DynamoDB
DESIGN PATTERNS:
TTL, DynamoDB Streams, and DAX
Single DynamoDB table for storing sensor data
Tiered storage to remove archive old events to S3
Data stored in data table
Serverless IoT
Schema Design
DataDeviceId
(Partition key)
EventEpoch
(sort key)
MyTTL
(TTL attribute)
… Attribute N
Requirements:
1.Get all events for a device
2.Archive old events after 90 daysDATA|
UserDevicesUserId
(Partition key)
DeviceId
(sort key)
Attribute 1 … Attribute N
Requirements:
1.Get all devices for a userUSERDEVICES|
References
DATA
DeviceId: 1
EventEpoch: 1492641900
MyTTL: 1492736400 Expiry
AWS Lambda
Amazon S3
Bucket
Amazon DynamoDB Amazon DynamoDB Streams
Single DynamoDB table for storing sensor data
Tiered storage to remove archive old events to S3
Data stored in data table
USERDEVICES
Serverless
IoT
Serverless IoT
DATA
Partition A Partition B Partition DPartition C
Throttling
Noisy sensor produces data at
a rate several times greater
than others
Data
00
3F
BF
FF
Partition A25.0 % Keyspace
25.0 % Provisioned Capacity
Partition B25.0 % Keyspace25.0 % Provisioned Capacity
Partition D
25.0 % Keyspace25.0 % Provisioned Capacity
Hash.MIN = 0
Hash.MAX = FF
Ke
ysp
ace
Partition C25.0 % Keyspace25.0 % Provisioned Capacity
7F
Data
00
3F
BF
FF
Partition A25.0 % Keyspace
25.0 % Provisioned Capacity
Partition B25.0 % Keyspace25.0 % Provisioned Capacity
Partition D
25.0 % Keyspace25.0 % Provisioned Capacity
Hash.MIN = 0
Hash.MAX = FF
Ke
ysp
ace
Partition C25.0 % Keyspace25.0 % Provisioned Capacity
7F1.Key choice: High key cardinality2.Uniform access: access is evenly spread
over the key-space3.Time: requests arrive evenly spaced in
time
Even Access:
Serverless IoT
Requirements:
1. Single DynamoDB table for storing sensor data
2. Tiered storage to remove archive old events to
S3
3. Data stored in data table
0. Capable of dynamically sharding to overcome
throttling
Schema Design
ShardDeviceId
(Partition key)
ShardCount
Requirements:
1. Get shard count for given device
2. Always grow the count of shardsSHARD|
Requirements:
1. Get all events for a device
2. Archive old events after 90 daysData |
DataDeviceId
(Partition key)
EventEpoch
(sort key)
MyTTL
(TTL attribute)
… Attribute N
A sharding scheme where the number of
shards is not predefined, and will grow over
time but never contract. Contrast with a fixed
shard count
Naïve Sharding
Range: 0..1,000
DATA
DeviceId_ShardId: 1_3
EventEpoch: 1492641900
MyTTL: 1492736400
SHAR
DDeviceId: 1
ShardCount: 10
1.
2.
Serverless IoT: Naïve Sharding
Request path:
1.Read ShardCount from Shard table
2.Write to a random shard
3.If throttled, review shard count
Expiry
Serverless IoT
DATA
Partition A Partition B Partition DPartition C
Pick a random shard to write data to
DeviceId_ShardId: 1_Rand(0,10)
EventEpoch: 1492641900
MyTTL: 1492736400
2.
?
SHAR
DDeviceId: 1
ShardCount: 10
1.
DATA
DeviceId: 1
EventEpoch: 1492641900
MyTTL: 1492736400 Expiry
AWS Lambda
Amazon S3
Bucket
Amazon DynamoDB
Streams
Single DynamoDB table for storing sensor data
Tiered storage to remove archive old events to S3
Data stored in data table
Capable of dynamically sharding to overcome throttling
USERDEVICES
Serverless
IoT
SHAR
DDeviceId: 1
ShardCount: 10
DAX
+
Amazon Kinesis
Firehose
DESIGN PATTERNS:
TTL, DynamoDB Streams, and DAX
Takeaways:
Use naïve write sharding to dynamically expand shards
Use DAX for hot reads, especially from Lambda
Use TTL to create tiered storage
Serverless IoT
Getting started?
DynamoDB Local Document SDKsDynamoDB
Developer Resources
https://aws.amazon.com/dynamodb/developer-resources/
Thank you!