Date post: | 16-Apr-2017 |
Category: |
Technology |
Upload: | amazon-web-services |
View: | 719 times |
Download: | 0 times |
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sr. Product Manager, AWS
Deep Dive on S3 Storage ManagementOmair Gillani
Batches and Streams
AWS Direct Connect
AWS Snowball, Snowball Edge,
Snowmobile
3rd Party Connectors
Transfer Acceleration
AWS Storage Gateway
Amazon Kinesis Firehose
File BlockObject
Amazon GlacierAmazon S3 Amazon EC2 Instance Store
(ephemeral)
Amazon EFS Amazon EBS (persistent)
AWS Direct Connect AWS Snowball ISV Connectors
Amazon Kinesis Firehose
S3 Transfer Acceleration
AWS Storage Gateway
Data transfer into Amazon S3
AWS Snowmobile
AWS Snowball Edge
Storage Gateway hybrid storage solutionsEnables using standard storage protocols to access AWS storage services
Customer Premises
StorageGateway
Amazon EBS snapshots
Amazon S3Amazon Glacier
AWS Identity and Access Management (IAM)
AWS Key Management Service (KMS)
AWS CloudTrail
Amazon CloudWatch
Enterprise storage
Devices
Applicationservers
What’s new? File gatewayFile interface to objects in your Amazon S3 buckets
Virtual on-premises file server
Store and retrieve Amazon S3 objects through standard file storage protocols (NFS v3 and v4.1)
Local cache for low-latency access
Direct access to objects with native S3 capabilities, such as versioning, lifecycle management, and cross region replication
Launched
@ re:Invent!
What’s new? We’re reducing pricesEffective December 1, 2016
Storage prices reducedVolume and virtual tape storage by up to 28% Archived virtual tape storage by up to 43%
Flat per GB price replaces monthly gateway fee$0.01 per GB written to AWS storage by your gateway*
Simplified virtual tape retrieval pricing$0.01 per GB retrieved replaces peak-rate fee
* Up to $125/month per gateway
Launched
@ re:Invent!
Cross-Region ReplicationLifecycle Policy S3 Object TagsEvent Notifications
Amazon S3 CloudWatch Metrics
S3 Inventory Audit with AWS CloudTrail S3 Data Events
S3 Analytics
Standard Standard - Infrequent Access Amazon Glacier
Storage Management for S3
Understand your storage usage
S3 InventoryAnalyze Logs with Amazon EMR
S3 Analytics
S3 Inventory
Save time Daily or Weekly delivery Delivery to S3 bucketCSV File Output
Trigger business workflows and applications such as secondary index, garbage collection, data auditing, and offline analytics
Half the price of LIST API at $0.0025 per million objects listed
S3 Inventory
More information about your objects than provided by LIST API such as replication
status, multipart upload flag, and delete marker
Name Value Type DescriptionBucket String Bucket name. UTF-8 encoded.Key String Object key name. UTF-8 encoded.Version Id String Version Id of the objectIs Latest Boolean true if object is the latest version (current version) of a versioned object, otherwise falseDelete Marker Boolean true if object is a delete marker of a versioned object, otherwise falseSize Long Object size in bytesLast Modified String Last modified timestamp. Format in ISO: YYYY-MM-DDTHH:mm:ss.SSSZETag String eTag in HEX encoded format
StorageClass String Valid values: STANDARD, REDUCED_REDUNDANCY, GLACIER, STANDARD_IA. UTF-8 encoded.
Multipart Uploaded Boolean true if object is uploaded by using multipart, otherwise falseReplication Status String Valid values: REPLICA, COMPLETED, PENDING, FAILED. UTF-8 encoded.
S3 Inventory
Setup notification when S3 Inventory is complete
/Data /<InventoryFile>.gz /<InventoryFile>.gz …
/<DayofReport> /manifest.json /manifest.checksum …
AWS Lambda
Amazon SQS
Amazon SNS
Eventually consistent rolling snapshot
S3 Inventory
New objects may not be listed Recently deleted objects may still be included
O1
O2
O3
O1
O2
O3
O1
O2
O1
O2
O3NEW
Validate before you act!Use HEAD OBJECT or GET OBJECT
S3 Analytics – Storage Class Analysis
Analyze buckets, prefixes or tags
$0.10 per million objects analyzed per month
Daily Storage Class Analysis & Lifecycle candidates
Data-driven storage management for S3
Export Analysis data to your S3 bucket
S3 Analytics – Storage Class Analysis
S3 Analytics – Storage Class Analysis
S3 Analytics – Storage Class Analysis
Export to use BI tool of your choice
S3 Analytics – Storage Class Analysis
Daily Export Dataset
Simple to configure S3 Analytics
S3 Management Console PUT Bucket AnalyticsMultiple Policy
Documents
<AnalyticsConfiguration> <Id>...</Id> <Filter> ... </Filter> <StorageClassAnalysis> <DataExport> ... </DataExport> </StorageClassAnalysis> </AnalyticsConfiguration>
Monitor your storage
Monitor and Alert with CloudWatch
Audit your storage with CloudTrail Data Events
Server Access Logs
CloudWatch metrics for S3
Operational & performance monitoring Generate metrics for data of your choice
Entire bucket, Prefixes, and Tags Up to 1,000 object groups
1-minute CloudWatch metrics Alert and alarm on metrics
CloudWatch metrics for S3
Metric Name Metric value
AllRequests Count
PutRequests Count
PostRequests Count
GetRequests Count
ListRequests Count
DeleteRequests Count
HeadRequests Count
Metric Name Metric value
BytesDownloaded MB
BytesUploaded MB
4xxErrors Count
5xxErrors Count
FirstByteLatency ms
TotalRequestLatency ms
$0.50 per metric per month
S3 Data Events in CloudTrail
Perform security analysis, meet your IT auditing and compliance needs, and take immediate action on object-level activity to immediately improve security posture
Pricing: $1 per million data events recorded and storage charges apply
Log object level operations
Changes to bucket configurations
SNS notification for log delivery
Manage your data
Cross-Region Replication
Lifecycle Policies Event Notifications
S3 Object Tags
Manage your data
S3 Object Tags
Easily manage and control access for Amazon S3 objects Classify your data
Tag your objects with key-value pairs
Write policies once based on the type of data
AnalyzeLifecycle PolicyAccess Control
Deep dive on tags Tags are key-value pairs Maximum 10 tags per object Maximum key length—127 Unicode characters Maximum value length—255 Unicode characters Tag keys and values are case-sensitive.
2 ways to put tags via API Put objects with tag parameter, or add tag API after object is created
Simple pricing $0.01 per 10,000 tags per month
Lifecycle policies based on tags<LifecycleConfiguration> <Rule> <ID>sample-rule</ID> <Filter> <And> <Prefix>documents/</Prefix> <Tag> <Key>Project</Key> <Value>Delta</Value> </Tag> <Tag> <Key>Data type</Key> <Value>HPI</Value> </Tag> </And> </Filter> <Status>Enabled</Status> <Transition> <Days>365</Days> <StorageClass>GLACIER</StorageClass> </Transition> <Expiration> <Days>3650</Days> </Expiration> </Rule></LifecycleConfiguration>
Transition or expire storage using tags Simplify S3 lifecycle policies Filter with prefix, tag, or both
Thank you!