Post on 22-May-2020
transcript
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jeff Bartley | Storage Solutions Architect | jeffbart@amazon.com
AWS Storage GatewayFred Hutch - Partly Cloudy Conference
October 25th, 2018
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analytics | File Services | Production Tiering | Data Processing Data Distribution | Backup | DR | Archive | Migration
Enabling a spectrum of hybrid use cases
Enterprise Data Centers
Amazon EBS
Amazon S3
Amazon Glacier
App. serversFile servers
Research Sites
AWS Storage Gateway
DevicesDatabasesMultimedia content
Amazon EC2
AWSLambda
Amazon CloudFront
AmazonAthena
Amazon EMR
Backup serversUsers
Remote Offices Small-Medium Businesses
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway Family
File GatewayStore and access objects in Amazon S3 from file-based applications with local caching
Volume GatewayBlock storage on-premises backed by cloud storage with local caching, EBS snapshots, and clones
Tape GatewayDrop-in replacement for physical tape infrastructure backed by cloud storage with local caching
Hybrid storage service enabling applications to seamlessly use AWS storage
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway Family
Native storage in AWS
• Objects in S3 (file)
• Snapshots in EBS (volume)
• Archives in Glacier (tape)
Gateway provides applications
• Protocol conversion and device emulation
• Caching (read-through / write-back)
• Optimized data transfer
AWS Identity and Access Management (IAM)
Amazon CloudWatch
AWS Key Management Service (AWS KMS)
AWS CloudTrail
Files(NFS or SMB)
Volumes(iSCSI)
Tapes(iSCSI VTL)
Amazon S3
Amazon Glacier
Amazon EBS snapshots
Customer Premises
AWS Storage Gateway
HTTPS
Gateway Service
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File GatewayStore and access objects in Amazon S3 from file-based applications with local caching
Customer Premises
HTTPSNFS or SMB
File Gateway Objects in your S3 bucket
Application Server
Reduce on-premises
storage infrastructureFully managed local
cache for low latency
access
Durability, scalability,
and reliability of
Amazon S3 storage
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway support for SMB
Store and access objects in Amazon S3 buckets
from file-based Windows applications
File Gateway Objects in your Amazon S3 bucket
SMB v2 or v3client
On-premises Active Directory (AD) or AWS
Directory services
Share-level access can be restricted to
configured users and groups
POSIX ACLs (compatible subset of NTFS) for object-level
permissions
Objects created directly in S3 inherit
ACLs from parent folder
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hybrid File Use Case – Backup to AWS
File GatewayDatabase server
Objects in S3
Amazon CloudWatch
Events
AWS RegionCustomer Premises
1
Database server creates backup file1
2
Database instance
3
2 Gateway takes care of upload to S3
3 Automate restore or validation in-cloud or
lifecycle to Glacier for archival
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hybrid File Use Case – Content Distribution
San Diego Data Center
File Gateway(read-only)
Application
Sydney Data Center
Objects in your S3 bucket
AWS Region
File Gateway
1
Application
2
Application in San Diego writes files which are uploaded to S3 by gateway1
2 After refresh cache, files are visible to applications in Sydney. Local cache
improves access performance.
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AmazonGlacier
S3 Standard S3-Infrequent Access
File GatewayApplicationServers
Hybrid File Use Case – Active Archive
AWS Direct Connect
Customer Data Center
LTO
NAS
1
2
Use Snowball to ship data from on-premises offline archives1
2 Online access to all data through gateway, reduces on-premises storage
cost, S3 lifecycle policies to lower costs further
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enabling Cloud Workloads
“We have applications that don’t support S3 natively, and we can’t make changes to the applications, so being able to use native protocols is very important to us.
Storage Gateway allows Celgene to get data to the cloud without making any changes to the application or environment.”
– Lance Smith, IT Director, Celgene
Easily migrate existing applications to the cloud
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
File Gateway Deep Dive
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway – Control Over Data Storage and Access
File share options
• IAM role• Storage class• Encryption with AWS KMS• Guess MIME type, requester pays,
bucket owner ACL, …
Amazon S3 options per bucket
• Restrict access by IP (NFS) or AD (SMB)• Read-only/read-write• Default ownership and permissions• User squashing (NFS)
File GatewayNFS or SMB client
Objects in your S3 bucket
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NFS v3 or v4client
s3://{Bucket2}/{Prefix}/{Object1}s3://{Bucket2}/{Prefix}/{Object2}s3://{Bucket2}/{Prefix}/{Object3}
File Gateway – Mapping Files to Objects
{IP}:/{Bucket2}/{Folder}/{File1}{IP}:/{Bucket2}/{Folder}/{File2}{IP}:/{Bucket2}/{Folder}/{File3}
File Gateway
SMB v2 or v3 client Objects in
your Amazon S3 bucket
\\{IP}\{Bucket1}\{Folder}\{File1}\\{IP}\{Bucket1}\{Folder}\{File2}\\{IP}\{Bucket1}\{Folder}\{File3}
s3://{Bucket1}/{Prefix}/{Object1}s3://{Bucket1}/{Prefix}/{Object2}s3://{Bucket1}/{Prefix}/{Object3}
Bucket1
Bucket2
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway – Storage Architecture
AWS RegionCustomer Premises
S3 BucketNFS or SMB client
Fileshare
File Gateway
Cache Disk
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway – Storage Architecture
AWS RegionCustomer Premises
S3 BucketNFS or SMB client
Cache Disk
Fileshare
File Gateway
Datacache
Metadata cache
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway – Write
AWS RegionCustomer Premises
S3 BucketNFS or SMB client
File Gateway
Datacache
Metadata cache
Fileshare
Writes to S3 are optimized and encrypted
Written data remains in cache
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway – Read from Cache
AWS RegionCustomer Premises
S3 BucketNFS or SMB client
File Gateway
Datacache
Metadata cache
Fileshare
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway – Read from S3
AWS RegionCustomer Premises
S3 BucketNFS or SMB client
File Gateway
Datacache
Metadata cache
Fileshare
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway – Low Latency Access to Active Data
AWS RegionCustomer Premises
S3 BucketNFS or SMB client
File Gateway
Datacache
Metadata cache
Fileshare
Durable storage of all of your data
Fully managed cache of
recent data
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway – Metadata Cache
AWS RegionCustomer Premises
S3 BucketNFS or SMB client
File Gateway
Datacache
Metadata cache
Fileshare
Metadata is cached based on operations performed by the file client
Bucket can have billions of objectsin it, gateway discovers them on-
demand
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway - File System Metadata
• File system metadata (permissions, ownership, ctime, etc.) persisted in object user-metadata
• Configurable defaults for objects that don’t have this metadata (i.e. objects that were already in the bucket)
• Changing file metadata results in a copy-put operation
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway - File system operations
directories
• Folder objects created in S3 using same pattern as S3 Console
• Walking the file system can be expensive (ls -R, find, etc).
• Caching of metadata locally reduces latency of directory level operations (such as “ls”).
delete
• Removes the object in S3
rename
• Atomic for clients connected to the same gateway
• Copy-put request on S3, so eventually consistent for S3 clients
links
• Hard links and symbolic links are not supported
• No analog in S3, so returns “Operation not supported”
sparse files
• Object is zero-filled for sparse ranges
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployment
On-Premises virtual environmentsRun as virtual machine on VMware ESXi and Microsoft Hyper-V
Hardware AppliancePre-installed with Storage Gateway software. Ideal for branch offices, warehouses, and “outpost” offices that lack dedicated IT resources.
Amazon EC2Run Storage Gateway in your AWS VPC environment
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployment Best Practices
VM environments
• Four virtual processors assigned to the VM.
• 16 GiB of reserved RAM assigned to the VM.
• 80 GiB of disk space for installation of VM image and system data.
Cache Disk
• One or more disks - 150 GB to 16 TB total cache
• Use redundant storage
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudWatch monitoring/alerting
Metrics
• Cache metrics
• S3 read/write bytes
• Share read/write bytes
Events
• Notify on file upload
• Cache refresh
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
To Learn More…
• Whitepapers
• Webinars
• Demos & Videos
• Developer Resources
• Recorded AWS re:Invent
Sessions
• Product Documentation
• Pricing
• FAQs
• Customer Case Studies
aws.amazon.com/storagegateway/
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Thank you!