1/24/2011
1
CIT 668: System Architecture
Amazon Web Services II
Topics
1. Elastic Block Service (EBS) 2. Simple Storage Service (S3) 3. CloudFront 4. CloudWatch (and AutoScaling) 5. Simple Notification Service (SNS) 6. Simple Queue Service (SQS) 7. SimpleDB 8. Relational Database Service (RDS) 9. Route 53 10. Elastic BeanStalk 11. Flexible Payment Service 12. Mechanical Turk
Elastic Block Store (EBS)
1/24/2011
2
Elastic Block Store (EBS)
EBS Volumes are up to 1TB in size – Attach to any EC2 instance in same AZ
– Create snapshots at any time
– Create new volumes based on snapshots
Reliability – Annual Failure Rate (AFR) of 0.1-0.5%
– Commodity hard disk AFR is ~4%
– About as reliable as a RAID set
– Use snapshots for backups
Pricing per GB-month
EBS Snapshots
Snapshots saved to S3
– Not visible by S3 API
– Include only disk blocks changed since last snapshot
Snapshots are fast
– Only changed blocks since last snapshot need to update
http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/
Creating an EBS Volume
Create volume of size 10GB in specified zone
ec2addvol -s 10 -z us-east-1c
EBS volumes are region specific
1/24/2011
3
Attaching an EBS Volume
Attach volume to an instance
ec2attvol -d /dev/sdh -i i-71b6181d vol-9452d0fc
Check volume status
ec2-describe-volumes vol-9452d0fc
Using an EBS Volume
Login to instance
ssh -i keypath root@public_name
Make partitions
fdisk /dev/sdh
Create ext3fs filesystem
mke2fs –j –v /dev/sdh1
Mount filesystem
mkdir /cit668
mount /dev/sdh1 /cit668
Detaching an EBS Volume
Detach volume from an instance
ec2-detach-volume vol-9452d0fc
Delete a volume permanently
ec2-delete-volume vol-9452d0fc
Attached volumes cannot be deleted
1/24/2011
4
Creating an EBS Snapshot
ec2addsnap -d 'test snapshot' vol-9452d0fc SNAPSHOT snap-3b0a4156 vol-9452d0fc pending 2011-01-22T21:10:10+0000137367125081 10 test snapshot
Attaching an EBS Snapshot
Create an EBS volume from the snapshot
ec2addvol --snapshot snap-3b0a4156 -z us-east-1c
VOLUME vol-aa65e7c2 10 snap-3b0a4156 us-east-1c creating 2011-01-22T21:20:13+0000
Attach the new EBS volume
ec2attvol -d /dev/sdi -i i-71b6181d vol-aa65e7c2
Login to instance and mount volume
mkdir /cit668-snap
mount /dev/sdi1 /cit668-snap
Simple Storage Service (S3)
1/24/2011
5
S3 Features
• An Internet-scale data storage service
– All data is stored redundantly in multiple AZs
– Data is located in the region you specify
• Stores objects from 1 byte to 5TB in size
• Objects are stored in a bucket and retrieved via a unique, developer-assigned URL
• You can have 100 named buckets
• Each bucket can store an unlimited
objects in a flat namespace.
S3 Pricing
Applications of S3
Fast, scalable, and reliable web file hosting
– Especially useful for audio and video files
http://aws.amazon.com/articles/1073
1/24/2011
6
S3 ACLs
• Both buckets and objects can have ACLs.
• ACLs can have up to 100 entries.
• Owner always has READ_ACP and WRITE_ACP on buckets and objects even if removed in ACL.
S3cmd
To use for the first time, set up credentials with s3cmd --configure
Bucket creation s3cmd mb s3://USERNAME-test
Upload files s3cmd put www/index.html s3://USERNAME-test
List files s3cmd ls s3://USERNAME-test
Download files s3cmd get s3://USERNAME-test/index.html /tmp/index.html
Remove files s3cmd del s3://USERNAME-test/index.html
Remove bucket s3cmd mb s3://USERNAME-test
1/24/2011
7
Configure Master Password in Firefox
Set Master Password
S3 Fox: Manage Accounts
1/24/2011
8
S3Fox: Create a Bucket
1. Right click in right pane to bring up S3 menu 2. Select “Create Directory”
S3Fox: Uploading Files
S3Fox: Edit ACL
1/24/2011
9
Accessing S3 Files
Import/Export
1. Prepare a portable storage device – eSATA or USB external hard drives
– 2.5” or 3.5” internal SATA hard drives
2. Submit create job request – Amazon S3 bucket for store
– AWS access key ID
– Return address
3. Securely identify device to placing signature file on it.
4. Ship device will all connectors
Import/Export Pricing
1/24/2011
10
CloudFront
CloudFront
“Amazon CloudFront delivers your static and streaming content using a global network of edge locations. Requests for your objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance.”
http://aws.amazon.com/cloudfront/
CloudFront = Content Distribution Network
Throughput Improvement
– Instead of being limited to one ISP
– Use multiple ISPs with much higher bandwidth
– Ex: If each ISP offers 1 Gbps, then CDN with 10 edge servers at different ISPs offers 10 Gbps
Latency Improvement
– Since edge servers are closer to customers, latency is lower, which is important for streaming media.
Availability
– Since edge servers use different ISPs and power grids, even large scale failures won’t bring down entire CDN.
1/24/2011
11
Other CDNs
Free CDNs Commercial CDNs
CloudFront Edge Locations
US Ashburn, VA
Dallas/Fort Worth, TX
Jacksonville, FL
Los Angeles, CA
Miami, FL
New York, NY
Newark, NJ
Palo Alto, CA
Seattle, WA
St. Louis, MO
Europe
Amsterdam
Dublin
Frankfurt
London
Asia
Hong Kong
Tokyo
Singapore
CloudFront Configuration
1. Specify origin server – S3 bucket OR
– Your own server URL
2. Get CloudFront domain: abc.cloudfront.net
3. Use CloudFront domain in your links – CloudFront will identify which edge location will best
serve your content
per 10,000 HTTPS requests (HTTP are 25% less)
per GB transferred (lower rates if > 10TB transferred)
1/24/2011
12
CloudWatch
CloudWatch
Monitoring service for EC2 – CPU utilization
– Data transfer
– Storage usage
Features – Notifications at user-specified metric thresholds
– Enables AutoScaling at metric thresholds
Pricing – Basic Monitoring with 5 minute granularity free
– Detailed Monitoring (1 minute) for 1.5₵ per hour
– 10₵ per alarm after first 10 alarms
AutoScaling
AutoScaling Group
– Set of EC2 instances that should scale together
Triggers
– Scale on CloudWatch alerts
– Scale on time-based schedule
– Fixed number of healthy instances
Examples
– Add 3 instances if CPU > 50%
– Remove 3 instances if CPU < 10%
1/24/2011
13
Simple Notification Service
Simple Notification Service (SNS) Setup
1. Create a topic
– Identifies a specific subject or event type
– Clients subscribe to topics to get notifications
2. Set topic policies
– Transport protocols: HTTP, HTTPS, SMTP
3. Add subscribers to topic
– Subscribers specify end point (URL, email address) for notification delivery
4. Publish notifications
– Topic owners publish update messages to topic
SNS Applications
• Relay events between distributed computing clients
• Update records in business systems, such as inventory changes and shipping status
• Relay real time information to mobile devices
• Monitoring notices from CloudWatch
1/24/2011
14
Simple Queue Service (SQS)
Simple Queue Service (SQS)
• SQS is a reliable, highly scalable hosted distributed queue for storing messages as they travel between computers.
• Features
– Reliable
– Scalable
– Secure
• Used to “pipe” data between
processes (possibly on different
instances)
Queues
• Developers can create an unlimited number of queues in each of the 4 regions
– Messages can be retained for up to 14 days
– Messages are up to 64KB in size
• Caveats about distributed message queues
– Messages may not be delivered in order
– Messages may be delivered multiple times
per 10,000 SQS requests
1/24/2011
15
SimpleDB
SimpleDB
• Cloud-based non-relational data store
• Data is stored in domains (tables)
– Domains have a set of attributes (columns)
– Attributes can have up to 256 values
– Domains can have up to a billion items (rows)
• SimpleDB can be queried using a simple version of SQL
– Does not support JOIN operations
Attributes can be added Dynamically
Initial model for person domain
Effect of adding Middle name attribute
1/24/2011
16
SimpleDB Pricing
Relational Database Service
Relational Database Service
Scalable MySQL 5.1 servers in the cloud with
– Automated backups and snapshots
– Replication (across availability zones if needed)
RDS instance classes
1/24/2011
17
Route 53
Route 53
Cloud-based DNS hosting
– Must have your own DNS domain
Pricing
per hosted zone per month
per million DNS queries per month (up to 1 billion)
per million DNS queries per month (after 1 billion)
Elastic Beanstalk
1/24/2011
18
Elastic Beanstalk
PaaS with many platforms, configurable IaaS To deploy Java applications using Elastic Beanstalk:
1. Create your application as you normally would 2. Package your deployable code into a standard Java Web
Application Archive (WAR file). 3. Upload your WAR file to Elastic Beanstalk 4. Deploy your application.
Elastic Beanstalk handles – provisioning of a load balancer – deployment of your WAR file – to EC2 instances running Apache Tomcat application server.
Access application via customized URL – http://myapp.elasticbeanstalk.com/
An Elastic Beanstalk Application
http://aws.typepad.com/aws/2011/01/aws-beanstalk-concepts.html
Configuration
Server - EC2 instance type, security groups, key pair (for SSH login), CloudWatch monitoring interval, and AMI ID (for customization).
Load Balancer - SSL Certificate, health check intervals, URL
Auto Scaling - Minimum and maximum instance count, and nine settings for the trigger that is used to drive scaling decisions.
Notifications - An email address that can receive notification of important application events such as scale-up and scale-down actions initiated by Auto Scaling.
Container - With the default (Java / Tomcat 6) container, this tab lets you control the size of the Java Virtual Machine's heap and garbage collection parameters.
1/24/2011
19
Flexible Payments Service
Flexible Payments Service (FPS)
Allows you to use Amazon’s payment service
– Customers must have an Amazon.com account
– Can also be used to pay third parties
Features
– One-time payments
– Configure periodic and delayed payments
– Aggregate micropayments into single transaction
– Facilitate transactions between buyer and a third party seller, while taking a cut of the fee
Mechanical Turk
1/24/2011
20
Mechanical Turk
How It Worked
How It Works on Amazon
www.mturk.com
Workers
Requester (Developer)
Human Intelligence Tasks (HITs)
Completed HITs
Worker Qualifications
1/24/2011
21
Applications
Software Developers
Solves problem of building applications that until now have not worked well without human intelligence
Businesses Efficiently, effectively and inexpensively complete millions of tasks that require human intelligence
Anyone 1. Make money using the skills you already have whenever your schedule allows.
2. Previously unimaginable business ideas are now feasible…
Business Model
– Podcast transcription service provider
– Transcribes audio into high-quality text
Amazon Mechanical Turk: Workers transcribe podcasts and index text within search engine
Amazon S3: Stores podcasts and related files
Key Points
Simple Storage Service (S3) is scalable web based storage based on – Buckets that contain a variable number of objects – Objects in buckets accessible via S3 URL
A Content Distribution Network (CDN) replicates and distributes content through a network of edge nodes near the customers. Queue services:
– Simple Notification Service for real-time notifications – Simple Queue Service for piping messages between app components
Cloud services: – SimpleDB is non-relational datastore with simple SQL – RDS uses configurable MySQL 5.1 instances – Route 53 provides scalable DNS
Mechanical Turk leverages global workforce to perform small tasks for micropayments
