1. Taking advantage of the Amazon Web Services Family
@Ben_Hall
Ben@BenHall.me.uk
Blog.BenHall.me.uk
MaydayHQ.com

2. Agenda
S3
EC2
Structuring applications
Others
3. What is AWS?
The way most startups survive
Collection of services each solving one small part of the overall problem
Platform / Foundation
4. 5. http://www.cargolaw.com/images/disaster2007.Ital.Florida7.GIF
Amazon S3Amazon Simple Storage Service
6. Amazon Product ImagesTwitter Profile ImagesSmugMug 10tb each monthDropboxCurrently has 449 billion files
7. AppHarbours (or any PaaS) storage recommendation
8. 9. 10. Designed to provide 99.99% durability and 99.99% availability
Designed to provide 99.999999999% durability and 99.99% availability
Also versioning allows you to preserve, retrieve, and restore
every version of every object stored in your bucket
11. 12. How to access files?
13. Bucket Name
https://s3.amazonaws.com/DDDNorth_Demo_1/
ddd_north_s3.html
File Name
14. 15. 16. REST API
GET
DELETE
PUT
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html
17. GET / HTTP/1.1
Host: BucketName.s3.amazonaws.com
Authorization: AWS 15B4D3461F177624206A:xQE0diMbLRepdf3YB+FIEXAMPLE=


bucket


1000
false

my-image.jpg
2009-10-12T17:50:30.000Z
"fba9dede5f27731c9771645a39863328"
434234
STANDARD

8a6925ce4a7f21c32aa379004fef
mtd@amazon.com



18. PUT /my-image.jpg HTTP/1.1
Host: myBucket.s3.amazonaws.com
Authorization: AWS 15B4D3461F177624206A:xQE0diMbLRepdf3YB+FIEXAMPLE=
Content-Type: text/plain
Content-Length: 11434
Expect: 100-continue
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
x-amz-id-2: LriYPLdmOdAiIfgSm/F1YsViT1LW94/xUQxMsF7xiEb1a0wiIOIxl+zbwZ163pt7
x-amz-request-id: 0A49CE4060975EAC
x-amz-version-id: 43jfkodU8493jnFJD9fjj3HHNVfdsQUIFDNsidf038jfdsjGFDSIRp
Date: Wed, 12 Oct 2009 17:50:00 GMT
ETag: "fbacf535f27731c9771645a39863328"
Content-Length: 0
Connection: close
Server: AmazonS3
19. .NET SDK
using (AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretKey))
{
MemoryStreamms = new MemoryStream();
PutObjectRequest request = new PutObjectRequest();
request.WithBucketName("Demo_1")
.WithCannedACL(S3CannedACL.PublicRead)
.WithKey("image.jpg").InputStream = file.InputStream;
S3Response response = client.PutObject(request);
}
http://aws.amazon.com/sdkfornet/
20. .NET SDK
ListObjectsRequest request = new ListObjectsRequest();
request.BucketName = bucketName;
using (ListObjectsResponse response = client.ListObjects(request))
{
foreach (S3Object entry in response.S3Objects)
{
Console.WriteLine("key = {0} size = {1}", entry.Key, entry.Size);
}
}
21. https://s3.amazonaws.com/DDDNorth_Demo_1/
ddd_north_s3.html?torrent
S3 Torrent Support
22. 23. 24. CloudFront
Video Streaming
25. Index Page
26. Bucket Name
27. 28. date = Time.now.strftime("%a, %d %b %Y %H:%M:%S %Z")
digest = Base64.encode64(OpenSSL::HMAC.digest(OpenSSL::Digest::Digest.new('sha1'), aws_secret, date)).strip
uri = URI.parse("https://cloudfront.amazonaws.com/2010-08-01/distribution/#{distribution}/invalidation")
req = Net::HTTP::Post.new(uri.path)
req.initialize_http_header({
'x-amz-date' => date,
'Content-Type' => 'text/xml',
'Authorization' => "AWS %s:%s" % [aws_account, digest]
})
req.body = "" + path + "ref_#{Time.now.utc.to_i}"
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
res = http.request(req)
29. WARNING
HTTPS and CNAMEs are not supported
S3 bucket name must only contain lower case alphanumeric characters, periods, or hyphens
Bucket names must be unique across entire AWS
You cant rename Delete + Recreate (which costs)
30. Cost
Data transfer from EC2 machines in same region are free
AppHarbour runs out of US East Region, meaning transfers to US Standard are free.
31. 1gb stored, 10gb downloaded via 1,000,000 requests == $2.22 per month
1gb stored, 1tb downloaded via 10,000,000 requests == $132.90 per month
CloudFront has additional pricing. 10gb = $2
http://calculator.s3.amazonaws.com/calc5.html
32. 33. Amazon EC2Amazon Elastic compute cloud
http://www.flickr.com/photos/clarkk/38817024/
34. Heroku, AppHarbouretcZynga > 12,000 nodesAmazon SilkAmazon
35. Built on top of XEN virtualisationYes, you could run a private EC2
36. 37. 38. 39. 40. 41. Be safe
42. 43. 44. 45. 46. 47. 48. 49. Instance Types
On demand
Reserved
Spot
50. Spot Instance Bidding
Think Differently.
Design a system capable of supporting it
Save!
51. 52. 53. Instance Sizes
32bit micro, small, medium
64bit Extra-large >
Super computer GPU
1 ECU (biggest == 20) == 2007 Xeon processor / 1.0-1.2 GHz 2007 Opteron
54. Availability Zones
Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zone
2-3 per region.
55. Regions
US East (Northern Virginia)
US West (Northern California)
EU (Ireland)
Asia Pacific (Singapore)
Asia Pacific (Tokyo)
56. Business / Technical Decisions
Geo-location
Its a data centre, it can die. Scale across multiple regions for maximum up-time.
57. EBS & Snapshots
EBS == Hard drive.
Can increase volume size, but means taking machine down.
Snapshots == Backup. Can attach to new / different AMI
58. 59. 60. Keypairs
BE CAREFUL DONT LOSE THE FILE
Used to generate password / SSH auth
What do you do if you lose the key pair?
Storing in AMI? Encrypt with key passed in via user-data
61. User Data
Install-lamp
#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
apt-get update && apt-get upgrade -y
tasksel install lamp-server
echo "Please remember to set the MySQL root password!
ec2-run-instances --key KEYPAIR --user-data-file install-lamp ami-bf5eb9d6
62. AMI
Amazon Machine Image
Official base snapshots or private
AMI images allow you to spin up multiple machines. Great when combined with Spot Pricing.
Before creating AMI for Windows:
Disable SysPrep (BundleConfig.xml)
Disable machine renaming (Config.xml)
63. 64. 65. 66. 67. CloudWatch Alerts
68. Auto-scaling
Spin up / down nodes
69. 70. Starting machines via cmd
71. ec2-describe-images -o amazon
IMAGE ami-23b6534aec2-public-images/fedora-core4-apache.manifest.xml
IMAGE ami-25b6534cec2-public-images/fedora-core4-apache-mysql.manifest.xml
IMAGE ami-26b6534fec2-public-images/developer-image.manifest.xml
IMAGE ami-2bb65342ec2-public-images/getting-started.manifest.xml
ec2-run-instances ami-23b6534a -k windows
RESERVATION r-xxxxxxxxxxxxxxxxxxxxdefault
INSTANCEi-xxxxxxxxami-23b6534apending windows
ec2-describe-instances
RESERVATION r-xxxxxxxxxxxxxxxxxxxxdefault
INSTANCEi-xxxxxxxxami-23b6534aec2-xx-xxx-xx-xx.compute-1.amazonaws.com
ec2-terminate-instances i-xxxxxxxx
72. Via .NET SDK
73. public void LaunchEc2ImageWithSqlServerInstalled()
{
RunningInstancetargetMachineToStart = null;
string ami = "ami-e0916389";
varrunInstancesRequest = new RunInstancesRequest();
runInstancesRequest.ImageId = ami;
runInstancesRequest.MinCount = 1;
runInstancesRequest.MaxCount = 1;
runInstancesRequest.KeyName = "aws";
runInstancesRequest.InstanceType = "t1.micro";
runInstancesRequest.Placement = new Placement { AvailabilityZone = "us-east-1b" };
var response = _ec2.RunInstances(runInstancesRequest);
foreach (var instance in response.RunInstancesResult.Reservation.RunningInstance)
{
if(instance.InstanceState.Name == "pending") //NEED TO WAIT FOR "Running
targetMachineToStart = WaitUntilInstanceStateIs(instance.InstanceId, "running");
}
}
74. DescribeInstancesRequest request = new DescribeInstancesRequest();
varresponse = _ec2.DescribeInstances(request);
varcount = response.DescribeInstancesResult.Reservation.Count;
foreach(var reservation in response.DescribeInstancesResult.Reservation)
{
var instance = reservation.RunningInstance[0];
}
75. public string GetPassword(string instanceId)
{
string rsaPrivateKey;
using (StreamReader reader = new StreamReader(pathToPem))
{
rsaPrivateKey = reader.ReadToEnd();
}
GetPasswordDataResult result = _ec2.GetPasswordData(new GetPasswordDataRequest()
.WithInstanceId(instanceId)).GetPasswordDataResult;
var pass = result.GetDecryptedPassword(rsaPrivateKey);
Console.WriteLine(pass);
return pass;
}
76. WinRM
string path = C:WindowsSystem32winrs.exe
string args = string.Format("-r:{0} -u:Administrator -p:{1} "{2}"", machine.Url, GetPassword(machine.InstanceId), cmd);
Process.start(path, args);
Remotely connects to EC2 machine, executes command.
77. List commands = new List();
commands.Add(@"New-Item c:tmp -type directory -ErrorActionSilentlyContinue");
commands.Add(@"$memcached_url = ""http://www.meerkatalyst.com/beta/agent.zip""");
commands.Add(@"$memcached_path = ""c:tmpmeerkatalyst_agent.zip""");
commands.Add(@"$zip_url = ""https://s3.amazonaws.com/meerkatalyst-us/utilities/unzip.exe""");
commands.Add(@"$zip_path = ""c:tmpunzip.exe""");
commands.Add("$client = new-object System.Net.WebClient");
commands.Add("$client.DownloadFile( $memcached_url, $memcached_path )");
commands.Add("$client.DownloadFile( $zip_url, $zip_path )");
commands.Add(@"invoke-expression ""C:tmpunzip.exe -o C:tmpmeerkatalyst_agent.zip -d C:tmpagent""");
StringBuilder builder = new StringBuilder();
foreach (var command in commands)
builder.AppendFormat("echo {0} >> script.ps1 && ", command);
cmd = string.Format("{0} powershell -File script.ps1 && del script.ps1 && {1} &", builder, pathToAgent)
var output = ExecuteRemoteCommand(cmd, machine);
//ask me for the code
78. WARNINGS
http://www.flickr.com/photos/gagilas/2659695352/
79. Temp Data
It will get deleted on reboots.
RabbitMQ + Hbase both used temp locations as default storage.
Reboot == lost work.
Takes a long time to debug.
80. Sharepoint / SQL Server + Rename
Microsoft products generally dont like a machine to be renamed.
Reboots == rename as its based on IP.
Return it off as described earlier.
81. Regions, Zones.
They can go down.
82. 83. Unofficial AMIs
They are configured by someone else
Who knows what they have done.
84. 85. 86. 87. Double prices if you want SQL Server
More of less anyway.
Large$0.48 per hour => $1.08 per hour
Extra Large$0.96 per hour => $1.56 per hour
88. 89. Architecture & AWSApplication considerations for AWS
http://www.flickr.com/photos/wouterpostma/695850212/
90. Configuration
Self discovery
Use DNS + Load Balancers to reduce config changes
Internal DNS on EC2?
HAProxy
Store in external system which can update / replicate to other nodes.
91. Hybrid Cloud Providers
One wont fit all
Optimize for use-case
Best of breed
CAN BE REALLY EXPENSIVE + COMPLEX!
92. Think!
At some point and scale, the cloud doesnt make sense.
Zynga has around 12,000 EC2 nodes
Once game is proven, moved to private cloud. EC2 for experiments and additional load.
93. Small, isolated components
Design system for scale
Design system for failure
Message Queues are great! Use them.
Keep everything async
94. Users Browser
Javascript File
via Lighttpd
ELB
Single EC2 Machine
Easily add additional nodes to ELB
Users Browser
Javascript File
via S3
CloudFront
95. Heroku + NodeJS
Data Collection
Ruby Data Processor
RabbitMQ Queue
HBase Data Storage
Heroku + Rails
UI
Single EC2 Machine
Three separate components, designed to scale each individually when required.
Deployment, Scale, Cost
96. Chaos Monkey
Kill random machines. See what breaks, fix it.
Amazon will do it without telling you, be prepared and plan ahead!
http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html
97. Other AWS ServicesElasticCache, MapReduce, SQS, SNS, etc
http://www.flickr.com/photos/kky/704056791/
98. 99. 100. Elastic Beanstalk Easily deploy Java based applications + Stack
VPC virtual network topology
Elastic MapReduce Hosted Hadoop
CloudFormation Similar to Chef / Puppet. Start group of machines
RDS Hosted MySQL / Oracle
ElastiCache Hosted Memcached
SNS - Simple Notification Service. Similar to a queue
IAM Sharing access to AWS with a team
101. Amazon Mechanical Turk
EC2 for Humans!
Crowdsourcing work
Grockit used it to verify content - http://mechanicalturk.typepad.com/blog/2011/10/editors-note-ari-bader-natal-is-the-chief-learning-architect-at-grockit-the-fast-growing-online-social-learning-startup-bac.html
Usability Test on www.somewebsite.com #1148 - $10
Click on google +1 for my website - $0.01
102. http://awsdocs.s3.amazonaws.com/toolkit-visualstudio/latest/aws-tkv-ug.pdf
AWS Toolkit for Visual Studio
http://aws.amazon.com/visualstudio/
103. 104. 105. 106. 107. 108. 109. 110. Summary
http://www.flickr.com/photos/leon_homan/2856628778/
111. Amazon Web Services
Fast
Cheap
Flexible
http://highscalability.com/
112. THANK YOU!
@Ben_Hall
Ben@BenHall.me.uk
Blog.BenHall.me.uk