Date post: | 12-May-2015 |
Category: |
Technology |
Upload: | jazkarta-inc |
View: | 2,657 times |
Download: | 2 times |
Scalable Plone hosting with Amazon EC2
for Rice University’sRhaptos open learning
platformNate Aune - Jazkarta.comPlone Conference 2010
Bristol, UK
1Sunday, November 7, 2010
Agenda
• Benefits of cloud computing
• What services does Amazon provide?
• 5 minute launch of Plone on EC2
• Case study: Rhaptos hosting on EC2
• Questions?
2Sunday, November 7, 2010
What makes the Cloud so attractive?
3Sunday, November 7, 2010
Abstract resourcesFocus on your needs, not on hardware specs.
As your needs change, so should your resources.
4Sunday, November 7, 2010
On-demand provisioningAsk for what you need, exactly when you need it.
Get rid of it when you don’t need it.
5Sunday, November 7, 2010
Scalability in minutesScale out or in depending on usage needs.
6Sunday, November 7, 2010
Pay per consumptionNo contracts or long-term commitments.
Pay only for what you use.
7Sunday, November 7, 2010
Efficiency of ExpertsUtilize the skills, knowledge and resources of experts.
8Sunday, November 7, 2010
Most applications need:
• Compute
• Storage
• Messaging
• Payment
• Distribution
• Scale
• Analytics
9Sunday, November 7, 2010
Flexible infrastructure
• Elastic Computing Cloud (EC2)
• Amazon Machine Images (AMI)
• Elastic IPs
• Elastic Block Storage (EBS)
• Auto-scaling
• Elastic Load Balancing (ELB)
10Sunday, November 7, 2010
Other services
• Simple Storage Solution (S3)
• CloudWatch (monitoring)
• CloudFront (CDN)
11Sunday, November 7, 2010
http://uec-images.ubuntu.com/
Text
12Sunday, November 7, 2010
mr.awsomeBuildout recipe to manage EC2 instances
13Sunday, November 7, 2010
Add an AWS part to buildout[buildout]
parts = aws
[aws]
recipe = zc.recipe.egg
eggs = mr.awsome
entry-points =
aws=mr.awsome:aws
assh=mr.awsome:aws_ssh
arguments = configpath="${buildout:directory}/etc"
14Sunday, November 7, 2010
Make a etc/aws.conf file[securitygroup:demo-server]description = Bristol Demo Serverconnections = tcp 22 22 0.0.0.0/0 tcp 80 80 0.0.0.0/0 tcp 8080 8080 0.0.0.0/0
[instance:demo-server]keypair = bristol-keypairsecuritygroups = demo-serverregion = us-east-1placement = us-east-1ainstance_type = t1.microimage = ami-480df921startup_script = startup-demo-serverfabfile = fabfile.py
15Sunday, November 7, 2010
Make a bash script: etc/startup-demo-server
#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
wget http://www.enfoldsystems.com/pubkey.gpg
apt-key add ./pubkey.gpg
echo "deb http://dist.clients.enfoldsystems.com/ubuntu lucid universe" >> /etc/apt/sources.list
apt-get update && apt-get upgrade -y
apt-get install plone-default -y
/etc/init.d/plone-default
16Sunday, November 7, 2010
Set your access keys
#!/bin/bash
export AWS_ACCESS_KEY_ID="<your_access_key_id>"
export AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"
Set these environment variables.
$ source setkeys.sh
$ env
File: setkeys.sh
17Sunday, November 7, 2010
Start the demo server$ ./bin/aws start demo-server
INFO: Instance 'demo-server' unavailable
INFO: Creating instance 'demo-server'
INFO: Instance created, waiting until it's available
....
INFO: Instance 'demo-server' available
INFO: Instance running.
INFO: Instances DNS name ec2-50-16-25-92.compute-1.amazonaws.com
INFO: Instances public DNS name ec2-50-16-25-92.compute-1.amazonaws.com
18Sunday, November 7, 2010
Benefits of EC2• Seamless disaster recovery with repeatable
deployments using buildout and mr.awsome
• Easy launching of testing, staging and QA servers that are isolated from your production environment (stop when not used to avoid paying for them)
• Pay-for-what-you-use metered pricing to launch servers on-demand during peak periods
• Auto-scaling to launch new Zeo client machines and shut them down when no longer needed
•
19Sunday, November 7, 2010
Connexions & RhaptosA case study in deploying OSS to the cloud
20Sunday, November 7, 2010
s
Log In Contact Us Report a Bug
SearchSearch Site
Home Content Lenses About Us Help MyCNX
FEATURED CONTENT
New features
1 2 3
MY ACCOUNT
SPOTLIGHT
Skip
ConnexionsSections
You are here: Home
Connexions is:a place to view and share educational material made ofsmall knowledge chunks called modules that can beorganized as courses, books, reports, etc. Anyone may viewor contribute:
authors create and collaborate
instructors rapidly build and share custom collections
learners find and explore content
More about us ...
Advanced Algebra IIThe world of second-yearalgebra comes alive in KennyFelder's Advanced Algebra II.Felder, a high school mathteacher in North Carolina,designed his course using anontraditional approach to a verytraditional subject. Centeredaround a Homework andActivities book, along withaccompanying Conceptual
Explanations and Teacher's Guide collections,Advanced Algebra II tackles topics by focusing oncomprehension rather than straight lecture delivery.
Advanced Algebra II was selected by California's FreeDigital Textbook Initiative to be included in a list offree textbooks available to California schools startingin Fall 2009.
Collaborative StatisticsCollaborative Statistics waswritten by two faculty members
FIND CONTENT
16547 reusable modules woven into 1007 collections.
or browse by ...
Language
Popularity
Title, author, etc.
CREATE CONTENT
Creating content in Connexions is as easy as 1, 2, 3:
Get an accountand log in to
your workspace.
Make a modulefrom scratch or
convert it from aWord doc.
Publish yourworks, sharing
them with the world.
Jump right in
Get an account
How to create a module inminutes
How to create a collectionwith existing modules
Guides and tutorials
New author guide
Connexions Tutorial andReference
Username
Password
Log in
Get an account
Forgot your password?
User feedback"I enjoy usingtheConnexionssite and thephilosophybehind itencouragesone to learnnew fields and
to network with others. Yoursite is my new homeschool onthe internet. It is helping meto grow as a writer ofacademic materials as I readwhat others have written so asto shape my own. I am alsoamazed at the range ofinstitutions where your writerscome from. Thanks for the useof these great facilities."
Go
ArtsBusinessHumanitiesMathematics and StatisticsScience and TechnologySocial Sciences
Search Content
Subject
21Sunday, November 7, 2010
Search
Login Preferences Help/Guide About Trac
Wiki Timeline Roadmap Browse Source View Tickets New Ticket Search
Start Page Index History Last Change
Welcome to Connexions and Rhaptos Software Development
Connexions ( cnx.org) is legos for education. It is a world-wide repository and publishing system to build textbooks, courses, lessons, andjournals. Individual authors, major content publishers, and educational organizations all contribute. Lenses provide a way for high-quality materialto be endorsed, recommended, categorized, and aligned to standards. Rhaptos is the open source software that runs the Connexions site andEnterprise Rhaptos is the software for running private content installations. Read more about Connexions and Rhaptos mission and architecturehere.
Enterprise Rhaptos Connexions and Rhaptos Development
API's for Accessing Data Quality Assurance and Testing
Enterprise Rhaptos
Enterprise Rhaptos enables the creation, sharing, modification, and vetting of educational materials accessible to anyone, anywhere, anytime viathe World Wide Web. Rhaptos' unique features of permanent versions of all materials, a simple, semantic document language, and a powerfullensing system for post-publication quality control, customized tagging, and community-based search and discovery are the engine behind a trulyreusable repository of knowledge and learning. Enterprise Rhaptos allows companies and individual organizations to host their own privateauthoring platform and publishing repository.
Installation Instructions for Enterprise Rhaptos
API's for Accessing Connexions Data
Table of ContentsWelcome to Connexions and Rhaptos Software DevelopmentEnterprise RhaptosAPI's for Accessing Connexions Data
Accessing content, feeds, and statisticsDownloading, transforming and branding contentSoftware Architecture and Application Programming Interfaces
Connexions Rhaptos DevelopmentDeveloper Resources (Getting involved)Roadmap, Designs, DesiresStarting Points for Connexions and Rhaptos Software Development
Quality Assurance and TestingTest, Release, and Configuration DocsUsing the Connexions Rhaptos Trac Bug System
Release and Configuration Details for Connexions and RhaptosMore about TRAC
22Sunday, November 7, 2010
Virtualization of Rhaptos
Two primary reasons to move to the cloud:
1) Hurricane territory (disaster recovery)
2) Increase adoption by other universities
Cost was also a consideration but not a primary reason for carrying out the project.
23Sunday, November 7, 2010
Deploy to Amazon EC2 with Python scripts
• Define server profiles with OS dependencies and turn into an AMI (Amazon Machine Image)
• Install & configure application on EBS volume
• Take snapshot of volume
• Launch new AMIs and attach EBS volume created from saved snapshot.
24Sunday, November 7, 2010
25Sunday, November 7, 2010
Plone Conference 2009 – Federico C. Guizzardi – 21 Aprile 2009
Distributed Architecture
Internet
Webserver
Cache Sys
Load Balancer
s1
s3
s4 s5
s2
10
proxy
frontend1
frontend2
frontend3
backend
26Sunday, November 7, 2010
[macro:base-securitygroup]connections = tcp 22 22 0.0.0.0/0 tcp 80 80 0.0.0.0/0
[securitygroup:backend]<= macro:base-securitygroupdescription = Bristol backend machine running Zeo server
[securitygroup:frontend1]<= macro:base-securitygroupdescription = Bristol frontend machine running Zeo client 1connections = tcp 8080 8080 0.0.0.0/0
[securitygroup:frontend2]<= macro:base-securitygroupdescription = Bristol frontend machine running Zeo client 2connections = tcp 8080 8080 0.0.0.0/0
[securitygroup:proxy]<= macro:base-securitygroupdescription = Bristol proxy machine running Apache, Varnish, HAProxy
aws.conf - define security groups
27Sunday, November 7, 2010
[macro:base-instance]image = ami-480df921keypair = bristol-keypairregion = us-east-1placement = us-east-1cinstance_type = t1.microfabfile = fabfile.py
[instance:backend]<= macro:base-instancesecuritygroups = backendstartup_script = backend-startup.ship = xxx.xxx.xxx.xxx
[instance:frontend1]<= macro:base-instancesecuritygroups = frontend1 startup_script = frontend-startup.ship = xxx.xxx.xxx.xxx
[instance:frontend2]<= macro:base-instancesecuritygroups = frontend2 startup_script = frontend-startup.ship = xxx.xxx.xxx.xxx
[instance:proxy]<= macro:base-instancesecuritygroups = proxy startup_script = proxy-startup.ship = xxx.xxx.xxx.xxx
28Sunday, November 7, 2010
[buildout]parts = backend-script frontend-script proxy-script
[backend-script]recipe = collective.recipe.templateinput = deployment/templates/zope-startup-script.shoutput = deployment/backend-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = backend.cfg
[frontend-script]recipe = collective.recipe.templateinput = deployment/templates/zope-startup-script.shoutput = deployment/frontend-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = frontend.cfg
[proxy-script]recipe = collective.recipe.templateinput = deployment/templates/proxy-startup-script.shoutput = deployment/proxy-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = proxy.cfg
29Sunday, November 7, 2010
#!/bin/bashset -e -xexport DEBIAN_FRONTEND=noninteractiveapt-get update && apt-get upgrade -y
echo "----- Install system packages for Plone"apt-get -y install python2.6-dev subversion build-essential
echo "----- Create an unprivileged user zope with password disabled"adduser zope --disabled-password --gecos ""
echo "----- Check out the buildout"cd /home/zopeecho 'p\n' | svn co http://svn.jazkarta.com/.../.../trunk buildout
echo "----- Make the zope user the owner of this directory"chown -R zope:zope buildout
echo "----- Run the buildout as user zope"cd buildoutsudo -u zope python2.6 bootstrap.py -c backend.cfgsudo -u zope ./bin/buildout -v -c backend.cfg
echo "----- Start Zeo processes"sudo -u zope ./bin/supervisordsudo -u zope ./bin/supervisorctl status
echo "----- Make sure that Supervisor starts on reboot"ln -s /home/zope/buildout/etc/supervisor /etc/init.d/supervisorchmod +x /etc/init.d/supervisorupdate-rc.d supervisor defaults
30Sunday, November 7, 2010
[buildout]backend-dns-name = ec2-174-129-192-63.compute-1.amazonaws.comfrontend1-dns-name = ec2-174-129-116-33.compute-1.amazonaws.comfrontend2-dns-name = ec2-184-73-240-104.compute-1.amazonaws.com
Must define the FQDN of the Zeo clients and
Zeo servers
When these names are resolved *on* the server, the DNS server returns the private IP, which means you avoid paying
bandwidth charges for outgoing/incoming traffic.31Sunday, November 7, 2010
$ ./bin/aws start backend$ ./bin/aws start frontend1$ ./bin/aws start frontend2$ ./bin/aws start proxy
Start up the servers with one command
32Sunday, November 7, 2010
Fault tolerantEliminating single points of failure.
33Sunday, November 7, 2010
Plone Conference 2009 – Federico C. Guizzardi – 21 Aprile 2009
Distributed Architecture
Internet
Webserver
Cache Sys
Load Balancer
s1
s3
s4 s5
s2
10
34Sunday, November 7, 2010
02/21/08
The following diagram summarizes this information.
Heartbeat monitors Zeo server and moves to backup in case of failure
DRDBDistributed storage system similar in principle in RAID,but which works over the network.
35Sunday, November 7, 2010
document jazhar_diagrams.graffle
modified 26 Oct 2009 20:51+0100 page 2
logical server
architecture
...
plone-fe-1.seas.harvard.eduplone-fe-0.seas.harvard.edu
plone-zope-0.web.private plone-zope-1.web.private
plone-zeo-0.web.private plone-zeo-1.web.private
Zope Client
ZEO Server ZEO Server
Zope Client Zope Client Zope Client
Apache
Varnish
Deliverance
Pound
browser clients
backend
frontend
internet
standard
connection
failover
connection
Apache
Varnish
Deliverance
Pound Supervisor
Supervisor
• HTTP:8221
• HTTP:8222 • HTTP:8223
• HTTP:8220
• HTTP:8889
• HTTP:9999• HTTP:80
• HTTPS:443
• HTTP:8000
• HTTP:9999
ldap
LDAPSupervisor
• HTTP:9999 • LDAP:389• HTTP:8222 • HTTP:8223
Supervisor
• HTTP:9999
host hostpassive
resource
active
resource
Supervisor
• HTTP:9999
• HTTP:8220
• HTTP:8889
• HTTP:80
• HTTPS:443
• HTTP:8000
36Sunday, November 7, 2010
Improvements to the deployment process
• time to launch a new site was days
• brittle non-repeatable installation process
• delay in procuring hardware and installing OS dependencies
• time to launch a new site was minutes
• streamlined and automated installation process
• virtual machines are launched instantly and OS dependencies are already installed on the disk image.
Before After
37Sunday, November 7, 2010
FabricScripts to deploy to remote hosts.
http://www.fabfile.org 38Sunday, November 7, 2010
Fabricset(fab_user='plone',
fab_hosts=['localhost'],
local='/home/plone/prod/',
remote='/opt/zope/prod/',
site='budapest.buildout')
def deploy():
local('cd $(local)$(site)')
local('tar cvfz $(site).tgz buildout --exclude=.svn --exclude=*.pyc')
run('cd $(remote)$(site); ./bin/supervisorctl shutdown')
run('rm -rf $(remote)$(site)')
put('$(site).tar.gz', '$(remote)$(site).tar.gz')
run('cd $(root)$(site) && tar zxf $(site).tar.gz')
start()
def start():
run('cd $(remote)$(site)')
run('./bin/supervisord')
39Sunday, November 7, 2010
Deploy
$ fab deploy
• Provides a local packaging of the source into a tarball
• Removes the old remotely deployed code
• Puts the source on the remote host
• Restarts the site
What does this one command do?
40Sunday, November 7, 2010
Benefits of Fabric
• Chain local-command, remote-command, and remote-copy commands
• Substitute variables, and the splitting of distinct operations (like in the case of the 'restart' command) so you run them separately, with a fab restart, for example.
41Sunday, November 7, 2010
Example Fabric file
• See Alex Clark’s Fabric file:
• http://github.com/aclark4life/Debian-Deploy-Plone/blob/master/fabfile.py
42Sunday, November 7, 2010
collective.hostoutBuildout-based recipe for deploying to remote host
http://plone.org/products/collective.hostout
43Sunday, November 7, 2010
[buildout]extends = buildout.cfg
parts += hostout prod
auto-checkout += collective.hostout
[sources]
collective.hostout = svn https://svn.plone.org/svn/collective/collective.hostout/trunk
[hostout]recipe = collective.hostout
user = root
#password = blahidentity-file = /Users/nateaune/id_rsa-gsg-keypair
pre-commands =
# ${buildout:directory}/bin/supervisorctl shutdown || echo 'Unable to shutdown'# post-commands =
# ${buildout:directory}/bin/supervisord
#effective-user = zope
[prod]
recipe = collective.hostout
extends = hostouthost = ec2-75-101-211-135.compute-1.amazonaws.com
buildout = buildout.cfg
path = /opt/zope/prod
44Sunday, November 7, 2010
Run the deploy commands
$ ./bin/buildout -NInstalling prod.Generated script '/bin/hostout'.
$ bin/hostout deployInvalid hostout hostouts are: prod
$ bin/hostout deploy prod...
45Sunday, November 7, 2010
Go to Dylan’s talk
• Deploying with collective.hostout with Dylan Jay.
• Today at 16:10-16:55 in Duchess 1+2.
46Sunday, November 7, 2010
Sprint idea
• Make a site to launch demo Plone sites using :
• t1.micro instances (free for 1st year)
• Enfold’s Ubuntu package
• xdv to show multitude of themes
• Could later include bundles of popular add-ons: PloneEdu, multimedia, etc.
47Sunday, November 7, 2010
Questions?
• http://rhaptos.org
• http://pypi.python.org/pypi/mr.awsome
• http://fabfile.org
• http://pypi.python.org/pypi/collective.hostout
•
48Sunday, November 7, 2010