www.shapeblue.com
Building CloudStack at Scale
Paul AngusCloud Architect ShapeBlue
[email protected]: @ShapeBlue
www.shapeblue.com
Overview
The design phaseRedundancy & resilienceAutomationDocumentation
Building CloudStack at Scale
www.shapeblue.com
The Design Phase
Tips for the design phase
www.shapeblue.com
Get everyone in the room
The Design Phase
www.shapeblue.com
Know the fine print
The Design Phase
www.shapeblue.com
Understand your work loads
The Design Phase
www.shapeblue.com
Understand your bottlenecks
Design Phase
www.shapeblue.com
Redundancy & Resilience
Designing for Redundancy and
Resiliency
www.shapeblue.com
CloudStack Management Server
MySQL Servers
Virtualised Management Farm
Redundancy & Resilience
www.shapeblue.com
CloudStack Management Server
Very easy to set up additional management servers
Load balancing required to give high availabilty
Redundancy & Resilience
www.shapeblue.com
MySQL
Master / Slave is ‘standard’
Alternatives includeMySQL Proxy (Mirroring)Galera ClusterMMM
Redundancy & Resilience
www.shapeblue.com
Redundancy & Resilience
Load Balance
r
MySQL
Galera
MySQL
MySQL
Load Balancer
CS Man
CS Man
www.shapeblue.com
Redundancy & Resilience
F5
MySQL
Galera
DC1 DC2
MySQL
MySQL
CS Man
CS Man
F5
F5
MySQL
Galera
MySQL
MySQL
CS Man
CS Man
F5
www.shapeblue.com
Server ‘pairs’
MySQL masters and slavesCS Man & MySQL mastervCenter & MS SQL serverAny other redundant servers i.e. DNS
Redundancy & Resilience
www.shapeblue.com
Laws of probability….
Same failure probability as raid 0If management server OR MySQLmaster fails – downtime for the whole management system.
Redundancy & Resilience
www.shapeblue.com
Automation
Automation of the infrastructure build
www.shapeblue.com
Uses:Why automateAutomation toolsExamples
Configuring management VMsBuild a CloudStack 4.0 management serverDeployment of hostsConfiguration of CloudStack (through API)
Automation
www.shapeblue.com
Work smarter, not harder
Automation
www.shapeblue.com
Automation gives you:
SpeedEfficiencyRepeatabilityAccuracy
Automation
www.shapeblue.com
Tools:
Chef, Puppet
KickStart, Python
Shell scripting
Automation
www.shapeblue.com
Example:
Automation
Base Build of Management Servers using Shell
Scripting
www.shapeblue.com
Automation – Management VM Configuration
Create Deployment VM
CentOS (Minimal) + wget
Download CSV & hostconfig script
Make it executable,run it.
www.shapeblue.com
The CSV:
Automation – Management VM Configuration
VMhostname Searchdomain PrimaryNICIPPrimaryNIC
PrefixPrimaryNICG
atewayPrimaryNIC
Network SecondaryNICIPSecondaryNIC
PrefixSecondaryNIC
Network DNS1NTPSERVERIP
1
CSMan1 cs4.net 10.14.16.67 27 10.14.16.65 Tools 10.144.72.21 21 Mgmt 10.128.9.9 64.238.19.20
CSMan2 cs4.net 10.14.16.68 27 10.14.16.65 Tools 10.144.72.22 21 Mgmt 10.128.9.9 64.238.19.20
MySQL1 cs4.net 10.14.1.35 27 10.14.1.33 Database 10.141.163.4 26 Mgmt 10.128.9.9 64.238.19.20
MySQL2 cs4.net 10.14.1.36 27 10.14.1.33 Database 10.141.163.5 26 Mgmt 10.128.9.9 64.238.19.20
www.shapeblue.com
#!/bin/bash
echo "Please enter hostname [without domain]: ie CSMan1"
read VMHOSTNAME
SearchDomain=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$2}'`
PrimaryNICIP=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$3}'`
PrimaryNICIP=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$3}'`
PrimaryNICPrefix=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$4}'`
PrimaryNICGateway=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$5}'`
PrimaryNICNetwork=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$6}'`
Automation – Management VM Configuration
www.shapeblue.com
echo "DEVICE=\"eth0\"BOOTPROTO=noneNM_CONTROLLED=\"no\"ONBOOT=yesTYPE=\"Ethernet\"IPADDR=$PrimaryNICIPPREFIX=$PrimaryNICPrefixGATEWAY=$PrimaryNICGatewayDEFROUTE=yesIPV4_FAILURE_FATAL=yesIPV6INIT=noNAME=\"$PrimaryNICNetwork\" "> /etc/sysconfig/network-scripts/ifcfg-eth0
echo "# ShapeBlue Script created file
search $SearchDomain
nameserver $DNS1
nameserver $DNS2
" > /etc/resolv.conf
echo "# ShapeBlue Script created file
NETWORKING=yes
HOSTNAME=$VMHOSTNAME
" > /etc/sysconfig/network
echo "$NTPSERVERIP1
$NTPSERVERIP2" > /etc/ntp.conf
Reboot
Automation – Management VM Configuration
www.shapeblue.com
# if vm management use this one
if [ "$SecondaryNICNetwork" == “Mgmt" ]; then
echo "ADDRESS0=10.128.9.9
NETMASK0=255.255.255.255
GATEWAY0=10.14.16.1
ADDRESS1=10.128.3.13
NETMASK1=255.255.255.255
GATEWAY1=10.14.16.1
ADDRESS2=64.238.199.202
NETMASK2=255.255.255.255
GATEWAY2=10.141.163.1
ADDRESS3=213.212.65.202
NETMASK3=255.255.255.255
GATEWAY3=10.141.163.1
ADDRESS4=213.212.68.168
NETMASK4=255.255.255.248
GATEWAY4=10.141.163.1
ADDRESS5=213.212.69.0
NETMASK5=255.255.255.0 GATEWAY5=10.141.163.1" > /etc/sysconfig/network- scripts/route-eth1
fi
Automation – Management VM Configuration
www.shapeblue.com
Example:
Automation
Build a CS4 management server
www.shapeblue.com
Automation – CS4 Management Server
Create Deployment VM
CentOS (Minimal) + wget
Set static IP address and ensure it has an
FQDN
Download the script, make it executable,
run it.
www.shapeblue.com
#!/bin/bash
setenforce permissive
sed -i "/SELINUX=/ c\SELINUX=permissive" /etc/selinux/config
echo "
[cloudstack]
name=cloudstack
baseurl=http://cloudstack.apt-get.eu/rhel/4.0/
enabled=1
gpgcheck=0" > /etc/yum.repos.d/cloudstack.repo
yum update -y
yum install ntp cloud-client mysql-server -y
sed -i -e '/datadir/ a\innodb_rollback_on_timeout=1' -e '/datadir/ a\innodb_lock_wait_timeout=600' -e '/datadir/ a\max_connections=350' -e '/datadir/ a\log-bin=mysql-bin' -e "/datadir/ a\binlog-format = 'ROW'" /etc/my.cnf
chkconfig ntpd on
chkconfig mysqld on
chkconfig nfs on
chkconfig rpcbind on
service ntpd restart
service mysqld restart
service rpcbind start
service nfs start
/usr/bin/mysqladmin -u root password 'password'
cloud-setup-databases cloud:cloud@localhost --deploy-as=root:password
cloud-setup-management
Automation – CS4 Management Server
www.shapeblue.com
Example:
Automation
Host deployment server build using shell scipting
www.shapeblue.com
Create Deployment VM
CentOS (Minimal) + wget
Download to VM Hypervisor installation media (inc. XenServer
Updates)
Download build script,make it executable,
run it.
Automation – Host Deployment
www.shapeblue.com
Deployment server (VM)
The script downloads, builds and configures:DHCPPXE (TFTP)HTTP server
Script writes the scripts needed to for PXE boot of XenServer & ESXi hostsScript also writes the answer files and post installation scripts to configure XenServer and ESXi hosts
Automation – Host Deployment
www.shapeblue.com
Automation – Host Deployment# write DHCP config file
echo "# DHCP for PXE
ddns-update-style interim;authoritative;log-facility local7;default-lease-time 600;max-lease-time 7200;option subnet-mask $DHCPNETMASK;option broadcast-address $DHCPBROADCAST;option routers $DHCPGATEWAY;option domain-name-servers $DNS1;option domain-name \"$DOMAIN\";
subnet $DHCPSUBNET netmask $DHCPMASK {range $DHCPSTART $DHCPFINISH; }
allow booting;allow bootp;use-host-decl-names true;option option-128 code 128 = string;option option-129 code 129 = text;option space gpxe;option gpxe-encap-opts code 175 = encapsulate gpxe;option gpxe.bus-id code 177 = string;
next-server $PXESERVERIP;filename \"/gpxelinux.0\";
" > /etc/dhcp/dhcpd.conf
www.shapeblue.com
# create pxelinux.cfg/default file
echo "# script built pxeconfig
DEFAULT menu.c32
menu title ShapeBlue Host Deployment
timeout 0
LABEL ESXi5.0
KERNEL http://$PXESERVERIP/esxi/mboot.c32
APPEND -c http://$PXESERVERIP/esxipxeboot/boot.cfg pxebooting +++
IPAPPEND 1
MENU LABEL ESXi-5.0.0
" > /tftpboot/pxelinux.cfg/default
Automation – Host Deployment
www.shapeblue.com
# Create xenserver answerfile
echo "<?xml version=\"1.0\"?>
<installation srtype=\"ext\">
<primary-disk gueststorage=\"yes\">sda</primary-disk>
<keymap>uk</keymap>
<root-password>P@ssword</root-password>
<source type=\"url\">http://$PXESERVERIP/xenserver/</source>
<script stage=\"filesystem-populated\" type=\"url\">http://$PXESERVERIP/xenpxeboot/postxeninstall.sh</script>
<admin-interface name=\"eth0\" proto=\"dhcp\" />
<timezone>Europe/Dublin</timezone>
<time-config-method>ntp</time-config-method>
<ntpservers>$NTPSERVER1</ntpservers>
<nameserver>$DNS1</nameserver>
</installation>
" > /var/www/html/xenpxeboot/xenanswerfile
Automation – Host Deployment
www.shapeblue.com
MAC IP HOSTNAME STORAGEIP
80:c1:6e:6a:b5:9c 10.14.73.9 XenHost1 10.14.97.9
80:c1:6e:6a:b4:0c 10.14.73.10 XenHost2 10.14.97.10
80:c1:6e:6a:f3:fc 10.14.73.11 XenHost3 10.14.97.11
80:c1:6e:6a:f0:ec 10.14.73.12 ESXiHost1 10.14.97.12
80:c1:6e:6a:b5:a4 10.14.73.13 ESXiHost2 10.14.97.13
Automation – Host Deployment
www.shapeblue.com
echo “# sample XenServer post install script
wget http://$PXESERVERIP/hostconfig.csv
MACADD=\`ifconfig | grep eth0 | awk '{print \$NF}'\`“
IPADDR=\’grep -i \$MACADD /tmp/arp_table.csv | awk -F ',' '{print \$2}‘\’
XENHOSTNAME =/`grep -i \$MACADD /tmp/arp_table.csv | awk -F ',' '{print \$3}‘/`
STORAGEIP=/`grep -i \$MACADD /tmp/arp_table.csv | awk -F ',' '{print \$4}‘/`
STORAGENICUUID=/’xe pif-list | awk '\$0 ~ str{print b}{b=\$0}' str="eth3" | awk '{print \$NF}‘/`
xe host-list params=uuid --minimal > /tmp/tmphostuuid
xe pif-list management=true params=uuid --minimal > /tmp/tmpnicuuid
read NICUUID < /tmp/tmpnicuuidread HOSTUUID < /tmp/tmphostuuid
xe host-set-hostname-live host-uuid=\$HOSTUUID host-name=\$XENHOSTNAME
xe host-param-set name-label=\$XENHOSTNAME uuid=\$HOSTUUID
xe pif-reconfigure-ip uuid=\$NICUUID mode=static IP=\$IPADDR netmask=$HOSTNETMASK gateway=$HOSTGATEWAY DNS=$DNS1,$DNS2
xe pif-param-set uuid=\$NICUUID other-config:domain=$DOMAIN
" > /var/www/html/xenpxeboot/firstfullbootxenscript.sh
Automation – Host Deployment
www.shapeblue.com
echo "#!/bin/bash
# Install XenServer Updates
xe host-list params=uuid --minimal > /tmp/tmphostuuid
files=\$(ls /tmp/*.xsupdate 2> /dev/null | wc -l)
if [ \"\$files\" != \"0\" ]; then
read HOSTUUID < /tmp/tmphostuuid
cd /tmp
for updatefile in /tmp/*.xsupdate; do
PATCHUUID=\$(xe patch-upload file-name=\$updatefile)
xe patch-apply host-uuid=\$HOSTUUID uuid=\$PATCHUUID
rm -f \$updatefile
PATCHACTION=\$(xe patch-list uuid=\$PATCHUUID params=after-apply-guidance --minimal)
echo \"Patch Action:\" \$PATCHACTION
if [ \"\$PATCHACTION\" == \"restartXAPI\" ]; then
xe-toolstack-restart
else
reboot
exit
fi
done
fi
# Disable boot script for subsequent reboots
rm -f /etc/rc3.d/S99postinstall
# Final Reboot
reboot
" > /var/www/html/xenpxeboot/xenupdatescript.sh
Automation – Host Deployment
www.shapeblue.com
Automation
Utilising the API when building the infrastructure
www.shapeblue.com
The API
What it is
How can you use it
Examples
Automation
www.shapeblue.com
What is the API
The API is the real engine of CloudStack
The web GUI is simply making API calls
Port 8096 by default
Automation – The API
www.shapeblue.com
Using the API
CloudStack GUIBrowserWord, ExcelUsing Firebug/IE Developer Tools with CloudStack
Automation – The API
www.shapeblue.com
API commands directly through a browser
Immediate response
Automation – The API
www.shapeblue.com
API calls from a Word document or Excel spread sheet
Automation – The API
www.shapeblue.com
Using Firebug / IE Developer Tools
IE press F12 or ‘view developer tools’
Firefox install Firebug add-on
Automation – The API
www.shapeblue.com
www.shapeblue.com
Global Settings
http://csman:8096/client/api?command=updateConfiguration&name=vmware.management.portgroup&value=svc-console
http://csman:8096/client/api?command=updateConfiguration&name=allow.user.create.projects&value=false
http://csman:8096/client/api?command=updateConfiguration&name=allow.public.user.templates&value=false
http://csman:8096/client/api?command=updateConfiguration&name=apply.allocation.algorithm.to.pods&value=true
http://csman:8096/client/api?command=updateConfiguration&name=cpu.overprovisioning.factor&value=2
http://csman:8096/client/api?command=updateConfiguration&name=vm.allocation.algorithm&value=random
Automation – The API
www.shapeblue.com
Getting informationhttp://csman1:8096/client/api?command=listServiceOfferings
http://csman1:8096/client/api?command=listTemplates&templatefilter=featured
Automation – The API
www.shapeblue.com
Deploying an instance
Base command:http://csman1:8096/client/api?command=deployVirtualMachine
The required options:
The Service Offering and Template IDs&serviceofferingid=XXX&templateid=XXX&zoneid=XXX&domainid=XXX&account=XXX
Optional options:&displayname=xxx
Automation – The API
www.shapeblue.com
Deploying an instance
http://csman1:8096/client/api?command=deployVirtualMachine&serviceofferingid=XXX&templateid=XXX
Can be used to create a large number of instances very quickly
Automation – The API
www.shapeblue.com
Documentation
A word on documentation
www.shapeblue.com
Dull, boring, tedious, slow – Crucial.
Write what you’re going to doFollow what you wroteUpdate itWith redundant servers; follow it againWith scripts ‘snapshot’ and start again
Documentation
www.shapeblue.com
Thank you
Paul