Nested CloudStack with VMwarePaul Angus

Cloud Architect ShapeBlue

paul.angus@shapeblue.com

Twitter: @CloudyAngus

@shapeblue #ccceu14

Why – not just a science project

How

What (you can do with it)

Next steps

Nested CloudStack with VMware

@shapeblue #ccceu14

Cloud Architect with ShapeBlue

Specialise in….

Designing and deploying enterprise and public clouds

Involved with CloudStack before donation to Apache

Designed Clouds Orange, TomTom, PaddyPower, Ascenty, BSkyB

CloudStack Committer

About Me

@shapeblue #ccceu14

“ShapeBlue are expert builders of public & private clouds. They are the leading global

Apache CloudStack integrator & consultancy”

About ShapeBlue

@shapeblue #ccceu14

Nested CloudStack with VMware

Why?

@shapeblue #ccceu14

CEO

Why

“Building CloudStack environments using CloudStack?Are you just having fun?”

@shapeblue #ccceu14

Why

Talented Cloud Architect

“If we can automate the building of environments including the networking, and generate any topology we want quickly, we can achieve the tasks that I’ve highlighted in these slides that I have with me, just in case you asked.”

“Building CloudStack environments using CloudStack?Are you just having fun?”

@shapeblue #ccceu14

Why

CEO“That would be excellent.Go ahead.Oh, and here’s a pay rise”

building of environments including the networking, and generate any topology we want quickly, we can achieve the tasks that I’ve highlighted in these slides that I have with me, just in case you asked.”

@shapeblue #ccceu14

Client:

“We have a large global multi-zone public deployment, which we need toupgrade. Upgrade requires all virtual routers to be restarted. Virtual routerrestarts need to be done in phases (due to time zones). We need to knowwhat the users will experience while CloudStack has been upgraded butthe virtual routers have not be restarted. We also want to know if thelatency between some zones will be an issue during the upgrade.

No, we don’t have a couple of dozen hosts that you can use”

Why?

@shapeblue #ccceu14

Test/Dev

Need to be able to create full environments to test:

CloudStack release candidates

CloudStack features

ShapeBlue patches

Why

@shapeblue #ccceu14

Training

Bootcamp runs in Virtualbox – limited by amount of RAM and cores in laptops

We can’t use KVM or ESXi hosts because no HVM support for guests.

Why?

@shapeblue #ccceu14

Client Demos

Need to create demo environments to show features

Why?

@shapeblue #ccceu14

Nested CloudStack with VMware

How

@shapeblue #ccceu14

Nested virtualisation has been possible in ESXi since 4.1 with some hacking (editing /etc/vmware/config and changing the CPU IDs) if the CPU supported VT-x or AMD-V

Since 5.1 nested virtualisation more powerful with the introduction of support for 2nd generation hardware virtualisation within processors. Extended Page Tables (EPT) from Intel (Nehalem onwards) or Rapid Virtualization (RV) from AMD (0x10 Barcelona onwards)

Still not supported for production loads

Nested Hypervisors

@shapeblue #ccceu14

Nested Hypervisors

Enabling nested virtualisation in CloudStack enables the hardware virtualisation feature.

@shapeblue #ccceu14

Trunking VLANs no problem with vSwitches/bridges on any hypervisor

Portgroup/vSwitch need to be set to promiscuous mode to nest network interfaces.

To create a hypervisor VM purely in CloudStack we need to add a network interface which can use a range of VLANs for the guest VLANs

But within CloudStack you can only map one VLAN to any network

Networking

@shapeblue #ccceu14

On an ESXi host, using VLANID 4095 tells the host to trunk all VLANs through the port group.

Networking

@shapeblue #ccceu14

Trunked Guest port group can share vSwitch with other traffic or be on a dedicated vSwitch

Guest Networks

@shapeblue #ccceu14

Create a shared network on VLAN 4095

Shared Guest Networks

@shapeblue #ccceu14

Each CloudStack zone has a range of guest VLANs.

The range will be created within the nested CloudStack.May need to avoid overlapping VLANs.

Isolation of Shared Networks

vSwitch0

VLANs 100-150

Top Secret Data

VLAN

150

SwitchESXi host

VLANs 100-150

vSwitch0

ESXi host

VLANs 120-200

@shapeblue #ccceu14

Either physically separate the vSwitch

Or do not connect an uplink to the vSwitch

Isolation of Shared Networks

vSwitch1

vSwitch0

VLANs 100-150

Top Secret Data

VLAN 150

SwitchESXi host

VLANs 151-200

@shapeblue #ccceu14

If isolation required:

Create additional vSwitch

Create additional physical guest network mapped to this vSwitch

Multiple guest networks means network offerings must be tagged.

If no isolation:

‘Usual’ guest networking vSwitch OK.

Isolation

@shapeblue #ccceu14

Hypervisor Instance

Guest Traffic- Shared (trunked) network

Public Traffic- Shared network

Management Traffic- Isolated network

Building Environments

@shapeblue #ccceu14

Simple CloudStackarchitecture

Building Environments

Host1

Host2

ACSMan

With NFS MySQL

Isolated Network Dynamic VLAN per tenant

Public Network (Shared) Fixed VLAN (no DHCP)

Gu

est N

etw

ork

(Sh

are

d) T

runke

d

@shapeblue #ccceu14

Nested CloudStack with VMware

What(you can do with it)

@shapeblue #ccceu14

Any hypervisor

Multiple management servers

MySQL master/slave, Galera clusters etc

Separate NFS inc.NetApp ONTAP Edge

Software load balancers

Virtual appliances

Object Storage for sec storage

User portals

Other Infrastructure Components

@shapeblue #ccceu14

Create bare VM / hosts from templates

Install Ansible on guest instance

Git clone mega-playbook

Update hosts and group_vars

Push application

stack to VMs

Building Environments

@shapeblue #ccceu14

Hypervisor Builds ESXi – multiple templates as cold standby

XenServer – deployed from template:

Reset installation_uuid, state.db and control_domain_uuid

Recreate network interfaces and storage devices

KVM – Base OS then push cloudstack-agent

Windows vCenter - multiple templates as cold standby

Building Environments

@shapeblue #ccceu14

Easy to build up complex environments

Building Environments

Host1

Host2

ACSMan1

MySQL

Master

Isolated Network Dynamic VLAN per tenant

Public Network (Shared) Fixed VLAN (no DHCP)

Gu

est N

etw

ork

(Sh

are

d) T

runke

d

NFS

MySQL

Slave

ACSMan1

NetScaler

NetScaler

CSForge

Server

@shapeblue #ccceu14

3 zones

2 geographic locations

Upgrade done, then tests run for a week. Then VRs restarted

Client Test Environment

CCP3.0.7B

MySQL

CPBM 2.2

MySQL

ESXi 1a

ESXi 1b

ESXi 1cvCenter

Appliance

ESXi 2a

ESXi 2b

ESXi 2cvCenter

ApplianceZone 1 (local)

Zone 2 (local)

Zone 3 (remote)

NFS

NFS

VPN VPN

ESXi 2a

ESXi 2b

ESXi 2cvCenter

Appliance

NFS

@shapeblue #ccceu14

Test Dev

Automate the building of entire environments to test releases, features and patches against.

Client Demos

Need to create demo environments to show features

What you can do with it

@shapeblue #ccceu14

Training

Hosted training in environments containing all hypervisors with a full infrastructure.

What you can do with it

@shapeblue #ccceu14

Nested CloudStack with VMware

Next Steps

@shapeblue #ccceu14

Next steps: Automated build of Windows VMs

Bootstrapped build of ESXi hosts.

Using AWS module to provision VMs

CloudStack module

Dynamic inventories

Nested CloudStack with VMware

Nested CloudStack with VMwarePaul Angus

Cloud Architect ShapeBlue

paul.angus@shapeblue.com

Twitter: @CloudyAngus