PuppetConf 2013 vCloud Hybrid Service and Puppet

transcript

Confidential

VMware vCloud Hybrid Service and Puppet Nan Liu, Sr. Systems Engineer - VMware Inc

2 Confidential

Overview

§  VMware vCloud Hybrid Service §  Automation with Puppet §  Lesson’s learned the hard way

3 Confidential

vCloud Hybrid Service

VMware vCloud Hybrid Service

Your Data Center Software-Defined Data Center

VMware vSphere & vCloud Suite

Existing & New Apps

Seamless Networking

Common management

One Support call

4 Confidential

vCloud Hybrid Service (vCHS)

§  Customer:

5 Confidential

vCloud Hybrid Services

§  Engineer:

-------------- 1 ---------------Init VPC- Engine - API Call

------------- 3 --------------Install ESXiEngine Broker Plugin - RazorInputs: - Razor IP - Blade - Hostname - Network InformationTasks: - Create Policy - Boot Blade

------------- 3 --------------Prep TLMEngine Broker Plugin - Puppet vCenterInputs: - TLM VCSA IP - TLM VCSA Creds - TLM DVS - Portgroup Name - Portgroup VLAN - Portgroup Settings - Ports: 128 - Load Balancing: IP Hash - Reset at Disconnect: EnabledTasks: - Create PGs - d#p#v#-dmz-pg-## - d#p#v#-esx-pg-## - d#p#v#-pvt-pg-##

----------- 4 ------------Deploy VPC VCSAEngine Broker Plugin - OVFToolInputs: - OVA - TLM vCenter IP - TLM vCenter Creds - Cluster - Datastore - VM Name - Network - VM Hardware Settings - Memory: 16384Tasks: - Deploy VPC VCSA - d#p#v#-mgmt-vc0 - Update VM Hardware

----------- 4 ------------Deploy VPC VSE Pub PairEngine Broker Plugin - Puppet vShieldInputs: - TLM VSM IP - TLM VSM Creds - Edge Settings - Name - Hostname - Enable HA - Declare Dead Time: 6 - Interface 1 - Heartbeat IP Addresses - TLM Datacenter - TLM Cluster - TLM Shared Datastore - Size: Compact - Interface Settings - Interface 0 - Name - Type - Portgroup - Subnet Settings - Interface 1 - Name - Type - Portgroup - Subnet SettingsTasks: - Create VPC Pub Edge - d#p#v#-mgmt-vse-pub

----------- 4 ------------Deploy VPC VSE Priv PairEngine Broker Plugin - Puppet VMware-vShieldInputs: - TLM VSM IP - TLM VSM Creds - Edge Settings - Name - Hostname - Enable HA - Declare Dead Time: 6 - Interface 2 - Heartbeat IP Addresses - TLM Datacenter - TLM Cluster - TLM Shared Datastore - Size: Compact - Interface Settings - Interface 0 - Name: InterVPC - Type - Portgroup - Subnet Settings - Interface 1 - Name: DMZ - Type - Portgroup - Subnet Settings - Interface 2 - Name: PVT - Type - Portgroup - Subnet Settings - Interface 3 - Name: ESX - Type - Portgroup - Subnet SettingsTasks: - Create VPC Priv Edge - d#p#v#-mgmt-vse-priv

----------- 5 ------------Configure VPC VSE Pub PairEngine Broker Plugin - Puppet vShieldInputs: - TLM VSM IP - TLM VSM Creds - Edge Settings - DNS Settings - Firewall Settings - IPsets - Application Groups - Applications - Firewall Rules - Load Balancer Settings - Enable - Pools - VIPs - Default Route - SyslogTasks: - Configure VPC VSE Pub Pair

----------- 5 ------------Configure VPC VSE Priv PairEngine Broker Plugin - Puppet VMware-vShieldInputs: - TLM VSM IP - TLM VSM Creds - Edge Settings: - Syslog - DNS - Firewall Settings - IPsets - Application Groups - Applications - Firewall Rules - Load Balancer Settings - Enable - Pools - VIPs - Default RouteTasks: - Configure VPC VSE Priv Pair

----------- 7 ------------Init VPC VCSAEngine Broker Plugin - Puppet VMware-VCSAInputs: - Credentials - NTP Settings - Syslog Settings - DB Settings - Sizing: LargeTasks: - Init VPC VCSA

----------- 4 ------------Deploy VPC VSMEngine Broker Plugin - OVFToolInputs: - OVA - TLM vCenter IP - TLM vCenter Creds - Cluster - Datastore - VM Name - NetworkTasks: - Deploy VPC VSM - d#p#v#-mgmt-vsm0

----------- 9 ------------Configure VPC VSMEngine Broker Plugin - Puppet VMware-vShieldInputs: - VPC VSM IP - VPC VSM Creds - VPC VCSA IP - VPC VCSA Creds - NTP - SyslogTasks: - Configure VPC VSM

----------- 4 ------------Deploy VPC vCloudEngine Broker Plugin - ??Inputs: - TLM vCenter IP - TLM vCenter Creds - Cluster - Datastore - VM Names - VM Settings - NetworkTasks: - Deploy VPC vCloud Cells - Deploy VPC vCloud NFS - Deploy VPC vCloud DB

-------------- 2 -----------------Reserve VPCEngine Broker Plugin: - RezInputs: - VPC #

----------- 8 ------------Configure VPC vCenterEngine Broker Plugin - Puppet VMware-vCenterInputs: - VPC VCSA IP Address - VPC VCSA Credentials - Licenses - vCenter - ESXi - vCloud Net & Sec - Retention Policies - Task: Enabled - Event: Enabled - Datacenter Name - Cluster Settings - Name - DRS Settings - EVC Settings? - DVS Configuration - Name: d#p#v# - Settings - Uplinks: 4 - MTU: 9000 - Enable NIOC - Portgroup settings - Name - VLAN - Settings-Tasks: - Configure VPC VCSA - Configure DVS

----------- 7 ------------Init VPC VSMEngine Broker Plugin - ??Inputs: - VPC VSM Network SettingsTasks: - Init VPC VSM

----------- 6 ------------VPC VCSA Port CheckEngine Broker Plugin - UtilInputs: - VPC VCSA IPTasks: - VPC VCSA Port Check

----------- 6 ------------VPC VSM Port CheckEngine Broker Plugin - UtilInputs: - VPC VSM IPTasks: - VPC VSM Port Check

----------- 6 ------------Verify VPC ESXiEngine Broker Plugin - Puppet Util?Inputs: - VPC ESXi IP - VPC ESXii CredsTasks: - Verify VPC ESXi

----------- 9 ------------Add ESXi to VPC DatacenterEngine Broker Plugin - Puppet VMware-vCenterInputs: - VPC ESXi IP - VPC ESXii Creds - VPC VCSA IP - VPC VCSA Creds - VPC Datacenter NameTasks: - Add ESXi to VPC Datacenter

----------- 10 ------------Configure VPC ESXiEngine Broker Plugin - Puppet VMware-vCenterInputs: - VPC VCSA IP - VPC VCSA Creds - DNS Settings - NTP Settings - Syslog Settings - VMK Configuration - vMotion - Storage - FT - ???Tasks: - Configure ESXi - Add to DVS - Add VMK Networking

----------- 11 ------------Add Storage to VPC ESXiEngine Broker Plugin - Puppet VMware-vCenterInputs: - VPC VCSA IP - VPC VCSA Creds - iSCSI Settings - LUN InformationTasks: - Configure iSCSI - Add LUNs

------------- 3 ---------------Configure VNX VPC Storage GroupEngine Broker Plugin: - Storage ControllerInputs: - VPC # - LUN IDs - ESXi iSCSI informationTasks: - Create VPC Storage Group - Configure VNX Host registrations - Add LUNs to VPC Storage Group

----------- 6 ------------VPC vCloud Cell Port CheckEngine Broker Plugin - UtilInputs: - VPC vCloud Cell IPTasks: - VPC vCloud Cell Port Check

----------- 6 ------------VPC vCloud NFS Port CheckEngine Broker Plugin - UtilInputs: - VPC vCloud Cell IPTasks: - VPC vCloud NFS Port Check

----------- 6 ------------VPC vCloud DB Port CheckEngine Broker Plugin - UtilInputs: - VPC vCloud Cell IPTasks: - VPC vCloud DB Port Check

----------- 7 ------------Configure VPC vCloud NFSEngine Broker Plugin - ??Inputs: - VPC vCloud NFS IP - VPC vCloud NFS Creds - NFS Export Settings - ???Tasks: - Config VPC vCloud NFS

----------- 7 ------------Configure VPC vCloud DBEngine Broker Plugin - ??Inputs: - VPC vCloud DB IP - VPC vCloud DB Creds - vCloud Database Config - ???Tasks: - Configure VPC vCloud DB

----------- 8 ------------Configure VPC vCloud CellEngine Broker Plugin - ??Inputs: - VPC vCloud Cell IP - VPC vCloud Cell Creds - VPC vCloud NFS Config - VPC vCloud Installation Responses.properties - VPC vCloud Cell Cert - NTP SettingsTasks: - Install and Configure NTP - Install VPC vCloud Cert - Configure vCD - Configure vCD Transfer Service

----------- 9 ------------Configure VPC vCloud CellEngine Broker Plugin - ??Inputs: - VPC vCloud Cell IP - VPC vCloud Cell Creds - VPC vCloud Installation Responses.properties - VPC vCloud NFS Config - VPC vCloud Cell Cert - NTP SettingsTasks: - Install and Configure NTP - Install VPC vCloud Cert - Configure vCD - Configure vCD Transfer Service

----------- 12 ------------Create VPC VM Storage ProfileEngine Broker Plugin - Puppet VMware-vCenterInputs: - VPC VCSA IP - VPC VCSA Creds - Storage Profile NameTasks: - Create VPC VM Storage Profile - Tag VPC Datastores

----------- 13 ------------Add VPC vCenter to vCloudEngine Broker Plugin - Puppet VMware-vCloudInputs: - VPC vCloud IP - VPC vCloud Creds - VPC VCSA IP - VPC VCSA Creds - ???Tasks: - Add VPC vCenter to VPC vCloud

----------- 10 ------------Configure VXLANEngine Broker Plugin - Puppet VMware-vShieldInputs: - VPC VSM IP - VPC VSM Creds - Multicast Information - VPC DVS Information - VPC Cluster - ???Tasks: - Configure VXLAN

----------- 11 ------------Reconfigure VXLAN VMK PortsEngine Broker Plugin - Puppet VMware-vCenterInputs: - VPC VCSA IP - VPC VCSA Creds - VXLAN VMK Network SettingsTasks: - Configure VXLAN VMK Ports

----------- 14 ------------Configure VPC vCloudEngine Broker Plugin - Puppet VMware-vCloudInputs: - VPC vCloud IP - VPC vCloud Creds - Provider VDC Name - External Org Network Name - ???Tasks: - Configure VPC vCloud

----------- 12 ------------Configure VPC Cluster HAEngine Broker Plugin - Puppet VMware-vCenterInputs: - VPC VCSA IP - VPC VCSA Creds - VPC ESXi Hosts - VPC Cluster - HA Settings - Failure: percentageTasks: - Add VPC ESXi hosts to Cluster- Configure VPC Cluster HA Settings

----------- 5 ------------Prep VPC vCloud CellEngine Broker Plugin - Puppet vCenter - ??Inputs: - VPC VCSA IP - VPC VCSA Cred - VPC vCloud Cell Name - Static RoutesTasks: - PowerOn VM - Set Hostname - Set Static Routes - Configure Networking

----------- 5 ------------Prep VPC VCSAEngine Broker Plugin - ??Inputs: - Credentials - Network SettingsTasks: - Configure Networking

----------- 5 ------------Prep VPC vCloud CellEngine Broker Plugin - Puppet vCenter - ??Inputs: - VPC VCSA IP - VPC VCSA Cred - VPC vCloud Cell Name - Static RoutesTasks: - PowerOn VM - Set Hostname - Set Static Routes - Configure Networking

----------- 5 ------------Prep VPC VSMEngine Broker Plugin - ??Inputs: - VPC VCSA IP - VPC VCSA Cred - VPC VSM name - Network SettingsTasks: - PowerOn VM - Configure Networking

6 Confidential

vCloud Hybrid Service

§  Start your engine:

7 Confidential

Click Fail

Add VPC Hosts to VCSA §  Continuing from the vCenter view of the vSphere Web Client §  In the main content pane, click Related Objects > Clusters §  Click on VPC on the list §  Click the second plus icon to add a host

• Host name: Enter first host FQDN •  Location: Confirm the location is set.

• Click NEXT

§ … §  Repeat Manually?

8 Confidential

Problem

Challenges: §  Reduce deployment time §  Reduce complexity §  Scale … Fast Solution: 1.  Automate 2.  See first rule

9 Confidential

vCloud Director

10 Confidential

Infrastructure Services

Server & OS Services (Puppet) §  NTP §  Syslog §  RabbitMQ § …

11 Confidential

Automation Challenges

§  Service APIs: •  vCenter: vSphere API (soap) •  vCNS: vShield API (REST)

•  vCD: vCD API (REST)

§  vCenter/vCNS servers are appliances

12 Confidential

Why not ‘puppet device’?

§  Limited to one device at a time:

Puppet Management Server

VMware vCenter

VMware vShield

Datacenter

ESXESXESX

Datacenter

ESXESX

vSphere API

vShield API

13 Confidential

Solution: Take the road less traveled

§  Everything is a Native Resource §  Transport to the rescue

•  vCenter/vCNS Appliance: SSH •  vCenter API: RbVmomi

•  vShield API: rest-client + subset of savon project •  vCD API: rest-client

14 Confidential

Transport Resource

§  Credentials §  Connectivity Options § Multiple connections

15 Confidential

Transport Example

16 Confidential

Transport

VMware vCenterSSH

VMware vCentervSphere API

Datacenter

Folders

Datacenter

17 Confidential

vCenter Appliance Resource

18 Confidential

vCenter Resource

19 Confidential

vCD Resources

20 Confidential

Transport

§  Persistent shared connection §  Connection cleanup after catalog apply § Open to supporting additional transport

•  VMware-RabbitMQ (REST)

21 Confidential

Modules

Puppet Enterprise Users (PE 2.7): §  http://forge.puppetlabs.com/vmware

Puppet Developers (Developing Puppet 3): §  http://github.com/vmware/vmware-vmware_lib §  http://github.com/vmware/vmware-vcsa §  http://github.com/vmware/vmware-vcenter §  http://github.com/vmware/vmware-vshield

22 Confidential

Lessons Learned

§ Working with APIs §  Puppet 2.7.x -> Puppet 3.x §  Puppet Wat?

23 Confidential

Working with APIs

§  Functionalities not always in API* §  Dealing with API versions § Metaprogramming * Even if you are @lamw :)

24 Confidential

Dealing with outliers

§  Accept work around ssh.exec( " esxcfg-vmknic -i #{opts[:new_mgmt_ip]} -n #{opts[:new_mgmt_mask]} -p 'Management Network' && esxcfg-route -a default #{opts[:new_mgmt_gw]} && esxcfg-vswitch -p 'Management Network' -v #{opts[:new_mgmt_vlan]} vSwitch0 " )

§  Last resort, file a ticket t = ServiceNow::Request.new( :subject => ”Please click yes" ) ServiceNow.create(t)

25 Confidential

Testing API versions

§  puppet apply --libdir=/dev/null §  bundler exec + Gemfile source 'file:///opt/repo/’ source 'https://rubygems.org’ gem 'facter’ gem 'puppet', '3.2.2’ gem 'gyoku', '1.0.0z2’ gem 'hashdiff’ gem 'net-ssh’ gem 'nokogiri’ gem 'nori', '1.1.4’ gem 'rbvmomi', '1.6.0.z1’ gem 'rest-client’ gem 'pry'

26 Confidential

Metaprogramming

Good §  Reduces boiler plate code §  No more API ‘transfer’ bugs §  API reference = Resource reference Bad §  Difficult to debug §  Fixing API issues results in surprises §  Not all APIs are designed to be idempotent

•  action => { :create, :modify }

27 Confidential

Puppet 3 Upgrade

§ What the scope? § Where’s my HOME? §  Ruby 1.9.3

28 Confidential

What the scope?

§  Fully qualify ::class::var §  Fix your template @var, scope.lookup(‘::class::var’)

29 Confidential

Where’s my HOME?

§  Exec specify HOME. §  Providers Fix:

if respond_to? :has_command has_command(:brew, "/usr/local/bin/brew") do environment({ 'HOME' => ENV['HOME'] }) end else commands :brew => "/usr/local/bin/brew” end

30 Confidential

Puppet Wat

Boolean: §  adrientthebo/boolean §  vmware_lib property §  Symbool in Hash

What the undef? §  Careful about behavior

ENC Data §  ENC integer .to_s

31 Confidential

Where we are Today

§  Deploy VPC: # §  95% reduction in deployment time §  Configuration Management = Version

§ Metrics: •  47 Modules •  70 Custom Resources

•  1400 Resources

32 Confidential

Work at VMware

Challenges: §  Software Defined Datacenter

• Data driven configuration management

§  Software Defined Networking •  vCNS, Nicira VSX

§  Scale + Speed •  10X, 100X, 1000X ? •  Faster !!! Now !!!

33 Confidential

Thanks!

§  Nicholas Weaver §  Randy Brown §  Shawn Holland §  Floyd Arguello §  David Scherer §  Ryan Zenker §  Justin Guidroz §  Dan Pittman §  Branan Purvine-Riley §  Zach Leslie §  vCHS R&D Team

34 Confidential

PuppetConf 2013 vCloud Hybrid Service and Puppet

Technology