WHITEPAPER

DEVOPS – VMWARE

CLOUD INSTANCE

DEPLOYMENT

A Python Implementation

2

O V E R V I E W

This white paper details the implementation of DevOps to deploy VM instances from templates, all the way through till the final form of the VM integrated with networking and settings.

The white paper is useful in the scenarios where requirement is to automate the process of VM deployment/modification tasks, depending on how various components are configured in a VMware Cloud.

3

T A B L E O F C O N T E N T S

Introduction 4

The Need for DevOps 5

Behind The Scenes 6

Developing The First Script 8

Analysis & Conclusion 12

4

I N T R O D U C T I O N

As the need of infrastructure increased in the past years, the IT world

inclined itself towards On-premise/public clouds in order to reduce the

overhead of maintaining the hardware inventory. The engineers became

more competent in terms of deploying virtual machines and

applications. However, the role of a System Administrator can never be

taken away. It is one key element that will remain forever as long as

there is IT. In order to reduce, the workload for deployment of

environment, DevOps emerged. It took the world by storm where the

very idea of Cloud Automation with the usual scripting languages and

then later bigger tools is promising in terms of optimization.

DevOps is one term which does not has a concrete definition in the IT

world yet. The uses and implementations differ at such huge extents

which a single definition cannot contain. For a layman, this is how

DevOps would be.

“DevOps is the way to optimize a process”

5

T H E N E E D F O R D E V O P S It first started when the cloud arrived. With “Pay as you go” model, industries could spend more. Hence more machines, more storage, more scalability etc. were the need of the hour. Although cloud answered the question of not maintaining your own hardware, a virtual machine is ultimate a machine. Configurations still needed to be done on the virtual machines manually which brought up the below challenges.

1.Time taken for mass deployments: Since the configurational parameters still needed to be fed in, a lot of time was invested in configuring thousands of VMs in a single go. This is applicable for development, testing and productions scenarios.

2.Manual error: During deployments of the same kind, there was a huge chance of manual errors. Imagine while deploying VMs if one puts include www for a FQDN field in a production environment.

3.Lack of artificial intelligence: Although cloud offered enormous number of solutions, but there needed to something that is end user driven and can help with the deployments of a specific configuration. In other words, not just what the cloud offers. But introducing capabilities based on defined logic.

DevOps offers itself as a solution to the above in the various ways we’re going to discuss here with the implementation of Vsphere Cloud VM deployment automation.

6

B E H I N D T H E S C E N E

The main purpose of DevOps is to optimize the current process. In order to do so, it is very important to understand:

What is being done manually? What calls are made in the operations performed? What happens in the background?

We’ll talk about VMware Cloud VM Deployment solution here. The components of primary use are:

1. Access/Login:

It has to be made sure that the connection made using a set of credentials has necessary rights for VM deployment. There are a lot of references made in the background by VMware when a VM is created. Without the access to these background functionalities, the provisions will fail. In simple words, we need a set of credentials which can create VMs.

2. Locating and Identifying the correct resources: Among the various components of VMware infrastructure, we need to make sure the object in need is identified correctly. Say, one wants to deploy a VM where a host cluster is present. Or one has a DataStore Cluster with Storage DRS enabled. In such cases, it is important to identify what to select. These features are enabled to support High Availability and Failover. If the automation does not make use of these features, we’re bypassing the manual workflow which is not advised. It can be related to the frustration of a system administrator, where the users are selecting individual nodes when they are advised to select the Cluster for best placement.

3. Managed Object References: These are the system references for each and every single entity in a VMware infrastructure. The objects should be identified using these references as they act like primary keys if VMware was a database with its objects as it contents. Moreover, for every task in VMware, these references are used in order to complete actions

7

as VMware understands not the names that we gave, but the references for those names.

4. pyVMomi/pysphere: pyVMomi is a python module developed to automate the VMware tasks. It lies at the lower level and is complex in terms of automation. Pysphere is a wrapper written of pyVMomi which understands the base and provides better automation scripting hands

8

D E V E L O P I N G T H E F I R S T S C R I P T

We’re now going to look into how to develop a basic script for deploying a VMware VM.

Step #1: Connecting to Vsphere: This is pretty much about the connection made to the ESX host. What’s tricky is the SSL certificate error that might be thrown. If a user has the certificates, the connection can be made without any issue. However, to bypass the warning, below SSL certificate code can be used. default_context = ssl._create_default_https_context ssl._create_default_https_context = ssl._create_unverified_context server_instance = pysphere.VIServer(); server_instance.connect(host,username,password) Here server_instance is the object created for connection to the ESX host using the pysphere module. This is used throughout the deployment for resource identification and implementation.

Step #2: Locating the objects required For a VM to be provisioned information about template to used, resource pool to be allocated, host to be used, datastore to be used for VM hosting, folder to place the VM etc. are required. These are located by Managed Object References which pysphere has the capability to identify. For locating the template as template is treated as a VM entity by VMware: sourceTemplate = server_instance.get_vm_by_name(template_vm) For locating a folder: folderDict = server_instance._get_managed_objects(MORTypes.Folder) for mor, fname in folderDict.items(): if fname == folder_name: destinationFolder_MOR = mor Here MORTypes.Folder returns a dictionary of folder names with managed object references as keys.

9

Consider the scenario where user selects a cluster to deploy the VM in. Here, the managed object reference for the cluster cannot be passed for VM creation as it requires the host MOR. For the same, we’ll use the code below to get the list of hosts in a cluster and then select a host from the same. clusterDict = server_instance.get_clusters() for mor, cname in clusterDict.items(): if cname == cluster_name: destinationCluster_MOR = mor destinationHost_MOR = random.choice(server_instance.get_hosts(from_mor=destinationCluster_MOR).keys()) Similar operations can be performed to locate the other resources.

Step #3: VM deployment Once all the resources are collected, it is now time to initiate the deployment. That can be done using the command below: sourceTemplate.clone(vm_name, True, folder=folder_name, datastore=destinationDataStore_MOR, host=destinationHost_MOR, resourcepool=destinationResourcePool_MOR, power_on=False, template=False) Here, the objects used are as follows: sourceTemplate MOR of the template to be cloned from vm_name Name of the VM to be deployed True This is for sync_run. if True (default) waits

for the task to finish, and returns a IVirtualMachine instance with the new VM (raises an exception if the task didn't succeed). If sync_run is set to False the task is started and a VITask instance is returned

folder_name name of the folder that will contain the new VM, if not set the vm will be added to the folder the original VM belongs to

destinationDataStore_MOR MOR of the datastore where the virtual machine should be located. If not specified, the current datastore is used

destinationHost_MOR MOR of the host where the virtual machine should be registered. If not specified:

if resourcepool is not specified, current host is used.

if resourcepool is specified, and the target pool represents a stand-alone

10

host, the host is used. if resourcepool is specified, and the

target pool represents a DRS-enabled cluster, a host selected by DRS is used.

if resource pool is specified and the target pool represents a cluster without DRS enabled, an InvalidArgument exception be thrown.

destinationResourcePool_MOR MOR of the resource pool to be used for the new vm. If not set, it uses the same resource pool than the original vm

template=False Specifies whether or not the new virtual machine should be marked as a template

This will deploy the VM in the power off state. Additional modifications can be done using VITask library of the pysphere module.

Step #4: Powering on the VM: To power on a VM, a simple code exists: deployedVM = server_instance.get_vm_by_name(vm_name) deployedVM.power_on() However, it needs to be ensured that the VM is completely powered on before any of the activities continue. For the same, VMware tools if present within the source template, help a great deal. Below code can be used to test whether a successful login can be done or not (VM up confirmation). It will check in every 5 seconds if the login is successful or not. run_loop = True while run_loop: try: deployedVM.login_in_guest('admin', 'pfsense') print "The VM has been powered on!" run_loop = False except: sleep(5) print('VM is still not completely up. Retrying. If required, please check the VM via console.')

Additional VM Configuration: For providing network setting, pySphere offers Guest Operations. Using the same, one can change the network configuration file, services restarts, copy file to guest, create directory in guest etc. The pre-

11

requisite is that VMware tools should be present on the VM, which can be taken care of while creation of the template. For changing a string inside the guest OS: vm.login_in_guest(username,password) vm.start_process("/usr/bin/sed",args=["-i",'',sedParamter,fileLocation]) Additional configuration can also be used VITask module of the pysphere library. Backing module is also implemented for editing VM virtual hardware components.

Additional Product Specific Configuration: Additional configuration on the product side can be done using services like Rest API, PHP API, UI Simulation (Selenium).

12

A N A L Y S I S & C O N C L U S I O N :

We were able to achieve below goals in the scenarios we tried:

• No manual supervision needed.

• No manual errors

• Deployment time reduced for a particular deployment scenario

from 60 minutes to less than 3 minutes

• Deployment time reduced for a particular deployment scenario

from 90 minutes to less than 6 minutes

• Running automation continuously overnight by CSV

implementation for user inputs.

• Number of engineers required for deployment reduced to 20%.

About the Author

Raghav Gaur is working as Principal DevOps Engineer with Afour

Technologies. He holds an experience of 5 years in Data center

configuration, DevOps and Virtualization. His main areas of

expertise are Hyper-V, VMware, Windows Applications and

clustering and Chef. He has been part of multiple projects related

to Cloud Deployments, Backup and Disaster Recovery, DevOps and

Automation.

13

C O M P A N Y P R O F I L E

AFour Technologies is an ISO ISMS 27001:2013 and ISO QMS

9001:2015 (For Software Testing Solutions and Software Quality

Engineering Services) certified organization specialized in software

quality engineering services. Established in 2007, the company has

brought together some of the strongest and most passionate experts in

the field of software quality. With a laser sharp focus on software

quality and engineering, AFour offers specialized services including

DevOps, Performance Engineering and Monitoring and Cloud

Enablement and testing, to complement its expert testing services. Even

as applications move to the cloud and online services continue to grow,

the nature of testing is evolving and so is our test organization. We have

aligned our capabilities and service offerings with the new trends in

technology. For instance, we recognize the fact that no matter how

extensive the functional testing of products and services is, a Test Plan

is incomplete if QA of service quality encompassing service availability,

resilience, fault-tolerance, and maintainability is not accounted for.

We help high technology products and services to migrate to the Cloud

with our Cloud Enablement and testing service. Our DevOps service

introduces agility in the SDLC by acting as a glue between Development

and Operations, and helping our customers rapidly deliver products and

services, through reduced downtime during deployment and more

structured, repeatable, measurable and continuous testing cycles.

Starting from the Architecture Performance review through deployment

and maintenance, robustness and resilience of the applications and

services is ensured with our Performance Engineering and Monitoring

service.

The co-founders of the company have extensive experience in the field

of Software Testing and Quality Engineering, having held senior

management roles in leading independent testing services organizations

like Disha, Aztecsoft itest and MindTree’s Testing Business Unit. The

company has test and engineering centers in Redmond, USA and Pune,

India.

US Off ice:

8201 164th Ave. NE, Suite 200

Redmond, WA 98052-7604

Phone: +1 425 241 0581 / 425 956

3543

INDIA Office:

Pune

501-502, 5th Floor, Kapil Zenith IT

Park, Near Chandani Chowk

Bavdhan (Kh). Pune - 411021

Bengaluru

504, Evoma Business Center, 1st

Floor "The Address" Building

Outer Ring Road, Kaverappa

Layout. Opp. Cessna Business Park,

Kadubeesanahalli , Bengaluru – 560

103

www.afourtech.com

Copyright © 2016 AFour Technologies.

Readers are free to distr ibute th is report within their own organizations,

provided the AFourTech header at the top of every page is also present. present.