CloudStack EU user group - Trillian

transcript

The Cloud Specialists

THE HITCH-HIKERS GUIDE TO CI/CD

ShapeBlue.com • @ShapeBlue

Paul Angus, VP Technology • @CloudyAngus

The Cloud Specialists ShapeBlue.com @ShapeBlue

“ShapeBlue are expert builders of public & private clouds. They are the leading

independent global CloudStack services company”

A b o u t S h a p e B l u e

C l i c k t o e d i t

Paul Angus – VP Technology• Global authority on CloudStack & cloud infrastructure design.

• 10+ years C-Level experience.

• Apache CloudStack project committer

• Specialising in deployment of CloudStackand surrounding infrastructure

• USP, Georgian Ministry of Justice, Orange, TomTom, PaddyPower, Ascenty, BSkyB

A b o u t M e

S h a p e B l u e c u s t o m e r s

Tr i l l i a n

“Building CloudStack environments using CloudStackand Ansible? Aren’t you just having fun?”

“Building CloudStack environments using CloudStack?Are you just having fun?”

Talented Cloud Architect

“If we can automate the building of environments including the networking, and generate any topology we want quickly, we can achieve the tasks that I’ve highlighted in these slides that I have with me, just in case you asked.”

building of environments including the networking, and generate any topology we want quickly, we can achieve the tasks that I’ve highlighted in these slides that I have with me, just in case you asked.”

CEO“That would be excellent.Go ahead.Oh, and here’s a pay rise”

We need to test:• New features• Hotfixes/patches• Full Versions• Physical integrations• New hypervisors• Upgrades

• Against a broad range of deployment architectures• Against a broad range of hypervisor types and

versions

Many, many, MANY permutations of a CloudStack deployment….• Basic / Advanced• Local / shared / mixed storage• More than 8 common hypervisor types/versions• 4 or 5 Management server OS possiblilities

• That’s 144 combinations only looking the basics.

• Previous attempts at test automation haven’t been able to get coverage or capacity.

Discovery projects:• New Features

• CloudStack integrations (ie Ceph)

• CloudStack ecosystem components (ie ELK stack)

Lab v1.0

• Templates to build nested hypervisor hosts

• Cloudmonkey scripts to configure zone

• Manual creation of storage mounts

Simple but pretty effective

H u m b l e B e g i n n i n g s

Lab v1.0

• A lot of small edits scripts to fit use cases.

• Human error still a big factor

• Can’t ‘set and forget’

• A lot of individual knowledge required

• Loose definition of ‘testing’.

H u m b l e B e g i n n i n g s

Trillian Tenets

Fully automated deployment of CloudStack environments from any CloudStack package repo

Tr i l l i a n Te n e n t s

• Ability to support multiple concurrent environments• No hardcoding to a specific (our) lab/environment• As few separate components as possible• Separation of ‘code’ and ‘configuration’• Scripted command-line use fully supported• Ability to support as many hypervisors as possible• As flexible as possible• Best ‘performance’ possible• Best utilization of hardware resources possible

Tr i l l i a n Te n a n t s

Meet The Family

T h e C I F a m i l y

Trillian

MarvinWinston

JenkinsBlueOrangutan

TrillianAnsible based environment builder

• Multi-tenant (and multiple concurrent environments)• CloudStack builds the Virtual Infrastructure• Hypervisor hosts are CloudStack templates• Supports multiple and mixed hypervisors• Supports Advanced and Basic networks• Open source

TrillianInstalls & Configures:

• CloudStack• MySQL• XenServer (configures)• vCenter (configures)• ESXi (configures)• KVM• Marvin hosts and Marvin test suites

MarvinPython based test framework based on ‘nose’ test platform. Configuration files define environment to be tested.

• Integral part of the CloudStack codebase• Recently given its own install package• Smoke and Component tests• 53 smoke test suites• 154 component test suites• Additional tests for plugins such as Nuage and SolidFire

JenkinsCI Platform

• Runs build jobs to produce RPMs from source code• Presents an integration point to wrap up the creation of

environments using Trillian and the running of tests using Marvin.

• Gives a UI to Trillian.

Click icon to add picture

Blue OrangutanBuild-bot

• Automation endpoint to send build jobs to Jenkins for CentOS 6/7 and Debian packages from CloudStack source

• Provides an integrated way to create packages from pull requests in Github.

• Can kick-off Jenkins jobs to build and test environments based on packages which it builds.

WinstonTest Orchestrator (Future project)

• Receives ‘offers’ of test type/capacity from remote test labs• Takes request from Blue Orangutan to test packages and

distributes tests to suitable remote labs.Offers might include:• Use of local storage• Use of basic vs advanced networks• Use of particular hypervisors/versions

How its used

Developers doDark Magic

Developer creates pull request on

GitHubInitial code review

Blue Orangutan calls Jenkins to build packages

based on PR

Blue Orangutan calls Jenkins to run

a test matrix

Jenkins calls Trillian to build

environments

Trillian builds environments (inc

Marvin host)

Jenkins calls respective Marvin hosts to run tests

Jenkins attaches results to pull

request on GitHub

The crowd goes WILD.

P r o c e s s u s e d i n A p a c h e C l o u d S t a c k P r o j e c t

• Test Pull Requests to add XenServer 7 support• XenServer won’t run on ESXi 5.5

• Create XenServer 7 VMs on stand-alone ESXi 6.0 host and ‘point’ Trillian at them.

• vSphere 6.5 – Create vCenter and ESXi templates and run tests.

X e n S e r v e r 7 & v S p h e r e 6 . 5

• Build base CloudStack environment quickly

• Leverage CloudStack to create other infrastructure components just as quickly.ie• HostBill server• Cloudian manager server and storage nodes• Leverage CloudStack networking to give external access

to environments – Static NAT, port forwarding or VPN.

C l o u d i a n & H o s t B i l l

• Build required configuration with required components• Specific versions of CloudStack• Specific hypervisor types• Specific hypervisor versions

• Manual creation of additional components still allowed• ie second zone with different hypervisor type or version

S o l i d F i r e P l u g i n

We can do all of these things

AT THE SAME TIME

C o u p D e G r â c e

Trillian

https://github.com/shapeblue/Trillian

Tr i l l i a n

• Single zone• Use pre-set or custom Repos• Local or shared storage• ESXi, KVM (C6,C7,U14.04),

XenServer hosts• Customisable number of primary

and secondary storage pools• Multiple Management servers• Selectable Management server

OS (C6.x, C7, U14.04)

• Multiple hypervisors• dvSwitch (public/guest traffic)• Configurable Java versions• Option to use pre-built hosts• Multiple MySQL configurations

A n s i b l e i n T r i l l i a n

• All driven from Ansible

• A global group_vars file stores all local configuration data such as:

• Passwords• All defaults ie basic vs

advanced or network naming

• EVERYTHING can be customised at runtime.

(except that)(and that)

ansible-playbook generate-cloudconfig.yml -i localhost --extra-vars "env_name='localstoragetest‘env_version=cs49mgmt=1db=0hvtype=x xs_ver=xs65sp1hv=2env_accounts=allpri=2sec=1build_marvin=yesmgmt_os=6use_local_storage=true

use_shared_storage_systemvms=trueuse_shared_storage=false"

ansible-playbook deployvms.yml -i ./hosts_localstoragetest

A n s i b l e i n T r i l l i a n

Tr i l l i a n i n J e n k i n s

• Different Jenkins’ jobs with specific‘assumptions’.

• Pull Requests receive URL from Blue Orangutan and automatically Run test suites, then clean up.

• Reference_Trillian is a reference implementation of Jenkins and has more corner-case options exposed.

• Allows user to force a version of Java

• Allows use of ‘pre-built’ hypervisors and hypervisors which cannot themselves be virtualised.

• If environment requires the hypervisor to use local storage, then CloudStack adds a data volume to hypervisor and configures hypervisor to use it for local storage. This keeps template size down

• Mixed local and shared storage is supported

Click icon to add pictureInternal ShapeBlue summary page UI plugin.

Tr i l l i a n & J e n k i n s i n C l o u d S t a c k

• UI gives overall status/health of ‘Lab’ and environments

• UI links to a job’s Jenkins output as well as UI of built environment and SSH session onto specific VMs

I n S u m m a r y

The Cloud Specialists

THE HITCH-HIKERS GUIDE TO CI/CD

ShapeBlue.com • @ShapeBlue

Paul Angus, VP Technology • @CloudyAngus