+ All Categories
Home > Software > OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Date post: 06-Jul-2015
Category:
Upload: netways
View: 382 times
Download: 1 times
Share this document with a friend
Description:
CentOS, the Community Enterprise OS, uses Opennebula as virtualization plattform for its automated QA-process. The opennebula setup consists of 3 nodes, all running CentOS-6, who handle the following tasks: – sunstone as cloud controller – local mirror/DNS-Server/http-Server for the VMs to pull in packages – one VM to run a jenkins instance to launch the various tests (ci.de.centos.org) – nginx on the cloud controller to forward http traffic to the jenkins VM A public git repository (http://www.gitorious.org/testautomation) is used to allow whoever wants to contribute to pull the current test suite – t_functional, a series of bash scripts used to do funtional tests of various applications, binaries, configuration files and Trademark issues. As new tests are added to the repo via personal clones and merge requests, those tests first need to complete a test run via jenkins. Each test run currently consists of 4 VMs (one for each arch for C5 and C6 – C7 to come), which run the complete test suite. All VMs used for theses tests are instantiated and torn down on demand, whenever the call to testrun a personal clone is issued (via IRC). Once completed successfully, the request is merged into the main repo. The jenkins node monitors this repository and which automatically triggers another complete test run. Besides these triggered test runs, the test suite is automatically triggered daily to run. This is used to verify functionality of published updates – a handfull of failty updates have allready been discovered this way. Besides t_functional, the Linux Test Project Suite of tests is also run on a daily basis, also to verify functionality of the OS and all updates. The third setup is used to test the available and functional integrity of published docker images for CentOS. All these tests are later – during the QA-phase of a point release – used to verify functionality of new packages inside the CentOS QA-Setup.
Popular Tags:
17
CentOS, QA and OpenNebula Opennebula Conference 2014 Berlin December 2 nd to 4 th Christoph Galuschka [email protected]
Transcript
Page 1: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

CentOS, QA and OpenNebula

Opennebula Conference 2014Berlin

December 2nd to 4th

Christoph [email protected]

Page 2: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Who am I

● Christoph Galuschka, age 37● from Innsbruck, Austria● working at TIWAG Tiroler Wasserkraft AG (utility company) in the IT

operations department● Using Linux since 1998, CentOS since 2005● Part of the CentOS-QA-Team since April 2012● Mainly working on release-QA, t_functional git repo for automated testing

and the testing infrastructure, wiki and forums

Page 3: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

CentOS CI

● created in june 2011● an approach to automate QA-release testing● create a set of tests to run for each point release QA-cycle ● use git as a common repository for other users (yes you too) to contribute

new tests● currently contains more then 430 scripts/functional/trademark tests for more

than 110 packages● additionally uses the Linux Test Project

● upcoming: to test every update before public release

Page 4: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

History

● First setup of CI: lost from memory :) and in a different DC● Second setup - move to opennebula● started with 4.0, 4.2 later (RPMs from CentOS-Developers)● 4.8 since November 2014● only “issue” during update: ruby gem● ‘/usr/share/one/install_gems’ vs. rpm

Page 5: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Setup I● 3 Dell 1950III with 4 cores, 16GB RAM and 2x300GB disks (raid1) each

● Hypervisor#1 (K6) primarily runs the jenkins VM, the other two Hypervisors are solely used for test VMs

Page 6: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Setup II

Page 7: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Setup III

● OpenNebula is used as the cloud controller● The sunstone interface is available on Hypervisor1● All VMs instantiated only available on the private network● Datastore transport → SSH● Hosts → KVM● The ci.dev.centos.org site runs as VM in that setup● nginx is used to forward http(s) from the public internet to the VM● local mirror for all CentOS-releases● mimics the setup of mirror.centos.org for all VMs● Jenkins is used to handle all running tests/VMs

Page 8: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Jenkins

● 4 different testgroups

● docker image (C6/C7 only), t_functional, LTP and reimzul

● run on a daily basis

● Infrastructure is hosted by GoDaddy

Page 9: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Jenkins II

Docker image

● pulls and runs current C5/6/7 docker image

● checks installation and command execution inside image

LTP

● Linux Test Project1

● comprehensive checks of basic linux functionality

t_functional

● functional tests of packages

● also checks ™ issues if valid

● 14 contributors● C7 running

since Nov. 7th

reimzul

● called on de- mand via IRC

● used for verifiying merge requests

● All tests run the current versions of CentOS (C5.11, C6.6, C7.0)

1) visit http://ltp.sourceforge.net/

Page 10: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Images used for CI● Created by Karanbir Singh and Jaime Melis● Creates images for C5 and C6 with generic kickstarts and

configurations● Images are

custombuilt on the fly and are downloadable afterwards

● C7 image comes from cloud.centos.org

Page 11: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

KS-Sample for prjrd.netinstallurl --url=http://repohost.raindrops.centos.org/centos/6/os/x86_64/lang en_US.UTF-8keyboard uknetwork --device eth0 --bootproto dhcprootpw --iscrypted $1$ZCDym637$Q8tqR6iCTag8t7a3RA5nI.firewall --service=sshauthconfig --enableshadow --passalgo=sha512 --enablefingerprintselinux --enforcingtimezone --utc Europe/Londonbootloader --location=mbr --driveorder=sdarepo --name="CentOS" --baseurl=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ --cost=100repo --name="ONE-context" --baseurl=http://repohost.raindrops.centos.org/opennebula/6/x86_64/ --cost=100zerombr yesclearpart --all --initlabelpart /boot --fstype ext3 --size=250part pv.2 --size=5000 --growvolgroup VolGroup00 --pesize=32768 pv.2logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --growlogvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512reboot%packages@Base@Coreopennebula-context%end%post --log=/root/post.log --nochroot...

Page 12: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

KS-Sample for prjrd.netsed -i "s/^ACTIVE_CONSOLES=\/dev\/tty\[1-6\]/ACTIVE_CONSOLES=\/dev\/tty1/" /mnt/sysimage/etc/sysconfig/initsed -i '/HWADDR/d' /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth\*rm -f /mnt/sysimage/etc/udev/rules.d/70-persistent-net.rulesfor f in /boot/grub/grub.conf; do /bin/sed -i "s/^serial.*$//" /mnt/sysimage/${f} /bin/sed -i "s/^terminal.*$//" /mnt/sysimage/${f} /bin/sed -i "s/console=ttyS0,115200//" /mnt/sysimage/${f}donerm -rf /mnt/sysimage/etc/rc.d/rc*.d/S*sshdcat >> /mnt/sysimage/etc/rc.d/rc.local <<EOF/sbin/restorecon -Rv /root/.sshif [ ! -e /var/update_done ] then yum update -y touch /var/update_done chkconfig sshd on chkconfig auditd on rebootfiEOF%end

● This kickstart creates an image with disabled ssh-server. After boot it pulls all updates, enables ssh and reboots once (ensuring current kernel for all things to come for this VM).

Page 13: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

introducing t_functional

● Git repo hosting all the functional tests (packages, roles)● Using #!/bin/bash (but other scripting languages are possible!)● Hosted on gitorious and git.centos.org (for QA)● Community driven (yes, you too)● Daily results at: http://ci.dev.centos.org● https://git.centos.org/summary/sig-core!t_functional.git● https://gitorious.org/testautomation/t_functional

Page 14: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Distro and functional tests

● Is release X.y installable?● Through ISO?● Through network (nfs/http/ftp) + pxe?● Mixed (netinstall.iso + network)?● Usb image?● UEFI vs Bios?

● Are all packages in the distro doing what they're supposed to do ?● Still working with different sets of packages (php vs php53 for example in

CentOS 5.x)

Page 15: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

t_functional git workflow

gitoriousClone & change

Request to merge

Jenkins CI merge request

No? No?Fix it

gitorious push git.centos.org pull QA-Env

Page 16: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

further opennebula usage

● Two further setups in DCs in London● controls four hosts in both cases● runs seven.c.o, planet.c.o, some testing-VMs, IPv6 to IPv4 Reverse Proxy,

some internal stuff● www and forum site to come● uses 4.8● GlusterFS over Infiniband● more on that: http://arrfab.net/blog/?p=485

Page 17: OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

Any Questions?

Thank you for your time


Recommended