OpenNebulaConf 2016 - OpenNebula, a story about flexibility and technological agnosticism by ...

2016 Todo en Cloud S.L. www.todoencloud.com

Todo En CloudArchitecture as a Service

OpenNebula, a story about flexibility and technological agnosticism

Alberto Picón Couselo

CIO Todoencloud

25/10/2016


Cloud providers are constantly

addressing the technology

limitations on their

infrastructures, which must be

overcome to meet customer

needs.

The technological agnosticism

and flexibility of OpenNebula

has allowed Todoencloud to

provide the most efficient

solutions to the needs of its

customers

“Opennebula’sTechnological Agnosticism”


Why do they choose us?

Our clients choose us for our

technical expertise, our stability

and our Disaster Recovery

capacity, our management,

High Security (certificates in

27001) and overall

management capacity of our

clients’ business architecture.

Todo En Cloud

2016 Todo en Cloud S.L. www.todoencloud.com 4

Public Cloud Solution

Thanks to our capabilities,

to our human, technical and

technological resources, we

are a reference in the

delivery of

“Architecture as a service”.

Our cloud infrastructure is largely

based on Free and Open Source

Software, which allows us to be more

efficient and competitive.

We do not subcontract nor delegate

the management of the Cloud

infrastructure to any third parties,

which means we can improve the

Agreement Standards in our services

with our clients, guaranteeing them

that their data will not be passed on to

any business competitors or

governments.

All the infrastructure is located in two

Data Centers.

.

A NEUTRAL Spanish Data Center

(Tier IV), by which we can guarantee

that it complies with LOPD

standards, and a second CPD Tier II

in order to provide DRP services.

Besides, its information is in a

completely clusterized environment.

It is possible to carry out complete

system copies within personalised

frequency periods.

Our communication backbone is of

10Gbe and of 40Gbe both in its core

and in INTERNET communications.

TodoenCloud Services


The New Cloud Generation

Our cloud core infrastructure is

always based on Free Software

and Open Source technology

solutions

This approach provide us the

chance to explore and deploy

technology without vendor lock-in

issues

TodoEnCloudTechnological

Approach


Opennebula. Our Cloud Orchestrator

� Opennebula since version 3.2

� Opennebula provide us the flexibility and technology agnosticism we need for

our cloud management core and to deploy services to our customers

� Few Components, robust and rock solid cloud orchestrator

Xen is our Core Virtualization Technology

� Technologically Neutral (Software and Hardware)

� High Performance and total VM isolation

� Low impact between VM Workloads

ZFS storage clusters

� Our storage core is based on ZFS storage clusters

� SSD and SATA with SSD R/W Cache Tiers

� VMs (DataStore Level) and NFS FileSystem Snapshots

� ZFS Send/Receive for remote replication

Todoencloud Technological Evolution (I)


Xen Virtualization and Opennebula

� Opennebula Xen Support is great!

� However:

More and more features are supported for Libvirtd Linux KVM World

� Limiting disk IOPs and network bandwidth on Xen is not supported

� Xen version support and migration can be a REAL challenge

� Xen live migration require same hardware architecture between

hypervisors

� Some Linux Distributions decided to remove Xen Support on new

releases (yes, those Xen-* packages are not used at all, you know)

Todoencloud Technological Evolution (II)


ZFS Known Limitations

ZFS Read and Write Speed is limited by:

� Maximum I/O of the slowest disk

� Maximum throughput of the ZIL and Read Cache

� RAIDZ design and number of disks per pool

ZFS scalability is expensive:

� Capacity expansion per tier can be very expensive and

delivers performance degradation

ZFS High Availability solutions can fail:

� ZFS export and import processes between

active/passive nodes can fail during takeover

� Migration impact over all ZFS pools on the active node

Todoencloud Technological Evolution (IV)


Opennebula Upgrade

� Migration to Opennebula with latest Ceph Features

Add Full KVM support (without loosing Xen hypervisors)

� Using Paravirtualized hardware support: virtio-scsi virtio-blk and virtio-net

� Disk IOPs Limitation using Opennebula Sunstone Templates and Libvirtd

� Network Bandwidth Limitation using Opennebula Hook VM control

� New! FreeBSD now supports KVM VirtIO disk and networking drivers!

� KVM Live migration is fully supported

Add a CEPH Cluster (without loosing ZFS Shared Model)

� Opensource Technology

� SSD Pool and SATA Ceph Pool with SSD Journals

� Scripting for Scheduled VM Snapshotting and snap rotation

� CephFS Support!!

� HA NFS-Ganesha VM servers

� HA Samba-VFS CephFS VM servers

Todoencloud Technological Evolution (V)


� Ceph design, deployment and maintenance is not an easy task (not at all!)

� Efficient CRUSHMap design and Node/Component Failure Testing is a MUST!

� 3 Ceph Monitor servers are required

1 CPU Intel(R) Xeon(R) CPU E5-2640 and 32 GB are enough

� At least 2 Ceph MDS servers (active/passive) for CephFS are required

1 CPU Intel(R) Xeon(R) CPU E5-2640

RAM: The bigger the better!

4Kbytes/inode -> 64GB available RAM ~ 16M of inodes on memory

� Ceph requires a minimum number of OSD nodes to be usable

10 OSD nodes per tier should be considered the minimum

1 Logical CPU (for instance, E5-2640) and 2GB RAM / OSD process/disk

� Special attention to the number of rotational disks per journal SSD disk

� Setting up the size of a SSD journal partition for a rotational SATA/SAS disk:

SSD partition size should not exceed 5s of maximum network throughput.

For example: 10GB NIC -> 6~8GB journal partition

Ceph v10.2.2. Deployment Challenges (I)


CephFS v10.2.2. Deployment Challenges (I)

� CephFS looks really cool, why not?. Let’s try it!

� What?!!, CephFS in Production!!!?

� CephFS “Jewel” is considered production ready since April 21st 2016

� However, CephFS Snapshots are NOT SUPPORTED -> FS CORRUPTION!!!

� Our customers need a scalable shared Filesystem to interconnect their VMs

� NFS servers using conventional LVM RBD disks for VM storage are dangerous too


CephFS v10.2.2. Deployment Challenges (II)

� CephFS Solution consists of the following:

Two MDS servers 1 CPU Intel(R) Xeon(R) CPU E5-2640 128GB RAM Active/Passive

CephFS Metadata over SSD Disks, CephFS Data over SATA/SSD Multitenant FS Structure

NFS-Ganesha 2.3, Samba 4, and CephFS Kernel driver active/passive PaceMaker cluster

� Tested during a whole month of an extreme 24/7 file copying process

45TB of files copied, more than 3500M of files

� Ceph metadata is read/written from CephFS clients using MDS Server memory:

4KBytes/inode -> 30M of inodes ~ 120GB of RAM

� MDS Ceph master configuration file /etc/ceph/ceph.conf:

[mds]mds_cache_size = 30000000

� MDS max inodes can be changed live using:

ceph daemon mds.mds_server_name config set mds_cache_s ize 30000000

� Directories and files on a CephFS can be assigned to a specific pool using setfattr/getfattr tool:

setfattr -n ceph.dir.layout.pool -v <ceph_pool> <direc tory>setfattr -n ceph.file.layout.pool -v <ceph_pool> <fil e>

� These attributes will be assigned to NEW files and nested directories created on CephFS

� Stablishing a correct directory hierarchy from start is VERY IMPORTANT


� NFS-Ganesha Compilation on Setup procedure:

apt-get install git-core cmake build-essential portm ap libcephfs-dev bison flex libkrb5-dev libtirpc1 uuid-dev apt-get install attr nfs-common libnfsidmap-dev

cd /usr/srcgit clone https://github.com/nfs-ganesha/nfs-ganesha .gitcd nfs-ganeshagit checkout -b V2.3-stable origin/V2.3-stablegit submodule update --init

mkdir buildcd buildcmake -DUSE_NFSIDMAP=ON ../srcmake; make install

� NFS-Ganesha DBUS Setup procedure:

cp ../src/scripts/ganeshactl/org.ganesha.nfsd.conf /e tc/dbus-1/system.d/

CephFS v10.2.2. NFS-Ganesha Compilation and Setup (I)


� /etc/ganesha/ganesha.conf

# NFS-Ganesha Master configuration fileNFSv4{

IdmapConf = /etc/idmapd.conf;}

EXPORT{

Export_ID = 1;Path = "/customer1";Pseudo = "/";Access_Type = RW;Protocols = 4;Squash = None;Transports = TCP;SecType = sys;FSAL {

Name = CEPH;}

}

CephFS v10.2.2. NFS-Ganesha Compilation and Setup (II)


� Systemd service file, /etc/systemd/system/nfs-ganesha.service

[Unit]Description=NFS GaneshaWants=network.target network-online.targetAfter=network.target network-online.target

[Service]Type=simpleExecStart=/usr/bin/ganesha.nfsd -F -f /etc/ganesha/g anesha.conf -L /tmp/ganesha.logExecStop=/bin/kill -SIGTERM $MAINPIDExecReload=/bin/kill -HUP $MAINPIDKillMode=processRestart=on-failureRestartSec=5s

[Install]WantedBy=multi-user.target

CephFS v10.2.2. NFS-Ganesha Compilation and Setup (III)


~# systemctl enable nfs-ganesha.service~# systemctl start nfs-ganesha.service~# systemctl systemctl status nfs-ganesha.service

� Add rpc.idmapd to /etc/rc.local:

/usr/sbin/rpc.idmapd

� Set NFS Domain to Idmap service, /etc/idmap.conf

Domain = localdomain

� Set IDMAPD service as a needed service, /etc/default/nfs-common

# Do you want to start the idmapd daemon? It is only needed for NFSv4.NEED_IDMAPD="yes“

� Create /etc/ceph/secret.conf

File contents the password hash of the user specified in /etc/fstab to mount CEPHFS target)

� Add CephFS mount point using native CephFS Kernel driver in /etc/fstab.conf file:

mon01,mon02,mon03:/cephfs/customer1 /mnt/cephfs/cu stomer1 cephname=customer1,secretfile=/etc/ceph/secret.conf,noa time,nodiratime,_netdev0 1

CephFS v10.2.2 Deployment Challenges (II). NFS-Ganesha Compilation and Setup (IV)


� Opennebula uses SSH access to KVM hypervisors to manage RBD images

� Ubuntu LTS 14.04 KVM hypervisor installation is easy and straightforward

� VM Live migration problems due to apparmor daemon and libvirtd security profiles:

ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmo r.d/disable/ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helpe r /etc/apparmor.d/disable/apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtdapparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.v irt-aa-helperinvoke-rc.d apparmor reload/etc/init.d/libvirt-bin restart

� RBD Ceph v.10.2.1 exclusive lock feature led to KVM random VM crash during

scheduled RBD snapshot execution (librbd/ExclusiveLock.cc: 197: FAILED assert)

https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1607694

� Workaround: Disable exclusive lock for registered RBD images and adding global option

“rbd default features = 33 ” to ceph.conf on KVM hypervisors

� Bug seems to be corrected on Ceph v.10.2.3 (Test Pending)

KVM and Ceph Challenges (I)


� Opennebula does not support BW limitation per interface through Libvirtd

� It is possible to limit BW using a VM Hook during VM Deployment when it reaches

RUNNING state

� VM Hook on /etc/one/oned.conf:

VM_HOOK = [name = "set_vm_network_speed“on = "RUNNING“command = “net_bwlimit.sh“arguments = "$ID $TEMPLATE" ]

� We can transfer TEMPLATE VARIABLES to our script using the following code:

XPATH="/var/tmp/one/datastore/xpath.rb -b $2“unset i j XPATH_ELEMENTSwhile IFS= read -r -d '' element; do XPATH_ELEMENTS[i++]="$element" done < <($XPATH /VM/TEMPLATE/VARIABLE) VARIABLE="${XPATH_ELEMENTS[j++]}"

KVM and Ceph Challenges (II)Opennebula Network Interface Bandwidth limitation through VM Hook (I)


� Create /var/lib/one/remotes/hooks/net_bwlimit.sh script:#!/bin/shVM_ID=$1XPATH="/var/tmp/one/datastore/xpath.rb -b $2“unset i j XPATH_ELEMENTSwhile IFS= read -r -d '' element; do XPATH_ELEMENTS[i++]="$element"

done < <($XPATH /VM/TEMPLATE/NETBW) NETBW="${XPATH_ELEMENTS[j++]}“if [ $(sudo virsh domiflist one-${VM_ID} | awk '/vnet/ { print $1 }' | wc -l) -ge 1 ]then

for iface in $(sudo virsh domiflist one-${VM_ID} | awk ' /vnet/ { print $1 }')do

sudo virsh domiftune one-${VM_ID} ${iface} --live --in bound $NETBW,$NETBW,$NETBW --outbound $NETBW,$NETBW,$NETBW

donefiexit 0

� chmod 755 /var/lib/one/remotes/hooks/net_bwlimit.sh� chown oneadmin:oneadmin /var/lib/one/remotes/hooks/ne t_bwlimit.sh

� Execute “onehost sync ” as user oneadmin on Opennebula node to sync script to HVs

KVM and Ceph Challenges (II)Opennebula Network Interface Bandwidth limitation through VM Hook (II)


� Use a CUSTOM Tag to set NETBW variable to the VM Template (value in KBps):

� After a successful VM deployment, the network interface bandwidth is correctly set-up:

~# virsh domiftune one-477 vnet0inbound.average: 131000inbound.peak : 131000inbound.burst : 131000inbound.floor : 0outbound.average: 131000outbound.peak : 131000outbound.burst : 131000

KVM and Ceph Challenges (IV)Opennebula Network Interface Bandwidth limitation through VM Hook (III)


The New Cloud Generation.

We are going to describe our

product and how it is

consumed, both from the

standpoint of an IT department

as from that of a business

department.

Cloud Bursting As

Business


More Information about us:

www.todoencloud.com

[email protected]

+34910801233

Thank you!

Date post:	07-Jan-2017
Category:	Technology
Upload:	opennebula-project
View:	67 times
Download:	1 times