A One-Stop Solution for Puppet and OpenStack

Post on 08-May-2015

6,388 views 1 download

Preview:

Click to see full reader

Report this document

description

Throughout the last year, we have been using and developing tools that allow us to have an IaaS where our data center is configured by Puppet and our virtualization and authentication needs are catered by Openstack. RedHat's foreman is our lifecycle management tool which we configured to support both bare metal and Openstack virtual machines. We use git to manage environments and hostgroup configurations and we will tell you how we deal with its security implications, how to store Hieradata secrets. Switching from a homebrew toolchain to open source tools like Facter, Foreman, Openstack has turned out into many contributions to these teams. Nearly everyone at CERN has started to wear the devops hat which brings new challenges in terms of development workflows and scalability. Daniel Lobato Garcia Software Engineer, CERN Daniel Lobato is a developer who has worked in very different environmentst, from data centers and mainframes to startups. Nowadays he has dived into the Agile Infrastructure team at CERN where the design and implementation of the new computing infrastructure is done. As for Puppet, he currently helps RedHat to develop Foreman, a lifecycle management tool for physical and virtual machines. One of his goals at CERN is to knot this tool to all the relevant parts of the infrastructure, which includes Puppet for configuration management, OpenStack for virtualization and authentication, Puppetdb and others. He is sure the source of all computer problems is between the chair and the keyboard.

transcript

A one stop solution

for Puppet and Openstack

Daniel Lobato Garcia

daniel.lobato.garcia@cern,ch

@eLobatoss

mailto:daniel.lobato.garcia@cern,ch

What is CERN

Between Geneva and the Jura mountains, straddling the Swiss-French border

Mission: learn what is the universe made of and how does it work?

3

Fundamental

questions in

physics

Why do particles have mass?

What is 96% of the universe made of?

Why isn’t there anti-matter in the universe?

What was the state of matter after the Big Bang?

4

8/12/2013 Document reference 5

8/12/2013 Document reference 6

8/12/2013 Document reference 7

8/12/2013 Document reference 8

Current status

• 270 Openstack hypervisors

• 2900 virtual machines

• 300 users

• 14 Puppet masters

• 6 Foreman backend nodes

• Some production services migrating to our

cloud – early birds

9

Goals

• Ramp up to 15K hypervisors – 150-200K

vms in 2015

• Multi-site (Hungary)

10

8/12/2013 Document reference 11

8/12/2013 Document reference 12

Why?

• Unnecessary homebrew stack of tools

• Shift to cloud standards with minimal

customizations

• High turnover – can’t teach new tools

13

Why?

• Symbiotic relationship with the community

14

Openstack?

• Modular IaaS free open source project

• APIs ~compatible with those of Amazon

15

Openstack Nova

(compute)

Cloud fabric controller

16

Openstack

Keystone (Identity)

RBAC

Integrated with LDAP

Multiple auth* methods

17

Openstack Glance

(Images)

Discovery, registration,

delivery of images

18

Openstack Horizon

(Dashboard)

19

Modules

• Puppet definitions for every use case you

can imagine.

• Dynamic environments

• Hadoop node

• Openstack hypervisor

• … you name it

20

21

Workflow..?

Modules and Git

• Manifests and hieradata are version

controlled

22

23

Git workflow

Puppet masters

24

Easy cherry pick

25

Git workflow

26

Git workflow

Jens

‘Puppetfiles’

Separate repositories

Makes environments and

creates them on the masters

Foreman

• Lifecycle management tool for VMs and

physical servers

• External Node Classifier – tells the puppet

master what a node should look like

27

28

29

Power operations & Foreman

8/12/2013 Document reference 30

Foreman Proxy

Physical

box IPMI

Physical

box IPMI

Physical

box IPMI

VM VM VM

Openstack

Nova API

Openstack VM creation

8/12/2013 Document reference 31

Openstack VM creation

8/12/2013 Document reference 32

Openstack VM creation

8/12/2013 Document reference 33

Scalability experiences

• Split up services

• Puppet – critical vs non critical

34

12 backend nodes

Batch

4 backend nodes

Interactive

Scalability experiences

• Foreman – split into different services

35

ENC Reports

processing UI/API

Load balancer

9443 – UI/API

9444 – Reports

9445 – ENC

…

Scalability experiences

• Autoscale via alarms (Heat)

• Define situations (i.e: load threshold..)

• Spin up VMs as needed

36

Scalability guidelines

37

github.com

/

cernops

38

39

Secrets provisioning (naïve)

• Use case: provision a db password

41

Secrets provisioning (hiera-gpg)

• Use case: provision a db password

42

Secrets provisioning (hack)

• Use case: provision a db password

43

Secrets provisioning

•Masters need not read secrets

44

Copyright © 2018 DOCUMENTS