Classification: Public

Puppet Camp Melbourne 2014

Puppet and A DevOps Journey

Keith Ferguson

Classification: Public

TODAY

Who am I?

What is ANZ GCIS?

Our challenges

Our approach

Lessons Learnt (a.k.a. Things I Wish We Knew at the Beginning)

Storing sensitive information in Hiera

2

Classification: Public

WHAT IS ANZ GCIS 3

Global Customer Integration Solution

Classification: Public

WHAT IS ANZ GCIS 4

0%

50%

100%

150%

200%

250%

300%

350%

Oct-12 Oct-13

Customer Growth

$-

$5.00

$10.00

$15.00

$20.00

$25.00

$30.00

Oct-12 Oct-13

Bill

ion

Value Processed

Classification: Public

GCIS DEVELOPMENT – MARCH 2014

4 teams, ~40 people

Windows Server / .NET / IIS / Microsoft SQL

~50 servers across 5 environments (Prod/DR + 4 non-prod environments)

Agile (scrum)

Supporting Tools:

Git

Atlassian: Bamboo, Stash, Jira, Confluence

Powershell

RDP & Elbow Grease

5

Classification: Public

GCIS DEVELOPMENT – MARCH 2014

Things We Did Well

CI & Deployment Automation – ~8k builds, 2.5k non-prod deploys / month

Test Automation – On commit & nightly for all channels

Delivering change to our existing channels

6

0

1

2

3

4

5

6

7

8

9

Jan Feb Mar Apr May Jun Jul Aug Sep Oct

Production Releases

Classification: Public

GCIS DEVELOPMENT – MARCH 2014

Things We Needed To Do Better

Management of complexity — Dependencies and change

Delivery of new channels — Infrastructure & configuration of application containers

3rd level support & ops

7

Classification: Public

UAT ENVIRONMENT

CHALLENGE 1 — DEPENDENCIES & CHANGE 8

1

2 3

We Had:

Classification: Public

PROJECT 3 PROJECT 2 PROJECT 1

CHALLENGE 1 — DEPENDENCIES & CHANGE 9

We Needed:

Classification: Public

CHALLENGE 2 — GROWTH IN CHANNELS 10

0

5

10

2012 2013 2014 2015

Classification: Public

CHALLENGE 3 — 3RD LEVEL OPS

Little to no production access

No broad knowledge of production infrastructure

Heavy reliance on key team members

Far removed from incidents

11

Classification: Public

12

APPROACH

APPLICATION INSIGHT CENTRALIZED LOGGING CONFIGURATION MANAGEMENT

Classification: Public

Classification: Public

Puppet Enterprise vs. Open Source

OUR APPROACH

Classification: Public

OUR APPROACH

1. Dedicate somebody

2. Spike it, try to understand it

14

Classification: Public

OUR APPROACH 15

https://docs.puppetlabs.com/pe/latest/images/puppet/pe-configuration-data.png

Classification: Public

OUR APPROACH

1. Dedicate somebody

2. Spike it, try to understand it

3. Identify the right questions

4. Training for a core group – Puppet Fundamentals

5. Implement the framework

6. Expand usage across team

16

Classification: Public

ROLES & PROFILES

Puppet Is Code. Abstractions Matter

A node has one role

A role is composed of one or more profiles

Profiles are composed of module declarations

Craig Dunn:

https://puppetlabs.com/presentations/designing-puppet-rolesprofiles-pattern

17

Classification: Public

ROLES & PROFILES 18

site.pp wordpress.pp

db.pp

node wp { include role::wordpress }

class role::wordpress { include profile::db include profile::db::php include profile::wordpress }

Adapted from https://github.com/hunner/roles_and_profiles

class profile::db { class { 'mysql::server': config_hash => { 'root_password' => '8ZcJZFHs...' } } }

class role::wordpress_web { include profile::wordpress }

class role::wordpress_db { include profile::db include profile::db::php }

site.pp node wp_web { include role::wordpress_web } node wp_db { include role::wordpress_db }

roles

Classification: Public

CLASSIFICATION

Site.pp

19

node 'www1.example.com’, 'www1.example.com' { include profile::common include profile::apache } node /^(foo|bar)\d+\.example\.com$/ { include profile::common }

Classification: Public

CLASSIFICATION AT THE NODE

Define a role Fact

20

1

On the node:

puppet config set role ‘role::some_role’ Or remotely:

Invoke-Command –ComputerName c1,c2 –ScriptBlock {puppet config set role ‘role::some_role’}

3

site.pp 2

CON

> Not secure for multi-tenant environments

> Node classification is not version controlled

PRO

> Simple & flexible

> Great for dynamic environments

node default { include $::role }

Classification: Public

PACKAGING ON WINDOWS

Like Yum/Apt-Get for Windows, Powered by NuGet & Powershell

21

https://chocolatey.org

Classification: Public

PACKAGING ON WINDOWS 22

Chocolatey Package Provider for Puppet:

https://forge.puppetlabs.com/rismoney/chocolatey

$chocolatey_package_list = [ 'powershell4', 'DotNet4.5', 'git.install', ] package { $chocolatey_package_list: ensure => installed, provider => 'chocolatey', }

Classification: Public

PACKAGING ON WINDOWS 23

Constraints

No internet access

Trust

Package availability

Classification: Public

PACKAGING ON WINDOWS

Package1 |--- Package1.nuspec |--- ChocolateyInstall.ps1 |--- ChocolateyUninstall.ps1 Package2 |--- Package2.nuspec |--- ChocolateyInstall.ps1 |--- ChocolateyUninstall.ps1

24

Binaries Archive

IIS nuget feed

Install-ChocolateyPackage ... -checksum 'EE01FC4110C73A8E5EFC7CABDA0F5FF7'

Classification: Public

R10K IS KEY TO WORKFLOW

Do you want?

Multiple environments from a single puppet master

Dynamic environments eg from feature branches

Declarative management of puppet modules: Puppetfile

If so you want r10k: https://github.com/adrienthebo/r10k

‘Smarter Puppet deployment, powered by killer robots’

25

forge "https://forge.puppetlabs.com" mod 'puppetlabs-ntp', "0.0.3” mod 'puppetlabs-apt', :git => "git://github.com/puppetlabs/puppetlabs-apt.git"

Classification: Public

R10K DEPLOYMENT

1. Repository Trigger 2. Webhook to r10k

3. git: fetch branch to puppet master

Classification: Public

GOVERNANCE

Trust, but verify: Code

Minimum approvers policy for PRs

Pull request please plugin:

Workzone plugin:

- PR reviewer addition via policy (Users + Groups)

- Automated un-approval on PR change

27

____ _ _ ____ _ ___ _ | _ \ _ _ | || | | _ \ ___ __ _ _ _ ___ ___ | |_ |__ \| | | |_) || | | || || | | |_) | / _ \ / _` || | | | / _ \/ __|| __| / /| | | __/ | |_| || || | | _ < | __/| (_| || |_| || __/\__ \| |_ |_| |_| |_| \__,_||_||_| |_| \_\ \___| \__, | \__,_| \___||___/ \__| (_) (_) |_| Please raise a pull request to merge code to this branch.

Classification: Public

GOVERNANCE

Trust, but verify: Deployment 28

Classification: Public

R10K & PUPPETFILE

Challenge:

Puppetfile functionality runs on the master

Master needs to download modules from somewhere

Master has no internet access

Trust

29

Classification: Public

R10K MODULE WORKFLOW

acidprime/r10k * master

devops/r10k

Puppetfile mod 'r10k',

:git => 'ssh://git@stash/r10k.git',

:ref => 'gcis-master'

* gcis-master /

* master

Classification: Public

WHEN IS AN ENVIRONMENT NOT AN ENVIRONMENT

Do you want to stage the rollout of puppet code across Production?

If yes: production is not a puppet environment, it’s an application environment (tier)

Puppet environments exist to apply different revisions of code to different nodes

Application environments are how you want to configure (hiera) your code

31

|-- prod_a.yaml

|-- prod_b.yaml

|-- dr_a.yaml

|-- dr_b.yaml

Problem: Hiera: "%{environment}”

Solution: Custom fact ‘tier’: puppet config set tier ‘production’

Hiera: "%{::tier}"

http://garylarizza.com/blog/2014/03/26/random-r10k-workflow-ideas/

|-- production.yaml

|-- uat.yaml

|-- sit.yaml

|-- st.yaml

Classification: Public

USE PUPPET TO MANAGE PUPPET

Example: Hiera.yaml

32

--- :backends: - yaml :logger: console :hierarchy: - "nodes/%{::clientcert}" - "%{environment}" - global :yaml: :datadir: /etc/puppetlabs/puppet/hiera/%{environment}/

Classification: Public

USE PUPPET TO MANAGE PUPPET

Example: Hiera.yaml via Puppet – https://forge.puppetlabs.com/hunner/hiera

33

class { '::hiera': backends => ['yaml’], datadir => '/etc/puppetlabs/puppet/hiera/%{environment}/', hierarchy => [ 'servers/%{::clientcert}’, '%{environment}’, 'global', ], }

Classification: Public

USE PUPPET TO MANAGE PUPPET 34

Example: Install & configure r10k - https://forge.puppetlabs.com/zack/r10k

class profile::puppet::master { class { 'r10k': version => '1.3.1', sources => { 'puppet' => { 'remote' => 'ssh://git@stash/puppet.git', 'basedir' => "${::settings::confdir}/environments", 'prefix' => false, }, 'hiera' => { 'remote' => 'ssh://git@stash/hiera.git', 'basedir' => "${::settings::confdir}/hiera", 'prefix' => false, }, }, purgedirs => ["${::settings::confdir}/environments", "${::settings::confdir}/hiera" ], manage_modulepath => true, modulepath => "${::settings::confdir}/environments/\$environment/modules:${::settings::… mcollective => true, } }

Classification: Public

STORING SENSITIVE INFORMATION IN HIERA

Hiera-eyaml: https://github.com/TomPoulton/hiera-eyaml

/etc/puppetlabs/puppet/hiera.yaml

35

--- :backends: - eyaml :eyaml: :datadir: /etc/puppet... :extension: "yaml" :pkcs7_private_key: "/etc/..." :pkcs7_public_key: "/etc/..."

/etc/puppetlabs/puppet/hiera/prod.yaml

--- profile::my_app::app_secret: > ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAWLZRfTW8YxvCOf4akyxYdBV5m2AdF+HF85Cz lqw8qQNKW9oMqktgyaoNzOMGufr0rp2BOzrdQ+/z3bmQ9HPK2qqJ1k/n/jX9 D5lf5ReEKj5VnhnLblbLyI5HCfLbR...]

Classification: Public

STORING SENSITIVE INFORMATION IN HIERA 36

0000 +

/etc/puppetlabs/puppet/hiera.yaml

--- :backends: - eyaml :eyaml: :datadir: /etc/puppet... :extension: "yaml" :pkcs7_private_key: "/etc/..." :pkcs7_public_key: "/etc/..."

/etc/puppetlabs/puppet/hiera/prod.yaml

--- profile::my_app::app_secret: > ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAWLZRfTW8YxvCOf4akyxYdBV5m2AdF+HF85Cz lqw8qQNKW9oMqktgyaoNzOMGufr0rp2BOzrdQ+/z3bmQ9HPK2qqJ1k/n/jX9 D5lf5ReEKj5VnhnLblbLyI5HCfLbR...]

Classification: Public

STORING SENSITIVE INFORMATION IN HIERA 37

Hardware Security Module (HSM)

Provides network based FIPS 140-2 Level 3 secure cryptographic services

Puppet Integration:

Custom hiera eyaml encryptor: https://github.com/acidprime/hiera-eyaml-pkcs11

Operates in two modes: Native pkcs11 or OpenSSL CHIL

RSA encryption – Private key is protected by HSM, public key is committed to version control

Native mode will not work in PE >= 3.4 due to jruby

Currently limited by RSA block size

Classification: Public

hiera-eyaml-pkcs11 38

Classification: Public

SUMMARY

Roles & Profiles Pattern

Classification at the node

Chocolatey for packaging on Windows

Use r10k & build a good workflow, governance included

Puppet environments aren’t application environments

Use Puppet to manage puppet

The community is excellent, use it

Storing sensitive information in hiera – This is a risk weighted decision

39

Classification: Public

WHERE ARE WE NOW

> Started in March

> Appdynamics, Splunk & Puppet all operational

> Deployed & configured AD + Splunk agents via Puppet & Chocolatey (and upgraded since)

> Functional vagrant workflow integrated with vmware through the vagrant-vsphere plugin

> Puppet framework is stable, expanding use to broader team

> On-demand testing environments are a WIP

40

Classification: Public

THANKS

My Team

— Dylan Ratcliffe

— William Gaunt

— Laith Murad

GCIS Infrastructure Team

— Lee Murphy

— Sathish Kannan

Brett Gray & Zack Smith

Gary Larizza & Craig Dunn

Rob Reynolds

41

Classification: Public

LINKS & INFORMATION

Contact Info:

keith.ferguson@anz.com; linkedin.com/in/keithferguson

Tools

r10k - https://github.com/adrienthebo/r10k – read the docs not just the readme

r10k configuration module - https://github.com/acidprime/r10k

hiera eyaml - https://github.com/TomPoulton/hiera-eyaml

hiera eyaml pkcs11 backend - https://github.com/acidprime/hiera-eyaml-pkcs11

Reading / Learning

Gary Larizza – http://garylarizza.com/ – Building a functional puppet workflow series is great

Rob Nelson Puppet for vSphere Admins – http://rnelson0.com/puppet-for-vsphere-admins/

Crag Dunn’s Roles & Profiles Pattern Talk – https://puppetlabs.com/presentations/designing-puppet-rolesprofiles-pattern

Puppet Conf & Puppet Camp talks in general

42

Classification: Public

QUESTIONS 43