I moved from managing 1 to 100 servers with Puppet

Eric Pizzani The Australian National University

Quick Intro: Me •  Works in Canberra for the Australian National

University, originally from Melbourne. •  Responsible for the technical infrastructure of

ANU College of Asia and the Pacific’s web presence.

Quick Intro: The College •  The college has roots dating back to 1946 as

the Universities’ Research School of Pacific Studies

•  220 academic staff supported by 170 general staff

•  2,252 students with over 9,600 alumni •  Subjects from economics, political studies to

Asian languages •  Have more IP addresses than North Korea

Quick Intro: The College More to the point… •  Hosts over 60 websites •  Average of five thousand visitors a day •  Peak of 100,000 on one night •  Running Drupal, Wordpress, and hand

crafted PHP

In the beginning… •  Small IT firm of 4 full time staff and a handful

of contractors •  Desktop support to system administration •  Configuration management was needed

In the beginning…

“Puppet sucks. Puppet is written in Ruby. Ruby sucks.”

In the beginning…

“Puppet sucks. Puppet is written in Ruby. Ruby sucks.”

– Eric Pizzani, 2009

In the beginning…

In the beginning…

In the beginning…

In the beginning…

What could possibly go wrong?

In the beginning…

In the beginning… Just a bad idea

And repeat every time there’s an update? For every machine?

In the beginning… Just a bad idea

And repeat every time there’s an update? For every machine?

In the beginning…

Fairly sure this sets up P HP in an insecure way…

In the beginning…

As before...

What if this needed to be larger?

In the beginning…

T URN OF F THE F IREWALL!?

Why are they instal led in the f irst place?

In the beginning…

Good start, but I’m sure there’s more to do...

ANU – ‘many’ years later •  Used pre-configured virtual appliances •  Low monthly fee gave you access to a wide

range of web applications- Wordpress, OTRS, Drupal

•  Simple to setup- Download the appliance, fill in a few questions in a form and done.

•  Updates were the same process

ANU – ‘many’ years later •  Used pre-configured virtual appliances •  Low monthly fee gave you access to a wide

range of web applications- Wordpress, OTRS, Drupal

•  Simple to setup- Download the appliance, fill in a few questions in a web form and done.

•  Updates were the same process ...in theory

ANU – ‘many’ years later

What could go wrong? •  Mistakes and typos •  Steps forgotten •  Poor asset management •  Repeating the entire initial configuration

process every time you needed to update was time consuming

What did go wrong? •  Custom configuration lost on upgrade or DR. •  Wordpress install inside VA provided

wordpress install. •  Files ended up in places the VA backup

service didn’t know about…

What else went wrong? •  DNS server addresses changed •  I would forget this important step

•  Systems were frequently not put into Nagios

Enter puppet

Enter puppet

=)

The Project •  Legacy Solaris Stack Decommissioning – Management machine was a 1997 vintage Sun

Enterprise 450 – Web server was a 2003 vintage Sun Fire V440 – Custom complied Apache and PHP – Non-existent documentation – 1200 days uptime – 40 websites and sub sites

The Project

The Project

The Project

also known as cheops

DNS

The Project

also known as cheops

DNS SMTP

How puppet helped?

How puppet helped?

How puppet helped?

How puppet helped?

How puppet helped? •  Quick site provisioning – No messing and crafting vhosts by hand

•  “Self” documentated •  Databases created and firewalls setup

Low hanging fruit…or continuous improvement

•  Improvements rolled out to all systems- new and old

•  Manual tasks no more! •  More monitoring checks!

Low hanging fruit…or continuous improvement

•  Improvements rolled out to all systems- new and old

•  Manual tasks no more! •  More monitoring checks!

Low hanging fruit…or continuous improvement

•  Improvements rolled out to all systems- new and old

•  Manual tasks no more! •  More monitoring checks!

Low hanging fruit…or continuous improvement

•  Improvements rolled out to all systems- new and old

•  Manual tasks no more! •  More monitoring checks! •  Auto-documentation

Low hanging fruit…or continuous improvement

Low hanging fruit…or continuous improvement

Night works

•  Server outages & updates

Night works

•  Server outages & updates •  Run yum update -y? Or apt-get update && apt-get upgrade?

Night works

•  Server outages & updates •  Run yum update -y? Or apt-get update && apt-get upgrade?

mco puppet resource exec yum command='/usr/bin/yum update -y' --with-fact osfamily=RedHat!

What’s next?

•  Working on a web infrastructure migration project – Leverage Hiera

What’s next?

•  Working on a web infrastructure migration project – Leverage Hiera – More git

What’s next?

•  Working on a web infrastructure migration project – Leverage Hiera – More git – More monitoring

What’s next?

•  Working on a web infrastructure migration project – Leverage Hiera – More git – More monitoring

...for now…

Questions

???

Feedback

Email me: eric.pizzani@anu.edu.au