September 24, 2014
Erik Dalén
Experiences from running masterless Puppet
whoami• System engineer at Spotify since almost 3 years.
• Community contributor to Puppet.
• Author of Puppet Explorer web UI.
• Author of the puppetdbquery, dnsquery and puppet ls modules.
2
Agenda• Background about our puppet setup
• How we implemented masterless
• Why we did the switch to masterless
• New workflows and future improvements
3
Background• Puppet users since 4 years
• Two large Puppet installations, 20+ Puppet masters.
• A single repository with all modules
• Git branches mapped to Puppet environments
• Code review in Gerrit to merge into production branch
• Tests using AWS virtual machines on every code review
4
Section nameSection name
ImplementationStarted with just
switching to the same workflow but masterless.
Using a custom Ruby “wrapper” around puppet apply.
5
SITE X
git mirror git mirror
SITE X
git mirror git mirror
SITE Y
PuppetDB
NodeNode
Node
Gerrit
● ServerDB● LDAP
Gerrit replication Gerrit replication
git pullgit pullfailover if pod mirrors are unavailable
Queries
* Queries* Store facts* Store catalog* Store report
hierasecrets
hiera secrets
HTTPS GET
auth by cert
Puppet CArequest
certificate
Benefits of this setup• Debug logging of compilation and hiera lookups
• Ability to get traceback of custom functions directly on the node
• Deprecation warnings from compilation visible on the node
$ sppuppet hiera resolvconf::servers 8.8.8.8 8.8.4.4
6
Drawbacks• Requires more configuration, and can’t use puppet to manage it.
• Quite specialised.
7
Secret management• Small web service where nodes authenticates using their SSL cert
• Calculates the facts for the node from the certificate
• Sends the secret hiera data that applies to that particular node
8
PuppetDB access control• Very binary per default, allow everyone or a whitelist
9
Node Apache PuppetDB
mod_ext_filter
jq
Why switch to masterless?• Scaling the Puppet masters not an issue
• But scaling the workflow is an issue
10
Growing amount of committers11
Num
ber
of m
onth
ly c
omm
itter
s
0
55
110
165
220
2 years ago 1 year ago now
Growing amount of modules12
Num
ber
of m
odul
es
0
150
300
450
600
2 years ago 1 year ago now
More complex dependencies• 126 modules using the apache module
• 91 modules using the nginx module
• 92 modules using the postgresl module
Backwards incompatible changes to shared modules almost impossible to do.
13
Flexible workflow needed• Allow one set of modules per Puppet run determined by hiera
• Easier ways to do continuous delivery of both application and configuration
• Gradual rollout of new module versions
14
Why not r10k or librarian-puppet?Both great utilities that allow building a environment dynamically. But it is still a fixed environment for many nodes.
production_new_apache production_new_apache_new_postgres production_old_apache_new_postgres production_new_apache_old_postgres
15
Our solution to the problemAt each puppet
run we look in hiera data which modules to install.
16
hiera/common.yaml: modules: spotify-hostbase: latest hiera/role/spotify_web.yaml modules: spotify-apache: latest hiera/role/puppetexplorer.yaml modules: puppetlabs-apache: latest
Internal forge• A simple forge implementation using the new V3 forge API
• Mirrors the upstream forge
• Will be used for distributing internal modules as well
github.com/jhaals/puppet-anvil
17
Questions?