Building scalable applications while scaling your infrastructure by rhommel lamas

transcript

November 2013

Puppet at scale

Rhommel Lamas

Thursday, November 28, 13

Who am I?

Scale experience•Physical servers Xtratelecom.es.•AWS and other cloud providers at Wuaki.tv and 3scale.net.

Who am I?

Config Management• Worked with CFEngine2 for 2 years.• Puppet user since 2009.

Who am I?

Config Management• Worked with CFEngine2 for 2 years.• Puppet user since 2009.

Obsessed about config management and I use Emacs.

What is 3scale?3scale provides tools such as traffic

management, API administration, traffic reports, and more, to help companies

publicize their APIs easily and power new types of connected applications.

What does scaling stands for?

Scalability is the ability of a system, network, or process to handle a growing amount of

work in a capable manner or its ability to be enlarged to accommodate that growth.

Wikipedia

What does scaling stands for?

What are our scaling needs?

• How many people do we need to scale our system infrastructure?

• How many environments and systems can our team maintain?

• Can we delegate configurations easily to our team members?

Puppet dynamic environments

• Purpose of Puppet environments.

• Puppet base environments at 3scale.

/etc/puppet

auth.conf

autosign.conf

config.ru

hiera.yaml

hipchat.yaml

fileserver.conf

unicorn.rb

environments

Puppet environments at 3scaleenvironments

production

manifests

production.pp

preview.pp

production

webserver.pp

preview

webserver.pp

modules

hieradata

common.yaml

production.yaml

preview.yaml

Capfile

preview

staging

[main] server = puppet.example.com environment = production[master] environment = production manifest = $confdir/environments/$environment/manifests/site.pp modulepath = $confdir/environments/$environment/modules

[main] server = puppet.example.com environment = production[master] environment = production manifest = $confdir/environments/$environment/manifests/$environment.pp modulepath = $confdir/environments/$environment/modules

# production.pp

import 'nodes/production/nodes.pp'import 'nodes/production/*.pp

:hierarchy: - '%{::osfamily}' - '%{::environment}' - common:backends: - yaml - json:yaml: :datadir: "/etc/puppet/environments/%{environment}/hieradata"

• Puppet environments at 3scale.

• Puppet development workflow (Git).• Puppet-lint• Puppet-rspec• Code Review

Puppet at 3scale.

• Since July 2013 we are running Puppet 3.0+.

Puppet at 3scale.

• 2 Puppet master’s.

Puppet at 3scale.

• 2 Puppet master’s.• Nginx + Unicorn.

Puppet at 3scale.

• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2.

Puppet at 3scale.

• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records.

Puppet at 3scale.

• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records.• The foreman as our ENC.

Puppet at 3scale.

• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records.• The foreman as our ENC.• Puppetdb and hiera.

Puppet at 3scale.

• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records.• The foreman as our ENC.• Puppetdb and hiera.

•MCollective with Rabbitmq connector.

Puppet at 3scale.

ROUTE53

_x-puppet._tcp.3scale.net_x-puppet-ca._tcp.3scale.net

PuppetMaster

+RabbitMQ

Cluster+

MCollectiveClient

+PuppetDB

Foreman

RabbitMQ Load Balancer

PostgreSQLRDS

MultiAZ

PuppetMaster

+RabbitMQ

Cluster+

MCollectiveClient

+PuppetDB

Foreman

PuppetMaster

+RabbitMQ

Cluster+

MCollectiveClient

+PuppetDB

Foreman

Puppet Agent

+ Mcollective

Server

Puppet Agent

+ Mcollective

Server

Puppet Agent

+ Mcollective

Server

What nobody will tell you about puppet.

• Puppet-lint is your friend and it will help others to understand your code, so USE IT.

• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.

• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.• Don’t code remotely on your servers, use your local development environment and git.

• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.• Don’t code remotely on your servers, use your local development environment and git.• Read puppet documentation on every version.

• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.• Don’t code remotely on your servers, use your local development environment and git.• Read puppet documentation on every version.• Be careful about dependencies.

• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.• Don’t code remotely on your servers, use your local development environment and git.• Read puppet documentation on every version.• Be careful about dependencies.• Once you start to puppetize you become an Addict.

Rhommel LamasTwitter: @rhoml

Questions?at 3scale

we’re hiringhttp://3scale.net/jobs

Rhommel LamasTwitter: @rhoml

Thank you.at 3scale

we’re hiringhttp://3scale.net/jobs

Useful links

• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records

Useful links

• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html

Useful links

• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html• http://puppetlabs.com/blog/git-workflow-and-puppet-environments

Useful links

• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html• http://puppetlabs.com/blog/git-workflow-and-puppet-environments• http://projects.puppetlabs.com/projects/1/wiki/using_unicorn

Useful links

• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html• http://puppetlabs.com/blog/git-workflow-and-puppet-environments• http://projects.puppetlabs.com/projects/1/wiki/using_unicorn• https://github.com/cernops/puppetdb_foreman

Useful links

Building scalable applications while scaling your infrastructure by rhommel lamas

Documents