2014

Presented by

The Grand Puppet Sub-Systems TourNick Fagerlund Technical Writer / Weird Bugs | Puppet Labs @nfagerlund

Presented by

PIPES

Presented by

What does Puppet DO?• Gather system information • Compile a catalog • Apply catalog • Report

Presented by

What does Puppet DO?AGENT

• Gather system information

• Apply catalog • Report

MASTER • Compile a catalog

Presented by

STOP 1: Command Line and the

Presented by

Someone ran puppet agent

Presented by

???• /opt/puppet/bin/puppet • Puppet::Util::CommandLine.new.execute • Puppet::Application.find • Puppet::Application::Agent.new • Puppet::Application::Agent#setup • Puppet::Application::Agent#run_command

Presented by

#run_command#  lib/puppet/application/agent.rb          def  run_command              if  options[:fingerprint]                  fingerprint              else                  #  It'd  be  nice  to  daemonize  later,  but  we  have  to  daemonize  before                  #  waiting  for  certificates  so  that  we  don't  block                  daemon  =  daemonize_process_when(Puppet[:daemonize])  !                wait_for_certificates  !                if  Puppet[:onetime]                      onetime(daemon)                  else                      main(daemon)                  end  

Presented by

onetime() and main()#  lib/puppet/application/agent.rb  begin              exitstatus  =  daemon.agent.run          rescue  =>  detail              Puppet.log_exception(detail)          end  !        daemon.stop(:exit  =>  false)  !        if  not  exitstatus              exit(1)          elsif  options[:detailed_exitcodes]  then              exit(exitstatus)          else              exit(0)          end  

#  lib/puppet/application/agent.rb          def  main(daemon)              if  Puppet[:listen]                  setup_listen(daemon)              end              Puppet.notice  "Starting  Puppet  client  version  #{Puppet.version}"  !            daemon.start          end  

Presented by

Obtaining the daemon#  lib/puppet/application/agent.rb          def  daemonize_process_when(should_daemonize)              daemon  =  Puppet::Daemon.new(Puppet::Util::Pidlock.new(Puppet[:pidfile]))              daemon.argv  =  @argv              daemon.agent  =  @agent  !            daemon.daemonize  if  should_daemonize  !            daemon          end  

Presented by

STOP 2: The Daemon and the

Presented by

DAEMON LAND?!

Nope Bye

Presented by

Puppet::Agent• Run a thing

Presented by

Puppet::Daemon• Run a thing that runs the other thing

Presented by

Lots of layers of abstraction??• Historical reasons!

Presented by

STOP 3: The Configurer

Presented by

Puppet::Configurer

Presented by

Puppet::Configurer#run_internal#  lib/puppet/configurer.rb      def  run_internal(options)          #  We  create  the  report  pre-­‐populated  with  default  settings  for          #  environment  and  transaction_uuid  very  early,  this  is  to  ensure          #  they  are  sent  regardless  of  any  catalog  compilation  failures  or          #  exceptions.          options[:report]  ||=  Puppet::Transaction::Report.new("apply",  nil,  @environment,  @transaction_uuid)          report  =  options[:report]          init_storage  !        Puppet::Util::Log.newdestination(report)  …  …  …

Presented by

Puppet::Configurer#run_internal#  lib/puppet/configurer.rb  …            query_options  =  get_facts(options)  unless  query_options  …  

Presented by

Puppet::Configurer#run_internal#  lib/puppet/configurer/fact_handler.rb  …            facts  =  Puppet::Node::Facts.indirection.find(Puppet[:node_name_value],  :environment  =>  @environment)  …  

Presented by

Puppet::Configurer#run_internal#  lib/puppet/configurer.rb  …            unless  catalog  =  prepare_and_retrieve_catalog(options,  query_options)              return  nil          end  …  

Presented by

Puppet::Configurer#run_internal#  lib/puppet/configurer.rb  …            apply_catalog(catalog,  options)  …  

Presented by

Where next?

Retrieve Apply

Presented by

STOP 4: The Transaction

Presented by

(Puppet::Resource::Catalog)

Presented by

Puppet::Resource::Catalog

Catalog

Resource Other Stuff

Presented by

Puppet::Configurer (again)#  lib/puppet/configurer.rb      def  convert_catalog(result,  duration)          catalog  =  result.to_ral          catalog.finalize          catalog.retrieval_duration  =  duration          catalog.write_class_file          catalog.write_resource_file          catalog      end  

Presented by

RAL Catalog

Presented by

RAL Catalog    #  lib/puppet/resource/catalog.rb      def  to_catalog(convert)          result  =  self.class.new(self.name,  self.environment_instance)          result.version  =  self.version          map  =  {}          resources.each  do  |resource|              next  if  virtual_not_exported?(resource)              next  if  block_given?  and  yield  resource              newres  =  resource.copy_as_resource              newres.catalog  =  result              if  convert  !=  :to_resource                  newres  =  newres.to_ral              end              #  We  can't  guarantee  that  resources  don't  munge  their  names              #  (like  files  do  with  trailing  slashes),  so  we  have  to  keep  track              #  of  what  a  resource  got  converted  to.              map[resource.ref]  =  newres              result.add_resource  newres          end  

Presented by

RAL CatalogPuppet::Resource

!

↓ !

(something) << Puppet::Type

Presented by

RAL Catalog

???

Presented by

RAL Catalog#  lib/puppet/configurer.rb          def  apply_catalog(catalog,  options)              report  =  options[:report]              report.configuration_version  =  catalog.version  !            benchmark(:notice,  "Finished  catalog  run")  do                  catalog.apply(options)              end  !            report.finalize_report              report          end  

Presented by

Puppet::Resource::Catalog#apply#  lib/puppet/resource/catalog.rb          def  apply(options  =  {})              Puppet::Util::Storage.load  if  host_config?  !            transaction  =  create_transaction(options)  !            begin                  transaction.report.as_logging_destination  do                      transaction.evaluate                  end              rescue  Puppet::Error  =>  detail  !

Presented by

Transaction

???  

Presented by

Transaction• Ral catalog • Prioritizer • Report

Presented by

Transaction• Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph

Presented by

Transaction• Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph • relationship_graph.traverse

Presented by

Transaction• Ral catalog • Prioritizer • Report • Event manager • Resource harness • Relationship graph • relationship_graph.traverse • *ollies out*

Presented by

STOP 5: The Indirector

Presented by

Back to Puppet::Configurer• #prepare_and_retrieve_catalog

Presented by

Back to Puppet::Configurer#  lib/puppet/configurer.rb              result  =  Puppet::Resource::Catalog.indirection.find(                  Puppet[:node_name_value],                  query_options.merge(:ignore_cache  =>  true,  :environment  =>  @environment,  :fail_on_404  =>  true)              )  

Presented by

The Indirector!

Puppet::Resource::Catalog.indirection.find()  

Presented by

The Indirector

←  ↑  ↓  ?

Presented by

The Indirector• Globally configured • Abstract • Gets stuff for you

Presented by

The Indirector  #  lib/puppet/resource/catalog.rb          extend  Puppet::Indirector          indirects  :catalog,  :terminus_setting  =>  :catalog_terminus  

Presented by

The Indirector• find • search • save • destroy

Presented by

The Indirector• find • search • save • destroy • terminus_class • cache_class • lib/puppet/indirector/…

Presented by

Puppet Agent• Catalog => REST • Report => REST • Facts => Facter

Presented by

Puppet Apply• Catalog => Compiler • Report => Processor • Facts => Facter

Presented by

Avoiding the cache#  lib/puppet/configurer.rb              result  =  Puppet::Resource::Catalog.indirection.find(                  Puppet[:node_name_value],                  query_options.merge(:ignore_cache  =>  true,  :environment  =>  @environment,  :fail_on_404  =>  true)              )  

Presented by

The Indirector• Why?

Presented by

The Indirector• Why? • Global? Rigid?

Presented by

The Indirector• Why? • Global? Rigid? • IT MAKES SENSE IN CONTEXT

Presented by

The Indirector• …Catalog.find( <FACTS> ) • REST terminus • HTTP POST request to master

Presented by

STOP 6: The Puppet Master’s

Presented by

RackWeb server

↓ Rack server

↓ “App” object that responds to #call()

↓ Actual application logic

Presented by

config.ru $0  =  "master"  !

ARGV  <<  "-­‐-­‐rack"  !

ARGV  <<  "-­‐-­‐confdir"  <<  "/etc/puppet"  ARGV  <<  "-­‐-­‐vardir"    <<  "/var/lib/puppet"  !

require  'puppet/util/command_line'  run  Puppet::Util::CommandLine.new.execute  !

Presented by

Starting master w/ Rack#  lib/puppet/application/master.rb      def  start_rack_master          require  'puppet/network/http/rack'  !

       announce_start_of_master  !

       return  Puppet::Network::HTTP::Rack.new()      end  

Presented by

The app object#  lib/puppet/network/http/rack.rb  class  Puppet::Network::HTTP::Rack    def  call(env)          request  =  Rack::Request.new(env)          response  =  Rack::Response.new          Puppet.debug  'Handling  request:  %s  %s'  %  [request.request_method,  request.fullpath]  !        begin              Puppet::Network::HTTP::RackREST.new.process(request,  response)          rescue  =>  detail              #  Send  a  Status  500  Error  on  unhandled  exceptions.              response.status  =  500              response['Content-­‐Type']  =  'text/plain'              response.write  'Internal  Server  Error:  "%s"'  %  detail.message              #  log  what  happened              Puppet.log_exception(detail,  "Puppet  Server  (Rack):  Internal  Server  Error:  Unhandled  Exception:  \"%s\""  %  detail.message)          end          response.finish  

Presented by

#process#  lib/puppet/network/http/handler.rb      def  process(request,  response)          new_response  =  Puppet::Network::HTTP::Response.new(self,  response)  !

       request_headers  =  headers(request)          request_params  =  params(request)          request_method  =  http_method(request)          request_path  =  path(request)  …  !

Presented by

#process#  lib/puppet/network/http/handler.rb              if  route  =  @routes.find  {  |route|                      route.matches?(new_request)  }                  route.process(new_request,  new_response)  

Presented by

lib/puppet/network/http/api/v1.rb

Presented by

lib/puppet/network/http/api/v1.rb• GET => indirection.find() • POST => indirection.find() (for catalogs)

• (There was an issue with some servers rejecting GET parameters that were too long. Funny story about that.)

• DELETE => indirection.destroy() • PUT => indirection.save() • GET to magical plural name => indirection.search()

Presented by

BIG PICTUREConfigurer calls catalog.indirection.find()

↓ HTTP POST request to master

↓ Request handler calls catalog.indirection.find()

↓ Compiler terminus handles it

↓

Presented by

STOP 7: The Compiler

Presented by

MOSTLY SKIPPING THISLexer ↓

Parser ↓

AST ↓

Evaluator ↓

Presented by

THIS IS THE TEST“Compiler terminus to the catalog

indirection” !

???

Presented by

THIS IS THE TEST

GOOD WORK EVERYBODY!

Presented by

Questions?

Presented by

THANKS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!•Hope we’ll see you next year in

Portland!