Large-Scale Deployments With Pogo At Yahoo!

Mike SchilliYAPC::NA 06/14/2012

What’s Pogo?

• Yahoo’s deployment system– Highly scalable (thousands of targets)– Highly parallelized rollouts– In active use for several years

• Open Source version available (pre-alpha)

Pogo in a CI environment

What’s “Deployment”?

• Run one command on many hosts• Each host “knows” what it’s supposed to look

like via a central database• One command brings host into “desired” state• Fetches package updates from edge-server

repos

Run one command on many hosts

• Act on all hosts immediately

Run one command on many hosts

• Act on all hosts immediately

Example Server Farm

Pogo Features

• Sequences• Parallelizes with constraints• Halts if errors violate constraints• Runs health checks (pre/post hooks)• Command agnostic (rpm, apt-get, custom,

etc.)

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Server Farm Release via Pogo

Pogo Configuration

Run a Pogo job (tags)

Pogo UI – Single Job View

Pogo UI Overview

Pogo UI – Host View

Pogo UI – Start it Up

$ perl -Ilib bin/pogo-apipogo-api-28> Listening on 0.0.0.0:7657

$ perl -Ilib bin/pogo-ui -vpogo-ui-22> Listening on port 5000

Run a Pogo job (targets)

Pogo Configuration

State of Pogo

• Pre-alpha Open Source version on Github

• OSS Pogo will be completed in the coming months

Pogo on Github

• Developed in the open:– https://github.com/ytoolshed/pogo

• Internally used with plugins to adapt to specific data sources

CI Tests with travis-ci.org

The Gory Details

Pogo – User View

Pogo Workflow

Authentication/Authorization

• Web server authentication for submitting jobs• Target authentication via passwords or

priv/pub keys• Credentials encrypted with worker pubkey• Credentials never stored persistently (or

ZooKeeper)

Pogo “Hooks”

• Scripts that run before or after Pogo commands

• Used to prepare host, perform check health, or other functions

• Installed locally on target hosts• rc.d style invocation: run in alphanumeric

order

Pogo “Hooks” (cont.)

• Two types:– pre hook: run before the specified command– post hook: run after the specified command

• If any hook script fails:– No further scripts or commands run– Host is marked as failed

Pre-hook Examples

• Take a host out of rotation• Drain database connections• Check that host is ready to be updated

Post-hook Examples

• Check whether local webserver is serving expected content

• Ping required services• Put host back into rotation

Host Failures and Concurrency

• If a host fails:– Command returns non-zero– Pre- or post-hook returns non-zero

• The failed host counts against the acceptable number/percent of unavailable hosts

• Example:– Four hosts in an app can run at a time– One host in the app fails– Only three hosts will be acted on at a time going forward

Implementation

• AnyEvent Framework• Asynchronous, event-based• Added Object::Event for event pub/sub

HTTP Client in AnyEvent

HTTP Client in AnyEvent

HTTP Client in AnyEvent

AnyEvent and Object::Event

AnyEvent and Object::Event

AnyEvent and Object::Event

Queue Processor with AnyEvent and Object::Event

AnyEvent

bin/pogo-dispatcher

Testing AnyEvent Components

All Pogo Components

All-In-One Pogo: pogo-one

All-In-One Pogo: pogo-one

All-In-One Pogo: pogo-one

Questions?

Thanks!

• Pogo on Github: – http://github.com/ytoolshed/pogo

Lessons learned along the Way

Set up Unit Tests with Travis-ci.org

Runs immediately after Github checkins

CI Tests with travis-ci.org

Configure CI with travis-ci.org

Select Github project

Authorize travis-ci.org

Travis-ci.org API Calls

Filling in Passwords

Filling in Passwords

• Test logging into machines without sshkeys• Convenient for testing• Answer to sudo prompts on targets• Answer to prompts for encrypted packages

Filling in Passwords

• Good password prompt:

mschilli@localhost's password:

• Not password prompt!

… user ''@'localhost' (using password: NO)

PasswordMonkey on CPAN

Plack

Plack

Plack/PSGI

• Plack used for server-independent web application– API– Dispatcher Control Port

Alternatives

• Capistrano• Rundeck• Aegir• Drush

Pogo UI – Start it Up

$ perl -Ilib bin/pogo-apipogo-api-28> Listening on 0.0.0.0:7657

$ perl -Ilib bin/pogo-ui -vpogo-ui-22> Listening on port 5000

Pogo UI

• All Cient Side-Javascript• Server just redirects to index.html– Except assets/* js/*

Thanks!

• Pogo on Github: – http://github.com/ytoolshed/pogo

• PasswordMonkey CPAN Module:– http://search.cpan.org/dist/PasswordMonkey/

• Travis CI Service: – http://travis-ci.org