Running productioninstance 1-localcopy

transcript

Running a production Jenkins instance

Harpreet Singh, Senior Director, Product Management

Kohsuke KawaguchiJenkins founder

• Failures – a fact of life– Getting ready for failures– Preventing failures– Debugging failures

• Run an efficient Jenkins installation

Agenda

Day: A period of 24 hours, mostly misspent…

• Jenkins founder on-board• Key Jenkins contributors on-board• Built Jenkins as a Service• Run the biggest Jenkins installation

anywhere (2k+) masters

CloudBees – Who are we?

• Eliminate time wasted due to– Jenkins issues– User issues– Lack of right tools…

• Improve efficiency for administrators and developers

• Rely on Jenkins…

CloudBees’ Mission - Eliminate Downtime

• Organize jobs better• Secure your jobs• Replicate good practices• Respond quicker to requests• Ensure compliance• Bounce back from failures• Prevent failures• Everything should be as fast as

possible…if not faster

Good Management of Jenkins

Recovering from failuresHigh Availability, Backing up

Problem: Disk Failures

• JENKINS_HOME– Plugins, users, jobs…

everything

Jenkins Enterprise Solution

• Backup plugin

• Backup-to-cloud

Backing up Jenkins

Solution: Back it up

• Push HOME to a repo– HOME tends to be

large– Commit only vital info– Run nightly

• Push to S3

• Backup as a Jenkins job

• What to backup– Job configuration– Build records– System

Configuration• Plugin binaries,

plugin configs etc• Everything except

• Where to backup– Local Directory– Sftp server– WebDav

• Retention Policy– All– Last N– Exponential decay

JE Backup Plugin

Problem: Jenkins failures• Machine/Jenkins failure has

high cost to productivity

Jenkins Enterprise Solution• Highly Available

– Setup multiple Jenkins masters

– Uses jgroups to elect a primary master

– Promotes a backup master as primary

Making Jenkins Highly Available

Solution: Notified by unhappy customers ;-)• Issues:

– Receive emails from unhappy customers and log in and fix it

• You do have JENKINS_HOME backed up else where – don’t you?

Bounce Back Faster: High Availability

JENKINS_HOME

Jenkins Cluster

Jenkins Master

Reverse Proxy

JENKINS_HOMENFS

Jenkins Cluster

Jenkins Master

Reverse Proxy

• Jenkins is not just JENKINS_HOME…think about the slaves– Offload builds onto slaves – Other executables on the system: git, ruby, java etc as

well– Preferably use Chef/Puppet to replicate installations

• What about geo redundancy?– Technically you can use HA but network latency comes

in play– Ideally, use HA in a localized data center and a manual

failover to a different geo• What HA is not?

– Does not load balance between instances

Miscellaneous

Preventing failuresGit Validated Merges plugin

How can you delegate more to Jenkins?• Does your CI server shift work from

laptops to servers?– You need to commit to have Jenkins

test it– But if your commit is bad, it blocks

others– You end up testing locally before

committing– FAIL

Motivation

• We want to make changes safely– Your mistake shouldn’t block others– Only push after changes are validated

• We want to run tests asynchronously– Your brain has more important things to do–Make change and move on– Even with TDD!

• We want to run tests on the server– Your laptop has more important things to

Solution: Jenkins should be Git server• I push to Jenkins• Jenkins merges it with upstream• Jenkins tests it• If good, Jenkins pushes it upstream

upstreamrepo

gate repo

Another way to look at it

Tip of master in upstream

My changes

Tip of master in upstream

Implementation

• Transport– HTTP– SSH

• JGit embedded in Jenkins for git server functionality– A bit of magic like Gerrit to make it

seamless

• Additional tags to let you pull submitted changes

Running an efficient production system

• Run mini 2nd instance– Test new core version before putting it

to prod– Test new versions of plugins– Play with new plugins

• Copy over some jobs from prod

• Bootstrap dry-run– -

Djenkins.model.Jenkins.killAfterLoad=true

Test Instance

• Fast archiver plugin– Conserve network bandwidth

• No build on master– Also good for security

Configuring Jenkins for efficiency

Problem: Discovering what plugins are used in an installation

• No visibility if a particular plugin is used or how many jobs use it

Jenkins Enterprise Solution• Plugin Usage Plugin

– Tabular view of Plugin name, # of jobs and the job names using the plugin

Managing and Pruning Plugins

Monitoring Jenkins

• What the user sees– GUI (load time)

• JVM memory size– Beware of several independent pieces

• System load• Free space on $JENKINS_HOME• Slave availability• Queue length

Groovy Console

$ cat queue.groovyj=Jenkins.instancesprintln j.queue.items.length

$ curl –u "user:apiToken“ \ –data-urlencode script@queue.groovy \ http://jenkins/scriptText13

Remote API

$ curl http://jenkins/computer/api/json?pretty=true{ busyExecutors: 0, totalExecutors: 2, ...}

• JavaMelody in Jenkins

Jenkins Monitoring plugin

• Server app for monitoringstuff– Extensible, allowing all sorts

of things to be monitored

• Used in jenkins-ci.org/DEV@cloud

Nagios (or others like it)

• Tells us where Jenkins is stuck• When?– Hang or slowness

• Look for threads that’s stuck– HTTP request threads– Executor threads

Thread dump

• http://jenkins/threadDump• kill -3 <PID>

How to get a thread dump

• Tells us what’s eating memory• When?– OutOfMemoryError–Monitoring shows abnormal growth

• Look for objects that are big– Sessions– Classes from plugins

Heap dump

• curl –L http://jenkins/heapDump > dump.hprof

• jmap -dump:format=b,file=dump.hprof PID

• -XX:+HeapDumpOnOutOfMemoryError

How to get a memory dump

More Info

Free Trial

Wiki Page

User Guide

http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-overview.cb

http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-download.cb

https://wiki.cloudbees.com/bin/view/Jenkins+Enterprise/WebHome

http://jenkins-enterprise.cloudbees.com/docs/user-guide-bundle/index.html#

Thank You!

Wrapping up

Day: A period of 24 hours, mostly misspent…

Running productioninstance 1-localcopy

Documents