Post on 19-Apr-2020
transcript
Continuous Integration of The FreeBSD Project
Li-Wen HsuThe FreeBSD ProjectlwhsuFreeBSDorg
Abstract
The FreeBSD project [22]rsquos continuous integrationproject started in the late 2013 We use Jenkins [17] au-tomation server to build our continuous integration sys-tem It monitors the svn repository for new commitsand triggers a new build of it In each build the buildmachine compiles the latest code creates disk imageand creates a virtual machine to run test suite In themeantime we collect the compiler warnings and performsome further checks like clang analyzer All these infor-mation are published to the developers and users to im-prove the quality of the FreeBSD project In this paperwe describe the details of the system implementation
1 Introduction
The term of continuous integration (CI) is first used byGrady Booch in 1991 [1] then Extreme programming(XP) extends the concept of CI and suggests to integrateas many times as possible per day Now CI is an impor-tant and common practice of software engineering Thebenefits of CI are developers can know the conflicts andbugs earlier This reduces the cost for resolving themlater Also the status of the project is clearer and it iseasier to estimate progress
There are many software designed for various CI needamong them the most popular one is Jenkins an opensource automation server written in Java created byKohsuke Kawaguchi We use Jenkins as the main com-ponent along with some other daemons like Pure-FTPd[20] to build FreeBSD projectrsquos CI system
We describe the history of FreeBSDrsquos CI work howwe trigger build from changes in version control systemdo tests and publish results in the rest of this paper
2 Automatic Build and History of CI workin FreeBSD
Dag-Erling Smrgrav created the first pubic CI sys-tem for the FreeBSD project httptinderboxFreeBSDorg this can be traced back in the mailarchive in March 2002 It is believed the first contin-uous integration service of the FreeBSD project Tin-derbox is an open source project codes are availableat svnsvnfreebsdorgbaseuserdestinderbox Documents can be found in FreeBSDdoc project [24] and wiki [25] Unfortunately this ser-vice stopped in September or October 2014 neverthe-less here are many scripts from this project derived orinspiring the successive projects
In late 2013 Craig Rodrigues founded jenkins-adminFreeBSDorg with several developers Themain duty of jenkins-admin is maintaining the Jenk-ins instance in FreeBSD cluster httpsjenkinsFreeBSDorg Which is an independent successorof tinderboxFreeBSDorg We configure it monitorsthe FreeBSD src and doc repositories and triggers anew build when there is new commit in each supportedbranch We created the first ldquobuild-and-test pipelinerdquofor the FreeBSD project In each build triggered by achange in repository after compiling successes we cre-ate a virtual machine disk image with the latest binariesand spawn a new virtual machine to run tests The testresults are collected and stored by Jenkins server
The highlight features of jenkinsFreeBSDorg are
bull Periodically build for head stable-11 stable-10 andstable-9
bull Architectures built against amd64 i386 sparc64and arm64
bull Build FreeBSD with GCC using amd64-xtoolchain-gcc
bull Experimental Jenkins 2 ldquopipeline as coderdquo
Shortly jenkinsFreeBSDorg become a significantrole in the FreeBSD development
In 2016 with the hardwares sponsored by theFreeBSD foundation we created httpsciFreeBSDorg an experimental site for testing newideas We still leverage Jenkins because it has provenuseful on jenkinsFreeBSDorg Aditionally we imple-mented
bull Artifact server
bull Job template
Having a centralized artifact server enables people re-trieving the files generated from each stage of a pipelinewhich can be used for reproducing issues for debuggingpurpose or doing further tests The details about jobtemplate are in ldquoopen configurationsrdquo section
Another experiment in this system is we try to makethe build pipeline more fine-grained and introducedmore build stages For example we not only create vir-tual machine disk image specialized for tests but also tryto produce the distribution files and virtual machine diskimages the same as the official release provided by re-lease engineering team Also we build LINT kernel formore test coverage
There are multiple stages in a pipelne each is a Jenk-ins job The naming conversion of the build jobs is
FreeBSD-branch-target_arch-stage
We create a pipeline for each supported branch and ar-chitecture Take head branch and amd64 architecture forexample
bull FreeBSD-head-amd64-buildBuild the world and kernel from scratch and createsdistribution files
bull FreeBSD-head-amd64-LINTBuild the LINT kernel
bull FreeBSD-head-amd64-imagesBuild virtual machine disk images from distributionfiles with the same configuration of official release
bull FreeBSD-head-amd64-testvmBuild virtual machine disk image from distributionfiles and test files and package are pre-installed
bull FreeBSD-head-amd64-testRun the image from ldquotestvmrdquo stage in a new createdvirtual machine
Their relationship is shown in 1The codes are available at httpsgithub
comlwhsufreebsd-ci and httpsgithubcomlwhsujjb-freebsd-cimerging back to the main freebsd-ci repository on githubor even svnsvnfreebsdorg is in progress
3 Self Tests
In 2013 Rui Paulo created the usrtests hierar-chy with the Kyua [18] test framework by Julio MerinoThese tests can be performed by command
cd usrtests ampamp kyua test
As of January 2017 we have 6023 tests for head branchon amd64 architecture
As previous section mentioned we build virtual ma-chine disk images with the required files and packagesfor running tests We pre-install the kyua package andput a script which runs kyua commands and generatesthe test reports after all tests are executed Finally it is-sues shutdown command After the disk image readyfor amd64 and i386 architecture we launch a bhyve [27]virtual machine with an expect script for starting run-ning tests in and also for catching timeout in case
After the testing virtual machine stopped we extractedthe test results from the disk image put to the workspacewhere Jenkins slave daemon will collect them and sendback to the Jenkins master for archiving and maintaininga trend record The script to create test virtual machineimage is scriptsbuildbuild-test_imagesh and the main script for executing tests in virtualmachines and extracting results is scriptstestrun-testssh
4 Access to Results
We configure Kyua to generate results as JUnit [4] XMLformat which is natively supported by Jenkins and manyother tools Jenkins default plugins take outputs fromthe builds make it a friendly accessible web inter-face at httpsltjob-urlgtltbuild-numbergttestReport (fig 2 3) and maintains a historicaldata (fig 4) This helps us tracking the software qualitytrending
The artifact server artifactciFreeBSDorg is newlyintroduced in ciFreeBSDorg its main responsibility isto share files between jobs in a pipeline The artifactserver is basically an FTP server powered by Pure-FTPdfor serving files uploading For downloading we use anginx [28] web server to provide read-only access Forsecurity reason we configure FTP server can only be ac-cessed by FTP over TLS As a result we also added FTPover TLS feature in ldquoPublish Over FTP Pluginrdquo [] ofJenkins
In the end of each stage Jenkins uploads createdfiles to the artifact server there is only one root di-rectory ldquosnapshotsrdquo on artifactciFreeBSDorg for nowThe directory layout of it is branchsvn_revisiontargettarget_arch for ex-ample headr303226amd64amd64 This layout
2
FreeBSD-head-amd64-build
FreeBSD-head-amd64-testvm
FreeBSD-head-amd64-LINT
FreeBSD-head-amd64-images
FreeBSD-head-amd64-test
Figure 1 head-amd64 pipeline
Figure 2 Test result from a test job
Figure 3 Failed test case display
3
Figure 4 Test result trend
is compatible with the official download site We hopethat in the future these snapshot artifacts can also be in-cluded and be more easily accessed by users
Currently we have these artifacts for each pipeline
bull txz
ndash From -build jobs
ndash Distribution files just like what on httpsdownloadFreeBSDorg
bull diskimgxz
ndash From -images jobs
ndash Virtual machine disk image file installed fromabove txz files
bull disk-testimgxz
ndash From -testvm jobs
ndash Virtual machine disk image file with testscript and test case
All files are publicly available at httpartifactcifreebsdorg
5 Open Configurations
It is very important to make public can setup a highlyidentical instance of a public service and security isalso need to be considered In the past Jenkins jobcan only be edited on web interface which is not easy
to tracking the changes and reproduce Users whohas no administrative permission cannot see exactlyconfigurations Jenkins 2rsquos ldquopipeline as coderdquorsquo isa very promising to solve this problem On jenk-insFreeBSDorg we have tested it for the jobs whichbuild head and stable-10 branch The source code isavailable at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsbuildbuild-testgroovy However currently not allthe required Jenkins plugins of ciFreeBSDorg supportsthis We will look into this again and may switch it later
We use an approach used by other project for a longtime Jenkins job builder [10] which converts YAMLfiles to Jenkins XML configurations And manipulateJenkins instance via web service With Jenkins jobbuilder the configuration of Jenkins is trackable and re-producible and we can separate security credentials toother storage
While working with other developers we found thefact that not everyone knows (and has to know) how toconfigure Jenkins So adding new build jobs requiresJenkins administrators configure and test build scripts byhand This is a bottleneck of the development We alsofound that most of the build jobs are just having differ-ent build scripts other parts of the configurations such asrepository URL and notification settings are the same Itinspired us to create a ldquojob templaterdquo for similar jobs Atypical job contains following steps
1 Environment setup
(a) Check out latest source code
4
(b) Setup needed version of FreeBSD
(c) Install required packages
2 Execute specified build script for that job
3 Environment cleanup
The respective job definition (of Jenkins job builder)is very simple (listing 1)
Listing 1 Job definition- job
name FreeBSD-head-scan_buildscm
- FreeBSD-src-headbuilders
- checkout-scripts- setup-jail- execute-in-jail
publishers- clean-jail
wrappers- timestamps
And we use jail to ensure the freshness of the buildenvironment and we can install ldquosudordquo [21] program in-side jail with full privileges granted for maximum thethings user can do for testing As a result ldquobuild-in-jail script setsrdquo is created they are currently developedat httpsgithubcomlwhsufreebsd-citreejjbscriptsjail Itrsquos first job is simplifyenvironment setup and job execution we have threescripts for each builder and publishers in definition ofJenkins job builder
bull setupshsetup-jail builder setup a jail jail and install requirepackages
bull executeshexecute-in-jail builder execute build scripts fromuser
bull cleanshclean-jail publisher clean the used jail to release re-source
A typical job configuration contains three files
bull buildshMain build script which is executed by execueshin the jail
bull jailconfJail environment configuration version arch etc
bull pkg-listPackage needs to be installed (from pkgFreeBSDorg)
With these adding a new job is much simplified Sub-mitter can just test their tested build script (buildsh)with some job definitions (jailconf pkg-list) and otherconfiguration files for example srcconf or makeconfThen jenkins-admin merges scripts to freebsd-cijobsltjobnamegt and jenkins-admin creates a newjob entry in jenkins job builder push the new configu-ration to the Jenkins server Everything is done OurJenkins server will execute it around the clock
For example FreeBSD-head-arm64-build containsfiles as listing 2 3 4
Listing 2 Job configuration jailconfTARGET=amd64TARGET_ARCH=amd64WITH_32BIT=0OSRELEASE=103-RELEASE
Listing 3 Job configuration pkg-listaarch64-binutils
Listing 4 Job configuration buildshbinshenv JFLAG=$BUILDER_JFLAG TARGET=arm64 TARGET_ARCH=aarch64 sh -x freebsd-ciscriptsbuildbuild-world-kernelsh
We also add a ldquoquarantine moderdquo of ldquobuild-in-jail script setsrdquo just specify ldquoQUARANTINE=1rdquo injailconf the the internet connection is removed after jailenvironment is setup and resource is also strictly limitedsuch as execution time
The other ciFreeBSDorg setup can be foundat httpswikifreebsdorgJenkinsSetup
6 Other Integrations
The best feature of Jenkins is its features can be extendedby many plugins we selected several plugins which con-sidered useful for us and integrated into our system
61 NotificationMake sure related developers can know the latest projectstatus is very important We use both Email and IRCnotifications The status for sending out notifications are
bull Build failed
bull Build unstable (compile successfully but some testcases failed)
bull Build back to stable
5
And for notification email we attach following infor-mation
bull Changes since last build (who amp what)
bull Tail of the console output
bull What are the test cases failed
bull Related URLs
Email notification is considered a ldquodouble-edgedswordrdquo once we send out too many mails and peo-ple will consider them as spam If nobody readsthe notification mails and takes action it is totallyno use So use carefully created a pre-send scriptwith ldquoEmail-ext pluginrdquo [7] for filter out the internalerrors of the jenkins cluster to reduce false alertThe script is available at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsemail-extpre-sendgroovy
62 Code Review SystemThe FreeBSD project owns a code review systemat httpsreviewsFreeBSDorg whichis setup with Phabricator [19] and jenkins has aldquoPhabricator Differential Pluginrdquo [11] plugin Itcurrently supports only git we patched it to supportsubversion httpsgithubcomlwhsuphabricator-jenkins-plugintreescmand are also preparing to upstream it
On ciFreeBSDorg we have two jobs
bull FreeBSD-head-amd64-build-phabric
bull FreeBSD-doc-head-igor-phabric
They are using ldquoquarantine moderdquo of the ldquobuild-in-jailscript setsrdquo and are triggered when a new patch is up-loaded to reviewsFreeBSDorg (fig 5)
And reports the build result back to reviewsFreeBSDorg (fig 6)
63 Clang Static AnalyzerWe created a ldquoclang scan-buildrdquo job with ClangStatic Analyzer [3] and the build script leveraged theldquobsdclang-analyzedmkrdquo from NetBSD [23] simplybuild the whole world and kernel with command in list-ing 5
Listing 5 Command to perform clang scan-buildmake analyze CLANG_ANALYZE_OUTPUT_DIR=clangScanBuildReports CLANG_ANALYZE_OUTPUT=html
The result is taken cared by ldquoClang Scan-Build Plu-ginrdquo [6] which generates good web interface and historytracking (fig 7 8 9)
64 Igor and CheckstyleFor document build we use igor [26] by Warren Blockand Checkstyle [2] for checking the style of documentXML We worked with Warren and added a new option(-X) to igor to output checkstyle XML The respectivejob is FreeBSD-doc-head-igor ldquoCheckstyle Pluginrdquo [5]of Jenkins also provides good web interface and historytracking (fig 10 11 12)
65 Compiler WarningsWe also collect compiler warnings with ldquoWarnings Plu-ginrdquo [16] For showing the trend of compiler warningsand this pushes developers produce higher quality code(fig 13 14)
66 Build Status BadgeWe provide build status badge for embedding in externalweb pages through ldquoEmbeddable Build Status Pluginrdquo[8] For example on httpswikiFreeBSDorgarm64 we have badge as fig 15
To show the latest status of each branch The badgeURLs are
bull httpsltjob-urlgtbadgeicon
bull httpsltjoburlgtlastCompletedBuildbadgeicon
67 OthersWe also use some other plugins for improving the inter-face and helping the maintenance task
bull SCM Sync configuration [14]
ndash Use post-commit hook to notify administra-tors
bull SafeRestart [15]
ndash Schedule restart jenkins when all current jobscomplete
ndash Save and resume the build queue
bull Green balls [9]
ndash To replace blue status ball with green ones tofulfill the convention for most culture
7 Conclusion
In this paper we described about the history and howwe setup the latest CI system of the FreeBSD projectCI does help the development of the FreeBSD projectand we hope these experience and codes can also helpdownstream projects
6
Figure 5 Build triggerd by a patch
Figure 6 Rsult of a build triggerd by a patch
7
Figure 7 Clang scan-build result
8 Future Work
The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas
bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo
bull Check for reproducible build
bull More tests even some are not enabled by defaultDtrace
bull Build for project branchesMake testing feature branches easier
bull Performance tests
bull Work with other projects
We hope these work can make the quality of FreeBSDeven better
AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware
bull jenkins-adminFreeBSDorg
bull The FreeBSD Foundation
bull clusteradminFreeBSDorg
bull phabic-adminFreeBSDorg
bull People on -testing -current -stable mailing lists
References
[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991
[2] Checkstylehttpcheckstylesourceforgenet
[3] Clang Static Analyzerhttpsclang-analyzerllvmorg
[4] JUnithttpjunitorg
[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin
8
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
Shortly jenkinsFreeBSDorg become a significantrole in the FreeBSD development
In 2016 with the hardwares sponsored by theFreeBSD foundation we created httpsciFreeBSDorg an experimental site for testing newideas We still leverage Jenkins because it has provenuseful on jenkinsFreeBSDorg Aditionally we imple-mented
bull Artifact server
bull Job template
Having a centralized artifact server enables people re-trieving the files generated from each stage of a pipelinewhich can be used for reproducing issues for debuggingpurpose or doing further tests The details about jobtemplate are in ldquoopen configurationsrdquo section
Another experiment in this system is we try to makethe build pipeline more fine-grained and introducedmore build stages For example we not only create vir-tual machine disk image specialized for tests but also tryto produce the distribution files and virtual machine diskimages the same as the official release provided by re-lease engineering team Also we build LINT kernel formore test coverage
There are multiple stages in a pipelne each is a Jenk-ins job The naming conversion of the build jobs is
FreeBSD-branch-target_arch-stage
We create a pipeline for each supported branch and ar-chitecture Take head branch and amd64 architecture forexample
bull FreeBSD-head-amd64-buildBuild the world and kernel from scratch and createsdistribution files
bull FreeBSD-head-amd64-LINTBuild the LINT kernel
bull FreeBSD-head-amd64-imagesBuild virtual machine disk images from distributionfiles with the same configuration of official release
bull FreeBSD-head-amd64-testvmBuild virtual machine disk image from distributionfiles and test files and package are pre-installed
bull FreeBSD-head-amd64-testRun the image from ldquotestvmrdquo stage in a new createdvirtual machine
Their relationship is shown in 1The codes are available at httpsgithub
comlwhsufreebsd-ci and httpsgithubcomlwhsujjb-freebsd-cimerging back to the main freebsd-ci repository on githubor even svnsvnfreebsdorg is in progress
3 Self Tests
In 2013 Rui Paulo created the usrtests hierar-chy with the Kyua [18] test framework by Julio MerinoThese tests can be performed by command
cd usrtests ampamp kyua test
As of January 2017 we have 6023 tests for head branchon amd64 architecture
As previous section mentioned we build virtual ma-chine disk images with the required files and packagesfor running tests We pre-install the kyua package andput a script which runs kyua commands and generatesthe test reports after all tests are executed Finally it is-sues shutdown command After the disk image readyfor amd64 and i386 architecture we launch a bhyve [27]virtual machine with an expect script for starting run-ning tests in and also for catching timeout in case
After the testing virtual machine stopped we extractedthe test results from the disk image put to the workspacewhere Jenkins slave daemon will collect them and sendback to the Jenkins master for archiving and maintaininga trend record The script to create test virtual machineimage is scriptsbuildbuild-test_imagesh and the main script for executing tests in virtualmachines and extracting results is scriptstestrun-testssh
4 Access to Results
We configure Kyua to generate results as JUnit [4] XMLformat which is natively supported by Jenkins and manyother tools Jenkins default plugins take outputs fromthe builds make it a friendly accessible web inter-face at httpsltjob-urlgtltbuild-numbergttestReport (fig 2 3) and maintains a historicaldata (fig 4) This helps us tracking the software qualitytrending
The artifact server artifactciFreeBSDorg is newlyintroduced in ciFreeBSDorg its main responsibility isto share files between jobs in a pipeline The artifactserver is basically an FTP server powered by Pure-FTPdfor serving files uploading For downloading we use anginx [28] web server to provide read-only access Forsecurity reason we configure FTP server can only be ac-cessed by FTP over TLS As a result we also added FTPover TLS feature in ldquoPublish Over FTP Pluginrdquo [] ofJenkins
In the end of each stage Jenkins uploads createdfiles to the artifact server there is only one root di-rectory ldquosnapshotsrdquo on artifactciFreeBSDorg for nowThe directory layout of it is branchsvn_revisiontargettarget_arch for ex-ample headr303226amd64amd64 This layout
2
FreeBSD-head-amd64-build
FreeBSD-head-amd64-testvm
FreeBSD-head-amd64-LINT
FreeBSD-head-amd64-images
FreeBSD-head-amd64-test
Figure 1 head-amd64 pipeline
Figure 2 Test result from a test job
Figure 3 Failed test case display
3
Figure 4 Test result trend
is compatible with the official download site We hopethat in the future these snapshot artifacts can also be in-cluded and be more easily accessed by users
Currently we have these artifacts for each pipeline
bull txz
ndash From -build jobs
ndash Distribution files just like what on httpsdownloadFreeBSDorg
bull diskimgxz
ndash From -images jobs
ndash Virtual machine disk image file installed fromabove txz files
bull disk-testimgxz
ndash From -testvm jobs
ndash Virtual machine disk image file with testscript and test case
All files are publicly available at httpartifactcifreebsdorg
5 Open Configurations
It is very important to make public can setup a highlyidentical instance of a public service and security isalso need to be considered In the past Jenkins jobcan only be edited on web interface which is not easy
to tracking the changes and reproduce Users whohas no administrative permission cannot see exactlyconfigurations Jenkins 2rsquos ldquopipeline as coderdquorsquo isa very promising to solve this problem On jenk-insFreeBSDorg we have tested it for the jobs whichbuild head and stable-10 branch The source code isavailable at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsbuildbuild-testgroovy However currently not allthe required Jenkins plugins of ciFreeBSDorg supportsthis We will look into this again and may switch it later
We use an approach used by other project for a longtime Jenkins job builder [10] which converts YAMLfiles to Jenkins XML configurations And manipulateJenkins instance via web service With Jenkins jobbuilder the configuration of Jenkins is trackable and re-producible and we can separate security credentials toother storage
While working with other developers we found thefact that not everyone knows (and has to know) how toconfigure Jenkins So adding new build jobs requiresJenkins administrators configure and test build scripts byhand This is a bottleneck of the development We alsofound that most of the build jobs are just having differ-ent build scripts other parts of the configurations such asrepository URL and notification settings are the same Itinspired us to create a ldquojob templaterdquo for similar jobs Atypical job contains following steps
1 Environment setup
(a) Check out latest source code
4
(b) Setup needed version of FreeBSD
(c) Install required packages
2 Execute specified build script for that job
3 Environment cleanup
The respective job definition (of Jenkins job builder)is very simple (listing 1)
Listing 1 Job definition- job
name FreeBSD-head-scan_buildscm
- FreeBSD-src-headbuilders
- checkout-scripts- setup-jail- execute-in-jail
publishers- clean-jail
wrappers- timestamps
And we use jail to ensure the freshness of the buildenvironment and we can install ldquosudordquo [21] program in-side jail with full privileges granted for maximum thethings user can do for testing As a result ldquobuild-in-jail script setsrdquo is created they are currently developedat httpsgithubcomlwhsufreebsd-citreejjbscriptsjail Itrsquos first job is simplifyenvironment setup and job execution we have threescripts for each builder and publishers in definition ofJenkins job builder
bull setupshsetup-jail builder setup a jail jail and install requirepackages
bull executeshexecute-in-jail builder execute build scripts fromuser
bull cleanshclean-jail publisher clean the used jail to release re-source
A typical job configuration contains three files
bull buildshMain build script which is executed by execueshin the jail
bull jailconfJail environment configuration version arch etc
bull pkg-listPackage needs to be installed (from pkgFreeBSDorg)
With these adding a new job is much simplified Sub-mitter can just test their tested build script (buildsh)with some job definitions (jailconf pkg-list) and otherconfiguration files for example srcconf or makeconfThen jenkins-admin merges scripts to freebsd-cijobsltjobnamegt and jenkins-admin creates a newjob entry in jenkins job builder push the new configu-ration to the Jenkins server Everything is done OurJenkins server will execute it around the clock
For example FreeBSD-head-arm64-build containsfiles as listing 2 3 4
Listing 2 Job configuration jailconfTARGET=amd64TARGET_ARCH=amd64WITH_32BIT=0OSRELEASE=103-RELEASE
Listing 3 Job configuration pkg-listaarch64-binutils
Listing 4 Job configuration buildshbinshenv JFLAG=$BUILDER_JFLAG TARGET=arm64 TARGET_ARCH=aarch64 sh -x freebsd-ciscriptsbuildbuild-world-kernelsh
We also add a ldquoquarantine moderdquo of ldquobuild-in-jail script setsrdquo just specify ldquoQUARANTINE=1rdquo injailconf the the internet connection is removed after jailenvironment is setup and resource is also strictly limitedsuch as execution time
The other ciFreeBSDorg setup can be foundat httpswikifreebsdorgJenkinsSetup
6 Other Integrations
The best feature of Jenkins is its features can be extendedby many plugins we selected several plugins which con-sidered useful for us and integrated into our system
61 NotificationMake sure related developers can know the latest projectstatus is very important We use both Email and IRCnotifications The status for sending out notifications are
bull Build failed
bull Build unstable (compile successfully but some testcases failed)
bull Build back to stable
5
And for notification email we attach following infor-mation
bull Changes since last build (who amp what)
bull Tail of the console output
bull What are the test cases failed
bull Related URLs
Email notification is considered a ldquodouble-edgedswordrdquo once we send out too many mails and peo-ple will consider them as spam If nobody readsthe notification mails and takes action it is totallyno use So use carefully created a pre-send scriptwith ldquoEmail-ext pluginrdquo [7] for filter out the internalerrors of the jenkins cluster to reduce false alertThe script is available at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsemail-extpre-sendgroovy
62 Code Review SystemThe FreeBSD project owns a code review systemat httpsreviewsFreeBSDorg whichis setup with Phabricator [19] and jenkins has aldquoPhabricator Differential Pluginrdquo [11] plugin Itcurrently supports only git we patched it to supportsubversion httpsgithubcomlwhsuphabricator-jenkins-plugintreescmand are also preparing to upstream it
On ciFreeBSDorg we have two jobs
bull FreeBSD-head-amd64-build-phabric
bull FreeBSD-doc-head-igor-phabric
They are using ldquoquarantine moderdquo of the ldquobuild-in-jailscript setsrdquo and are triggered when a new patch is up-loaded to reviewsFreeBSDorg (fig 5)
And reports the build result back to reviewsFreeBSDorg (fig 6)
63 Clang Static AnalyzerWe created a ldquoclang scan-buildrdquo job with ClangStatic Analyzer [3] and the build script leveraged theldquobsdclang-analyzedmkrdquo from NetBSD [23] simplybuild the whole world and kernel with command in list-ing 5
Listing 5 Command to perform clang scan-buildmake analyze CLANG_ANALYZE_OUTPUT_DIR=clangScanBuildReports CLANG_ANALYZE_OUTPUT=html
The result is taken cared by ldquoClang Scan-Build Plu-ginrdquo [6] which generates good web interface and historytracking (fig 7 8 9)
64 Igor and CheckstyleFor document build we use igor [26] by Warren Blockand Checkstyle [2] for checking the style of documentXML We worked with Warren and added a new option(-X) to igor to output checkstyle XML The respectivejob is FreeBSD-doc-head-igor ldquoCheckstyle Pluginrdquo [5]of Jenkins also provides good web interface and historytracking (fig 10 11 12)
65 Compiler WarningsWe also collect compiler warnings with ldquoWarnings Plu-ginrdquo [16] For showing the trend of compiler warningsand this pushes developers produce higher quality code(fig 13 14)
66 Build Status BadgeWe provide build status badge for embedding in externalweb pages through ldquoEmbeddable Build Status Pluginrdquo[8] For example on httpswikiFreeBSDorgarm64 we have badge as fig 15
To show the latest status of each branch The badgeURLs are
bull httpsltjob-urlgtbadgeicon
bull httpsltjoburlgtlastCompletedBuildbadgeicon
67 OthersWe also use some other plugins for improving the inter-face and helping the maintenance task
bull SCM Sync configuration [14]
ndash Use post-commit hook to notify administra-tors
bull SafeRestart [15]
ndash Schedule restart jenkins when all current jobscomplete
ndash Save and resume the build queue
bull Green balls [9]
ndash To replace blue status ball with green ones tofulfill the convention for most culture
7 Conclusion
In this paper we described about the history and howwe setup the latest CI system of the FreeBSD projectCI does help the development of the FreeBSD projectand we hope these experience and codes can also helpdownstream projects
6
Figure 5 Build triggerd by a patch
Figure 6 Rsult of a build triggerd by a patch
7
Figure 7 Clang scan-build result
8 Future Work
The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas
bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo
bull Check for reproducible build
bull More tests even some are not enabled by defaultDtrace
bull Build for project branchesMake testing feature branches easier
bull Performance tests
bull Work with other projects
We hope these work can make the quality of FreeBSDeven better
AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware
bull jenkins-adminFreeBSDorg
bull The FreeBSD Foundation
bull clusteradminFreeBSDorg
bull phabic-adminFreeBSDorg
bull People on -testing -current -stable mailing lists
References
[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991
[2] Checkstylehttpcheckstylesourceforgenet
[3] Clang Static Analyzerhttpsclang-analyzerllvmorg
[4] JUnithttpjunitorg
[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin
8
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
FreeBSD-head-amd64-build
FreeBSD-head-amd64-testvm
FreeBSD-head-amd64-LINT
FreeBSD-head-amd64-images
FreeBSD-head-amd64-test
Figure 1 head-amd64 pipeline
Figure 2 Test result from a test job
Figure 3 Failed test case display
3
Figure 4 Test result trend
is compatible with the official download site We hopethat in the future these snapshot artifacts can also be in-cluded and be more easily accessed by users
Currently we have these artifacts for each pipeline
bull txz
ndash From -build jobs
ndash Distribution files just like what on httpsdownloadFreeBSDorg
bull diskimgxz
ndash From -images jobs
ndash Virtual machine disk image file installed fromabove txz files
bull disk-testimgxz
ndash From -testvm jobs
ndash Virtual machine disk image file with testscript and test case
All files are publicly available at httpartifactcifreebsdorg
5 Open Configurations
It is very important to make public can setup a highlyidentical instance of a public service and security isalso need to be considered In the past Jenkins jobcan only be edited on web interface which is not easy
to tracking the changes and reproduce Users whohas no administrative permission cannot see exactlyconfigurations Jenkins 2rsquos ldquopipeline as coderdquorsquo isa very promising to solve this problem On jenk-insFreeBSDorg we have tested it for the jobs whichbuild head and stable-10 branch The source code isavailable at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsbuildbuild-testgroovy However currently not allthe required Jenkins plugins of ciFreeBSDorg supportsthis We will look into this again and may switch it later
We use an approach used by other project for a longtime Jenkins job builder [10] which converts YAMLfiles to Jenkins XML configurations And manipulateJenkins instance via web service With Jenkins jobbuilder the configuration of Jenkins is trackable and re-producible and we can separate security credentials toother storage
While working with other developers we found thefact that not everyone knows (and has to know) how toconfigure Jenkins So adding new build jobs requiresJenkins administrators configure and test build scripts byhand This is a bottleneck of the development We alsofound that most of the build jobs are just having differ-ent build scripts other parts of the configurations such asrepository URL and notification settings are the same Itinspired us to create a ldquojob templaterdquo for similar jobs Atypical job contains following steps
1 Environment setup
(a) Check out latest source code
4
(b) Setup needed version of FreeBSD
(c) Install required packages
2 Execute specified build script for that job
3 Environment cleanup
The respective job definition (of Jenkins job builder)is very simple (listing 1)
Listing 1 Job definition- job
name FreeBSD-head-scan_buildscm
- FreeBSD-src-headbuilders
- checkout-scripts- setup-jail- execute-in-jail
publishers- clean-jail
wrappers- timestamps
And we use jail to ensure the freshness of the buildenvironment and we can install ldquosudordquo [21] program in-side jail with full privileges granted for maximum thethings user can do for testing As a result ldquobuild-in-jail script setsrdquo is created they are currently developedat httpsgithubcomlwhsufreebsd-citreejjbscriptsjail Itrsquos first job is simplifyenvironment setup and job execution we have threescripts for each builder and publishers in definition ofJenkins job builder
bull setupshsetup-jail builder setup a jail jail and install requirepackages
bull executeshexecute-in-jail builder execute build scripts fromuser
bull cleanshclean-jail publisher clean the used jail to release re-source
A typical job configuration contains three files
bull buildshMain build script which is executed by execueshin the jail
bull jailconfJail environment configuration version arch etc
bull pkg-listPackage needs to be installed (from pkgFreeBSDorg)
With these adding a new job is much simplified Sub-mitter can just test their tested build script (buildsh)with some job definitions (jailconf pkg-list) and otherconfiguration files for example srcconf or makeconfThen jenkins-admin merges scripts to freebsd-cijobsltjobnamegt and jenkins-admin creates a newjob entry in jenkins job builder push the new configu-ration to the Jenkins server Everything is done OurJenkins server will execute it around the clock
For example FreeBSD-head-arm64-build containsfiles as listing 2 3 4
Listing 2 Job configuration jailconfTARGET=amd64TARGET_ARCH=amd64WITH_32BIT=0OSRELEASE=103-RELEASE
Listing 3 Job configuration pkg-listaarch64-binutils
Listing 4 Job configuration buildshbinshenv JFLAG=$BUILDER_JFLAG TARGET=arm64 TARGET_ARCH=aarch64 sh -x freebsd-ciscriptsbuildbuild-world-kernelsh
We also add a ldquoquarantine moderdquo of ldquobuild-in-jail script setsrdquo just specify ldquoQUARANTINE=1rdquo injailconf the the internet connection is removed after jailenvironment is setup and resource is also strictly limitedsuch as execution time
The other ciFreeBSDorg setup can be foundat httpswikifreebsdorgJenkinsSetup
6 Other Integrations
The best feature of Jenkins is its features can be extendedby many plugins we selected several plugins which con-sidered useful for us and integrated into our system
61 NotificationMake sure related developers can know the latest projectstatus is very important We use both Email and IRCnotifications The status for sending out notifications are
bull Build failed
bull Build unstable (compile successfully but some testcases failed)
bull Build back to stable
5
And for notification email we attach following infor-mation
bull Changes since last build (who amp what)
bull Tail of the console output
bull What are the test cases failed
bull Related URLs
Email notification is considered a ldquodouble-edgedswordrdquo once we send out too many mails and peo-ple will consider them as spam If nobody readsthe notification mails and takes action it is totallyno use So use carefully created a pre-send scriptwith ldquoEmail-ext pluginrdquo [7] for filter out the internalerrors of the jenkins cluster to reduce false alertThe script is available at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsemail-extpre-sendgroovy
62 Code Review SystemThe FreeBSD project owns a code review systemat httpsreviewsFreeBSDorg whichis setup with Phabricator [19] and jenkins has aldquoPhabricator Differential Pluginrdquo [11] plugin Itcurrently supports only git we patched it to supportsubversion httpsgithubcomlwhsuphabricator-jenkins-plugintreescmand are also preparing to upstream it
On ciFreeBSDorg we have two jobs
bull FreeBSD-head-amd64-build-phabric
bull FreeBSD-doc-head-igor-phabric
They are using ldquoquarantine moderdquo of the ldquobuild-in-jailscript setsrdquo and are triggered when a new patch is up-loaded to reviewsFreeBSDorg (fig 5)
And reports the build result back to reviewsFreeBSDorg (fig 6)
63 Clang Static AnalyzerWe created a ldquoclang scan-buildrdquo job with ClangStatic Analyzer [3] and the build script leveraged theldquobsdclang-analyzedmkrdquo from NetBSD [23] simplybuild the whole world and kernel with command in list-ing 5
Listing 5 Command to perform clang scan-buildmake analyze CLANG_ANALYZE_OUTPUT_DIR=clangScanBuildReports CLANG_ANALYZE_OUTPUT=html
The result is taken cared by ldquoClang Scan-Build Plu-ginrdquo [6] which generates good web interface and historytracking (fig 7 8 9)
64 Igor and CheckstyleFor document build we use igor [26] by Warren Blockand Checkstyle [2] for checking the style of documentXML We worked with Warren and added a new option(-X) to igor to output checkstyle XML The respectivejob is FreeBSD-doc-head-igor ldquoCheckstyle Pluginrdquo [5]of Jenkins also provides good web interface and historytracking (fig 10 11 12)
65 Compiler WarningsWe also collect compiler warnings with ldquoWarnings Plu-ginrdquo [16] For showing the trend of compiler warningsand this pushes developers produce higher quality code(fig 13 14)
66 Build Status BadgeWe provide build status badge for embedding in externalweb pages through ldquoEmbeddable Build Status Pluginrdquo[8] For example on httpswikiFreeBSDorgarm64 we have badge as fig 15
To show the latest status of each branch The badgeURLs are
bull httpsltjob-urlgtbadgeicon
bull httpsltjoburlgtlastCompletedBuildbadgeicon
67 OthersWe also use some other plugins for improving the inter-face and helping the maintenance task
bull SCM Sync configuration [14]
ndash Use post-commit hook to notify administra-tors
bull SafeRestart [15]
ndash Schedule restart jenkins when all current jobscomplete
ndash Save and resume the build queue
bull Green balls [9]
ndash To replace blue status ball with green ones tofulfill the convention for most culture
7 Conclusion
In this paper we described about the history and howwe setup the latest CI system of the FreeBSD projectCI does help the development of the FreeBSD projectand we hope these experience and codes can also helpdownstream projects
6
Figure 5 Build triggerd by a patch
Figure 6 Rsult of a build triggerd by a patch
7
Figure 7 Clang scan-build result
8 Future Work
The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas
bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo
bull Check for reproducible build
bull More tests even some are not enabled by defaultDtrace
bull Build for project branchesMake testing feature branches easier
bull Performance tests
bull Work with other projects
We hope these work can make the quality of FreeBSDeven better
AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware
bull jenkins-adminFreeBSDorg
bull The FreeBSD Foundation
bull clusteradminFreeBSDorg
bull phabic-adminFreeBSDorg
bull People on -testing -current -stable mailing lists
References
[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991
[2] Checkstylehttpcheckstylesourceforgenet
[3] Clang Static Analyzerhttpsclang-analyzerllvmorg
[4] JUnithttpjunitorg
[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin
8
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
Figure 4 Test result trend
is compatible with the official download site We hopethat in the future these snapshot artifacts can also be in-cluded and be more easily accessed by users
Currently we have these artifacts for each pipeline
bull txz
ndash From -build jobs
ndash Distribution files just like what on httpsdownloadFreeBSDorg
bull diskimgxz
ndash From -images jobs
ndash Virtual machine disk image file installed fromabove txz files
bull disk-testimgxz
ndash From -testvm jobs
ndash Virtual machine disk image file with testscript and test case
All files are publicly available at httpartifactcifreebsdorg
5 Open Configurations
It is very important to make public can setup a highlyidentical instance of a public service and security isalso need to be considered In the past Jenkins jobcan only be edited on web interface which is not easy
to tracking the changes and reproduce Users whohas no administrative permission cannot see exactlyconfigurations Jenkins 2rsquos ldquopipeline as coderdquorsquo isa very promising to solve this problem On jenk-insFreeBSDorg we have tested it for the jobs whichbuild head and stable-10 branch The source code isavailable at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsbuildbuild-testgroovy However currently not allthe required Jenkins plugins of ciFreeBSDorg supportsthis We will look into this again and may switch it later
We use an approach used by other project for a longtime Jenkins job builder [10] which converts YAMLfiles to Jenkins XML configurations And manipulateJenkins instance via web service With Jenkins jobbuilder the configuration of Jenkins is trackable and re-producible and we can separate security credentials toother storage
While working with other developers we found thefact that not everyone knows (and has to know) how toconfigure Jenkins So adding new build jobs requiresJenkins administrators configure and test build scripts byhand This is a bottleneck of the development We alsofound that most of the build jobs are just having differ-ent build scripts other parts of the configurations such asrepository URL and notification settings are the same Itinspired us to create a ldquojob templaterdquo for similar jobs Atypical job contains following steps
1 Environment setup
(a) Check out latest source code
4
(b) Setup needed version of FreeBSD
(c) Install required packages
2 Execute specified build script for that job
3 Environment cleanup
The respective job definition (of Jenkins job builder)is very simple (listing 1)
Listing 1 Job definition- job
name FreeBSD-head-scan_buildscm
- FreeBSD-src-headbuilders
- checkout-scripts- setup-jail- execute-in-jail
publishers- clean-jail
wrappers- timestamps
And we use jail to ensure the freshness of the buildenvironment and we can install ldquosudordquo [21] program in-side jail with full privileges granted for maximum thethings user can do for testing As a result ldquobuild-in-jail script setsrdquo is created they are currently developedat httpsgithubcomlwhsufreebsd-citreejjbscriptsjail Itrsquos first job is simplifyenvironment setup and job execution we have threescripts for each builder and publishers in definition ofJenkins job builder
bull setupshsetup-jail builder setup a jail jail and install requirepackages
bull executeshexecute-in-jail builder execute build scripts fromuser
bull cleanshclean-jail publisher clean the used jail to release re-source
A typical job configuration contains three files
bull buildshMain build script which is executed by execueshin the jail
bull jailconfJail environment configuration version arch etc
bull pkg-listPackage needs to be installed (from pkgFreeBSDorg)
With these adding a new job is much simplified Sub-mitter can just test their tested build script (buildsh)with some job definitions (jailconf pkg-list) and otherconfiguration files for example srcconf or makeconfThen jenkins-admin merges scripts to freebsd-cijobsltjobnamegt and jenkins-admin creates a newjob entry in jenkins job builder push the new configu-ration to the Jenkins server Everything is done OurJenkins server will execute it around the clock
For example FreeBSD-head-arm64-build containsfiles as listing 2 3 4
Listing 2 Job configuration jailconfTARGET=amd64TARGET_ARCH=amd64WITH_32BIT=0OSRELEASE=103-RELEASE
Listing 3 Job configuration pkg-listaarch64-binutils
Listing 4 Job configuration buildshbinshenv JFLAG=$BUILDER_JFLAG TARGET=arm64 TARGET_ARCH=aarch64 sh -x freebsd-ciscriptsbuildbuild-world-kernelsh
We also add a ldquoquarantine moderdquo of ldquobuild-in-jail script setsrdquo just specify ldquoQUARANTINE=1rdquo injailconf the the internet connection is removed after jailenvironment is setup and resource is also strictly limitedsuch as execution time
The other ciFreeBSDorg setup can be foundat httpswikifreebsdorgJenkinsSetup
6 Other Integrations
The best feature of Jenkins is its features can be extendedby many plugins we selected several plugins which con-sidered useful for us and integrated into our system
61 NotificationMake sure related developers can know the latest projectstatus is very important We use both Email and IRCnotifications The status for sending out notifications are
bull Build failed
bull Build unstable (compile successfully but some testcases failed)
bull Build back to stable
5
And for notification email we attach following infor-mation
bull Changes since last build (who amp what)
bull Tail of the console output
bull What are the test cases failed
bull Related URLs
Email notification is considered a ldquodouble-edgedswordrdquo once we send out too many mails and peo-ple will consider them as spam If nobody readsthe notification mails and takes action it is totallyno use So use carefully created a pre-send scriptwith ldquoEmail-ext pluginrdquo [7] for filter out the internalerrors of the jenkins cluster to reduce false alertThe script is available at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsemail-extpre-sendgroovy
62 Code Review SystemThe FreeBSD project owns a code review systemat httpsreviewsFreeBSDorg whichis setup with Phabricator [19] and jenkins has aldquoPhabricator Differential Pluginrdquo [11] plugin Itcurrently supports only git we patched it to supportsubversion httpsgithubcomlwhsuphabricator-jenkins-plugintreescmand are also preparing to upstream it
On ciFreeBSDorg we have two jobs
bull FreeBSD-head-amd64-build-phabric
bull FreeBSD-doc-head-igor-phabric
They are using ldquoquarantine moderdquo of the ldquobuild-in-jailscript setsrdquo and are triggered when a new patch is up-loaded to reviewsFreeBSDorg (fig 5)
And reports the build result back to reviewsFreeBSDorg (fig 6)
63 Clang Static AnalyzerWe created a ldquoclang scan-buildrdquo job with ClangStatic Analyzer [3] and the build script leveraged theldquobsdclang-analyzedmkrdquo from NetBSD [23] simplybuild the whole world and kernel with command in list-ing 5
Listing 5 Command to perform clang scan-buildmake analyze CLANG_ANALYZE_OUTPUT_DIR=clangScanBuildReports CLANG_ANALYZE_OUTPUT=html
The result is taken cared by ldquoClang Scan-Build Plu-ginrdquo [6] which generates good web interface and historytracking (fig 7 8 9)
64 Igor and CheckstyleFor document build we use igor [26] by Warren Blockand Checkstyle [2] for checking the style of documentXML We worked with Warren and added a new option(-X) to igor to output checkstyle XML The respectivejob is FreeBSD-doc-head-igor ldquoCheckstyle Pluginrdquo [5]of Jenkins also provides good web interface and historytracking (fig 10 11 12)
65 Compiler WarningsWe also collect compiler warnings with ldquoWarnings Plu-ginrdquo [16] For showing the trend of compiler warningsand this pushes developers produce higher quality code(fig 13 14)
66 Build Status BadgeWe provide build status badge for embedding in externalweb pages through ldquoEmbeddable Build Status Pluginrdquo[8] For example on httpswikiFreeBSDorgarm64 we have badge as fig 15
To show the latest status of each branch The badgeURLs are
bull httpsltjob-urlgtbadgeicon
bull httpsltjoburlgtlastCompletedBuildbadgeicon
67 OthersWe also use some other plugins for improving the inter-face and helping the maintenance task
bull SCM Sync configuration [14]
ndash Use post-commit hook to notify administra-tors
bull SafeRestart [15]
ndash Schedule restart jenkins when all current jobscomplete
ndash Save and resume the build queue
bull Green balls [9]
ndash To replace blue status ball with green ones tofulfill the convention for most culture
7 Conclusion
In this paper we described about the history and howwe setup the latest CI system of the FreeBSD projectCI does help the development of the FreeBSD projectand we hope these experience and codes can also helpdownstream projects
6
Figure 5 Build triggerd by a patch
Figure 6 Rsult of a build triggerd by a patch
7
Figure 7 Clang scan-build result
8 Future Work
The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas
bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo
bull Check for reproducible build
bull More tests even some are not enabled by defaultDtrace
bull Build for project branchesMake testing feature branches easier
bull Performance tests
bull Work with other projects
We hope these work can make the quality of FreeBSDeven better
AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware
bull jenkins-adminFreeBSDorg
bull The FreeBSD Foundation
bull clusteradminFreeBSDorg
bull phabic-adminFreeBSDorg
bull People on -testing -current -stable mailing lists
References
[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991
[2] Checkstylehttpcheckstylesourceforgenet
[3] Clang Static Analyzerhttpsclang-analyzerllvmorg
[4] JUnithttpjunitorg
[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin
8
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
(b) Setup needed version of FreeBSD
(c) Install required packages
2 Execute specified build script for that job
3 Environment cleanup
The respective job definition (of Jenkins job builder)is very simple (listing 1)
Listing 1 Job definition- job
name FreeBSD-head-scan_buildscm
- FreeBSD-src-headbuilders
- checkout-scripts- setup-jail- execute-in-jail
publishers- clean-jail
wrappers- timestamps
And we use jail to ensure the freshness of the buildenvironment and we can install ldquosudordquo [21] program in-side jail with full privileges granted for maximum thethings user can do for testing As a result ldquobuild-in-jail script setsrdquo is created they are currently developedat httpsgithubcomlwhsufreebsd-citreejjbscriptsjail Itrsquos first job is simplifyenvironment setup and job execution we have threescripts for each builder and publishers in definition ofJenkins job builder
bull setupshsetup-jail builder setup a jail jail and install requirepackages
bull executeshexecute-in-jail builder execute build scripts fromuser
bull cleanshclean-jail publisher clean the used jail to release re-source
A typical job configuration contains three files
bull buildshMain build script which is executed by execueshin the jail
bull jailconfJail environment configuration version arch etc
bull pkg-listPackage needs to be installed (from pkgFreeBSDorg)
With these adding a new job is much simplified Sub-mitter can just test their tested build script (buildsh)with some job definitions (jailconf pkg-list) and otherconfiguration files for example srcconf or makeconfThen jenkins-admin merges scripts to freebsd-cijobsltjobnamegt and jenkins-admin creates a newjob entry in jenkins job builder push the new configu-ration to the Jenkins server Everything is done OurJenkins server will execute it around the clock
For example FreeBSD-head-arm64-build containsfiles as listing 2 3 4
Listing 2 Job configuration jailconfTARGET=amd64TARGET_ARCH=amd64WITH_32BIT=0OSRELEASE=103-RELEASE
Listing 3 Job configuration pkg-listaarch64-binutils
Listing 4 Job configuration buildshbinshenv JFLAG=$BUILDER_JFLAG TARGET=arm64 TARGET_ARCH=aarch64 sh -x freebsd-ciscriptsbuildbuild-world-kernelsh
We also add a ldquoquarantine moderdquo of ldquobuild-in-jail script setsrdquo just specify ldquoQUARANTINE=1rdquo injailconf the the internet connection is removed after jailenvironment is setup and resource is also strictly limitedsuch as execution time
The other ciFreeBSDorg setup can be foundat httpswikifreebsdorgJenkinsSetup
6 Other Integrations
The best feature of Jenkins is its features can be extendedby many plugins we selected several plugins which con-sidered useful for us and integrated into our system
61 NotificationMake sure related developers can know the latest projectstatus is very important We use both Email and IRCnotifications The status for sending out notifications are
bull Build failed
bull Build unstable (compile successfully but some testcases failed)
bull Build back to stable
5
And for notification email we attach following infor-mation
bull Changes since last build (who amp what)
bull Tail of the console output
bull What are the test cases failed
bull Related URLs
Email notification is considered a ldquodouble-edgedswordrdquo once we send out too many mails and peo-ple will consider them as spam If nobody readsthe notification mails and takes action it is totallyno use So use carefully created a pre-send scriptwith ldquoEmail-ext pluginrdquo [7] for filter out the internalerrors of the jenkins cluster to reduce false alertThe script is available at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsemail-extpre-sendgroovy
62 Code Review SystemThe FreeBSD project owns a code review systemat httpsreviewsFreeBSDorg whichis setup with Phabricator [19] and jenkins has aldquoPhabricator Differential Pluginrdquo [11] plugin Itcurrently supports only git we patched it to supportsubversion httpsgithubcomlwhsuphabricator-jenkins-plugintreescmand are also preparing to upstream it
On ciFreeBSDorg we have two jobs
bull FreeBSD-head-amd64-build-phabric
bull FreeBSD-doc-head-igor-phabric
They are using ldquoquarantine moderdquo of the ldquobuild-in-jailscript setsrdquo and are triggered when a new patch is up-loaded to reviewsFreeBSDorg (fig 5)
And reports the build result back to reviewsFreeBSDorg (fig 6)
63 Clang Static AnalyzerWe created a ldquoclang scan-buildrdquo job with ClangStatic Analyzer [3] and the build script leveraged theldquobsdclang-analyzedmkrdquo from NetBSD [23] simplybuild the whole world and kernel with command in list-ing 5
Listing 5 Command to perform clang scan-buildmake analyze CLANG_ANALYZE_OUTPUT_DIR=clangScanBuildReports CLANG_ANALYZE_OUTPUT=html
The result is taken cared by ldquoClang Scan-Build Plu-ginrdquo [6] which generates good web interface and historytracking (fig 7 8 9)
64 Igor and CheckstyleFor document build we use igor [26] by Warren Blockand Checkstyle [2] for checking the style of documentXML We worked with Warren and added a new option(-X) to igor to output checkstyle XML The respectivejob is FreeBSD-doc-head-igor ldquoCheckstyle Pluginrdquo [5]of Jenkins also provides good web interface and historytracking (fig 10 11 12)
65 Compiler WarningsWe also collect compiler warnings with ldquoWarnings Plu-ginrdquo [16] For showing the trend of compiler warningsand this pushes developers produce higher quality code(fig 13 14)
66 Build Status BadgeWe provide build status badge for embedding in externalweb pages through ldquoEmbeddable Build Status Pluginrdquo[8] For example on httpswikiFreeBSDorgarm64 we have badge as fig 15
To show the latest status of each branch The badgeURLs are
bull httpsltjob-urlgtbadgeicon
bull httpsltjoburlgtlastCompletedBuildbadgeicon
67 OthersWe also use some other plugins for improving the inter-face and helping the maintenance task
bull SCM Sync configuration [14]
ndash Use post-commit hook to notify administra-tors
bull SafeRestart [15]
ndash Schedule restart jenkins when all current jobscomplete
ndash Save and resume the build queue
bull Green balls [9]
ndash To replace blue status ball with green ones tofulfill the convention for most culture
7 Conclusion
In this paper we described about the history and howwe setup the latest CI system of the FreeBSD projectCI does help the development of the FreeBSD projectand we hope these experience and codes can also helpdownstream projects
6
Figure 5 Build triggerd by a patch
Figure 6 Rsult of a build triggerd by a patch
7
Figure 7 Clang scan-build result
8 Future Work
The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas
bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo
bull Check for reproducible build
bull More tests even some are not enabled by defaultDtrace
bull Build for project branchesMake testing feature branches easier
bull Performance tests
bull Work with other projects
We hope these work can make the quality of FreeBSDeven better
AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware
bull jenkins-adminFreeBSDorg
bull The FreeBSD Foundation
bull clusteradminFreeBSDorg
bull phabic-adminFreeBSDorg
bull People on -testing -current -stable mailing lists
References
[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991
[2] Checkstylehttpcheckstylesourceforgenet
[3] Clang Static Analyzerhttpsclang-analyzerllvmorg
[4] JUnithttpjunitorg
[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin
8
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
And for notification email we attach following infor-mation
bull Changes since last build (who amp what)
bull Tail of the console output
bull What are the test cases failed
bull Related URLs
Email notification is considered a ldquodouble-edgedswordrdquo once we send out too many mails and peo-ple will consider them as spam If nobody readsthe notification mails and takes action it is totallyno use So use carefully created a pre-send scriptwith ldquoEmail-ext pluginrdquo [7] for filter out the internalerrors of the jenkins cluster to reduce false alertThe script is available at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsemail-extpre-sendgroovy
62 Code Review SystemThe FreeBSD project owns a code review systemat httpsreviewsFreeBSDorg whichis setup with Phabricator [19] and jenkins has aldquoPhabricator Differential Pluginrdquo [11] plugin Itcurrently supports only git we patched it to supportsubversion httpsgithubcomlwhsuphabricator-jenkins-plugintreescmand are also preparing to upstream it
On ciFreeBSDorg we have two jobs
bull FreeBSD-head-amd64-build-phabric
bull FreeBSD-doc-head-igor-phabric
They are using ldquoquarantine moderdquo of the ldquobuild-in-jailscript setsrdquo and are triggered when a new patch is up-loaded to reviewsFreeBSDorg (fig 5)
And reports the build result back to reviewsFreeBSDorg (fig 6)
63 Clang Static AnalyzerWe created a ldquoclang scan-buildrdquo job with ClangStatic Analyzer [3] and the build script leveraged theldquobsdclang-analyzedmkrdquo from NetBSD [23] simplybuild the whole world and kernel with command in list-ing 5
Listing 5 Command to perform clang scan-buildmake analyze CLANG_ANALYZE_OUTPUT_DIR=clangScanBuildReports CLANG_ANALYZE_OUTPUT=html
The result is taken cared by ldquoClang Scan-Build Plu-ginrdquo [6] which generates good web interface and historytracking (fig 7 8 9)
64 Igor and CheckstyleFor document build we use igor [26] by Warren Blockand Checkstyle [2] for checking the style of documentXML We worked with Warren and added a new option(-X) to igor to output checkstyle XML The respectivejob is FreeBSD-doc-head-igor ldquoCheckstyle Pluginrdquo [5]of Jenkins also provides good web interface and historytracking (fig 10 11 12)
65 Compiler WarningsWe also collect compiler warnings with ldquoWarnings Plu-ginrdquo [16] For showing the trend of compiler warningsand this pushes developers produce higher quality code(fig 13 14)
66 Build Status BadgeWe provide build status badge for embedding in externalweb pages through ldquoEmbeddable Build Status Pluginrdquo[8] For example on httpswikiFreeBSDorgarm64 we have badge as fig 15
To show the latest status of each branch The badgeURLs are
bull httpsltjob-urlgtbadgeicon
bull httpsltjoburlgtlastCompletedBuildbadgeicon
67 OthersWe also use some other plugins for improving the inter-face and helping the maintenance task
bull SCM Sync configuration [14]
ndash Use post-commit hook to notify administra-tors
bull SafeRestart [15]
ndash Schedule restart jenkins when all current jobscomplete
ndash Save and resume the build queue
bull Green balls [9]
ndash To replace blue status ball with green ones tofulfill the convention for most culture
7 Conclusion
In this paper we described about the history and howwe setup the latest CI system of the FreeBSD projectCI does help the development of the FreeBSD projectand we hope these experience and codes can also helpdownstream projects
6
Figure 5 Build triggerd by a patch
Figure 6 Rsult of a build triggerd by a patch
7
Figure 7 Clang scan-build result
8 Future Work
The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas
bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo
bull Check for reproducible build
bull More tests even some are not enabled by defaultDtrace
bull Build for project branchesMake testing feature branches easier
bull Performance tests
bull Work with other projects
We hope these work can make the quality of FreeBSDeven better
AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware
bull jenkins-adminFreeBSDorg
bull The FreeBSD Foundation
bull clusteradminFreeBSDorg
bull phabic-adminFreeBSDorg
bull People on -testing -current -stable mailing lists
References
[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991
[2] Checkstylehttpcheckstylesourceforgenet
[3] Clang Static Analyzerhttpsclang-analyzerllvmorg
[4] JUnithttpjunitorg
[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin
8
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
Figure 5 Build triggerd by a patch
Figure 6 Rsult of a build triggerd by a patch
7
Figure 7 Clang scan-build result
8 Future Work
The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas
bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo
bull Check for reproducible build
bull More tests even some are not enabled by defaultDtrace
bull Build for project branchesMake testing feature branches easier
bull Performance tests
bull Work with other projects
We hope these work can make the quality of FreeBSDeven better
AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware
bull jenkins-adminFreeBSDorg
bull The FreeBSD Foundation
bull clusteradminFreeBSDorg
bull phabic-adminFreeBSDorg
bull People on -testing -current -stable mailing lists
References
[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991
[2] Checkstylehttpcheckstylesourceforgenet
[3] Clang Static Analyzerhttpsclang-analyzerllvmorg
[4] JUnithttpjunitorg
[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin
8
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
Figure 7 Clang scan-build result
8 Future Work
The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas
bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo
bull Check for reproducible build
bull More tests even some are not enabled by defaultDtrace
bull Build for project branchesMake testing feature branches easier
bull Performance tests
bull Work with other projects
We hope these work can make the quality of FreeBSDeven better
AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware
bull jenkins-adminFreeBSDorg
bull The FreeBSD Foundation
bull clusteradminFreeBSDorg
bull phabic-adminFreeBSDorg
bull People on -testing -current -stable mailing lists
References
[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991
[2] Checkstylehttpcheckstylesourceforgenet
[3] Clang Static Analyzerhttpsclang-analyzerllvmorg
[4] JUnithttpjunitorg
[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin
8
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
Figure 8 Clang scan-build result explanation
Figure 9 Clang scan-build trend
9
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
Figure 10 Igor + Checksytle - result
Figure 11 Igor + Checkstyle - comparing with previous build
Figure 12 Igor + Checkstyle - trend
10
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
Figure 13 Compiler Warnings - result
Figure 14 Compiler Warnings - trend
Figure 15 Build satus badge
11
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12
[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin
[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin
[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin
[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls
[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder
[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin
[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin
[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin
[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin
[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin
[17] Jenkinshttpsjenkinsio
[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua
[19] Phabricatorhttpswwwphacilitycomphabricator
[20] Pure-FTPdhttpswwwpureftpdorg
[21] Sudohttpwwwsudows
[22] The FreeBSD ProjecthttpswwwFreeBSDorg
[23] The NetBSD Projecthttpwwwnetbsdorg
[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml
[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox
[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor
[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve
[28] nginxhttpsnginxorg
12