Continuous Integration of The FreeBSD Project

Li-Wen HsuThe FreeBSD ProjectlwhsuFreeBSDorg

Abstract

The FreeBSD project [22]rsquos continuous integrationproject started in the late 2013 We use Jenkins [17] au-tomation server to build our continuous integration sys-tem It monitors the svn repository for new commitsand triggers a new build of it In each build the buildmachine compiles the latest code creates disk imageand creates a virtual machine to run test suite In themeantime we collect the compiler warnings and performsome further checks like clang analyzer All these infor-mation are published to the developers and users to im-prove the quality of the FreeBSD project In this paperwe describe the details of the system implementation

1 Introduction

The term of continuous integration (CI) is first used byGrady Booch in 1991 [1] then Extreme programming(XP) extends the concept of CI and suggests to integrateas many times as possible per day Now CI is an impor-tant and common practice of software engineering Thebenefits of CI are developers can know the conflicts andbugs earlier This reduces the cost for resolving themlater Also the status of the project is clearer and it iseasier to estimate progress

There are many software designed for various CI needamong them the most popular one is Jenkins an opensource automation server written in Java created byKohsuke Kawaguchi We use Jenkins as the main com-ponent along with some other daemons like Pure-FTPd[20] to build FreeBSD projectrsquos CI system

We describe the history of FreeBSDrsquos CI work howwe trigger build from changes in version control systemdo tests and publish results in the rest of this paper

2 Automatic Build and History of CI workin FreeBSD

Dag-Erling Smrgrav created the first pubic CI sys-tem for the FreeBSD project httptinderboxFreeBSDorg this can be traced back in the mailarchive in March 2002 It is believed the first contin-uous integration service of the FreeBSD project Tin-derbox is an open source project codes are availableat svnsvnfreebsdorgbaseuserdestinderbox Documents can be found in FreeBSDdoc project [24] and wiki [25] Unfortunately this ser-vice stopped in September or October 2014 neverthe-less here are many scripts from this project derived orinspiring the successive projects

In late 2013 Craig Rodrigues founded jenkins-adminFreeBSDorg with several developers Themain duty of jenkins-admin is maintaining the Jenk-ins instance in FreeBSD cluster httpsjenkinsFreeBSDorg Which is an independent successorof tinderboxFreeBSDorg We configure it monitorsthe FreeBSD src and doc repositories and triggers anew build when there is new commit in each supportedbranch We created the first ldquobuild-and-test pipelinerdquofor the FreeBSD project In each build triggered by achange in repository after compiling successes we cre-ate a virtual machine disk image with the latest binariesand spawn a new virtual machine to run tests The testresults are collected and stored by Jenkins server

The highlight features of jenkinsFreeBSDorg are

bull Periodically build for head stable-11 stable-10 andstable-9

bull Architectures built against amd64 i386 sparc64and arm64

bull Build FreeBSD with GCC using amd64-xtoolchain-gcc

bull Experimental Jenkins 2 ldquopipeline as coderdquo

Shortly jenkinsFreeBSDorg become a significantrole in the FreeBSD development

In 2016 with the hardwares sponsored by theFreeBSD foundation we created httpsciFreeBSDorg an experimental site for testing newideas We still leverage Jenkins because it has provenuseful on jenkinsFreeBSDorg Aditionally we imple-mented

bull Artifact server

bull Job template

Having a centralized artifact server enables people re-trieving the files generated from each stage of a pipelinewhich can be used for reproducing issues for debuggingpurpose or doing further tests The details about jobtemplate are in ldquoopen configurationsrdquo section

Another experiment in this system is we try to makethe build pipeline more fine-grained and introducedmore build stages For example we not only create vir-tual machine disk image specialized for tests but also tryto produce the distribution files and virtual machine diskimages the same as the official release provided by re-lease engineering team Also we build LINT kernel formore test coverage

There are multiple stages in a pipelne each is a Jenk-ins job The naming conversion of the build jobs is

FreeBSD-branch-target_arch-stage

We create a pipeline for each supported branch and ar-chitecture Take head branch and amd64 architecture forexample

bull FreeBSD-head-amd64-buildBuild the world and kernel from scratch and createsdistribution files

bull FreeBSD-head-amd64-LINTBuild the LINT kernel

bull FreeBSD-head-amd64-imagesBuild virtual machine disk images from distributionfiles with the same configuration of official release

bull FreeBSD-head-amd64-testvmBuild virtual machine disk image from distributionfiles and test files and package are pre-installed

bull FreeBSD-head-amd64-testRun the image from ldquotestvmrdquo stage in a new createdvirtual machine

Their relationship is shown in 1The codes are available at httpsgithub

comlwhsufreebsd-ci and httpsgithubcomlwhsujjb-freebsd-cimerging back to the main freebsd-ci repository on githubor even svnsvnfreebsdorg is in progress

3 Self Tests

In 2013 Rui Paulo created the usrtests hierar-chy with the Kyua [18] test framework by Julio MerinoThese tests can be performed by command

cd usrtests ampamp kyua test

As of January 2017 we have 6023 tests for head branchon amd64 architecture

As previous section mentioned we build virtual ma-chine disk images with the required files and packagesfor running tests We pre-install the kyua package andput a script which runs kyua commands and generatesthe test reports after all tests are executed Finally it is-sues shutdown command After the disk image readyfor amd64 and i386 architecture we launch a bhyve [27]virtual machine with an expect script for starting run-ning tests in and also for catching timeout in case

After the testing virtual machine stopped we extractedthe test results from the disk image put to the workspacewhere Jenkins slave daemon will collect them and sendback to the Jenkins master for archiving and maintaininga trend record The script to create test virtual machineimage is scriptsbuildbuild-test_imagesh and the main script for executing tests in virtualmachines and extracting results is scriptstestrun-testssh

4 Access to Results

We configure Kyua to generate results as JUnit [4] XMLformat which is natively supported by Jenkins and manyother tools Jenkins default plugins take outputs fromthe builds make it a friendly accessible web inter-face at httpsltjob-urlgtltbuild-numbergttestReport (fig 2 3) and maintains a historicaldata (fig 4) This helps us tracking the software qualitytrending

The artifact server artifactciFreeBSDorg is newlyintroduced in ciFreeBSDorg its main responsibility isto share files between jobs in a pipeline The artifactserver is basically an FTP server powered by Pure-FTPdfor serving files uploading For downloading we use anginx [28] web server to provide read-only access Forsecurity reason we configure FTP server can only be ac-cessed by FTP over TLS As a result we also added FTPover TLS feature in ldquoPublish Over FTP Pluginrdquo [] ofJenkins

In the end of each stage Jenkins uploads createdfiles to the artifact server there is only one root di-rectory ldquosnapshotsrdquo on artifactciFreeBSDorg for nowThe directory layout of it is branchsvn_revisiontargettarget_arch for ex-ample headr303226amd64amd64 This layout

2

FreeBSD-head-amd64-build

FreeBSD-head-amd64-testvm

FreeBSD-head-amd64-LINT

FreeBSD-head-amd64-images

FreeBSD-head-amd64-test

Figure 1 head-amd64 pipeline

Figure 2 Test result from a test job

Figure 3 Failed test case display

3

Figure 4 Test result trend

is compatible with the official download site We hopethat in the future these snapshot artifacts can also be in-cluded and be more easily accessed by users

Currently we have these artifacts for each pipeline

bull txz

ndash From -build jobs

ndash Distribution files just like what on httpsdownloadFreeBSDorg

bull diskimgxz

ndash From -images jobs

ndash Virtual machine disk image file installed fromabove txz files

bull disk-testimgxz

ndash From -testvm jobs

ndash Virtual machine disk image file with testscript and test case

All files are publicly available at httpartifactcifreebsdorg

5 Open Configurations

It is very important to make public can setup a highlyidentical instance of a public service and security isalso need to be considered In the past Jenkins jobcan only be edited on web interface which is not easy

to tracking the changes and reproduce Users whohas no administrative permission cannot see exactlyconfigurations Jenkins 2rsquos ldquopipeline as coderdquorsquo isa very promising to solve this problem On jenk-insFreeBSDorg we have tested it for the jobs whichbuild head and stable-10 branch The source code isavailable at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsbuildbuild-testgroovy However currently not allthe required Jenkins plugins of ciFreeBSDorg supportsthis We will look into this again and may switch it later

We use an approach used by other project for a longtime Jenkins job builder [10] which converts YAMLfiles to Jenkins XML configurations And manipulateJenkins instance via web service With Jenkins jobbuilder the configuration of Jenkins is trackable and re-producible and we can separate security credentials toother storage

While working with other developers we found thefact that not everyone knows (and has to know) how toconfigure Jenkins So adding new build jobs requiresJenkins administrators configure and test build scripts byhand This is a bottleneck of the development We alsofound that most of the build jobs are just having differ-ent build scripts other parts of the configurations such asrepository URL and notification settings are the same Itinspired us to create a ldquojob templaterdquo for similar jobs Atypical job contains following steps

1 Environment setup

(a) Check out latest source code

4

Figure 4 Test result trend

is compatible with the official download site We hopethat in the future these snapshot artifacts can also be in-cluded and be more easily accessed by users

Currently we have these artifacts for each pipeline

bull txz

ndash From -build jobs

ndash Distribution files just like what on httpsdownloadFreeBSDorg

bull diskimgxz

ndash From -images jobs

ndash Virtual machine disk image file installed fromabove txz files

bull disk-testimgxz

ndash From -testvm jobs

ndash Virtual machine disk image file with testscript and test case

All files are publicly available at httpartifactcifreebsdorg

5 Open Configurations

It is very important to make public can setup a highlyidentical instance of a public service and security isalso need to be considered In the past Jenkins jobcan only be edited on web interface which is not easy

to tracking the changes and reproduce Users whohas no administrative permission cannot see exactlyconfigurations Jenkins 2rsquos ldquopipeline as coderdquorsquo isa very promising to solve this problem On jenk-insFreeBSDorg we have tested it for the jobs whichbuild head and stable-10 branch The source code isavailable at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsbuildbuild-testgroovy However currently not allthe required Jenkins plugins of ciFreeBSDorg supportsthis We will look into this again and may switch it later

We use an approach used by other project for a longtime Jenkins job builder [10] which converts YAMLfiles to Jenkins XML configurations And manipulateJenkins instance via web service With Jenkins jobbuilder the configuration of Jenkins is trackable and re-producible and we can separate security credentials toother storage

While working with other developers we found thefact that not everyone knows (and has to know) how toconfigure Jenkins So adding new build jobs requiresJenkins administrators configure and test build scripts byhand This is a bottleneck of the development We alsofound that most of the build jobs are just having differ-ent build scripts other parts of the configurations such asrepository URL and notification settings are the same Itinspired us to create a ldquojob templaterdquo for similar jobs Atypical job contains following steps

1 Environment setup

(a) Check out latest source code

4

(b) Setup needed version of FreeBSD

(c) Install required packages

2 Execute specified build script for that job

3 Environment cleanup

The respective job definition (of Jenkins job builder)is very simple (listing 1)

Listing 1 Job definition- job

name FreeBSD-head-scan_buildscm

- FreeBSD-src-headbuilders

- checkout-scripts- setup-jail- execute-in-jail

publishers- clean-jail

wrappers- timestamps

And we use jail to ensure the freshness of the buildenvironment and we can install ldquosudordquo [21] program in-side jail with full privileges granted for maximum thethings user can do for testing As a result ldquobuild-in-jail script setsrdquo is created they are currently developedat httpsgithubcomlwhsufreebsd-citreejjbscriptsjail Itrsquos first job is simplifyenvironment setup and job execution we have threescripts for each builder and publishers in definition ofJenkins job builder

bull setupshsetup-jail builder setup a jail jail and install requirepackages

bull executeshexecute-in-jail builder execute build scripts fromuser

bull cleanshclean-jail publisher clean the used jail to release re-source

A typical job configuration contains three files

bull buildshMain build script which is executed by execueshin the jail

bull jailconfJail environment configuration version arch etc

bull pkg-listPackage needs to be installed (from pkgFreeBSDorg)

With these adding a new job is much simplified Sub-mitter can just test their tested build script (buildsh)with some job definitions (jailconf pkg-list) and otherconfiguration files for example srcconf or makeconfThen jenkins-admin merges scripts to freebsd-cijobsltjobnamegt and jenkins-admin creates a newjob entry in jenkins job builder push the new configu-ration to the Jenkins server Everything is done OurJenkins server will execute it around the clock

For example FreeBSD-head-arm64-build containsfiles as listing 2 3 4

Listing 2 Job configuration jailconfTARGET=amd64TARGET_ARCH=amd64WITH_32BIT=0OSRELEASE=103-RELEASE

Listing 3 Job configuration pkg-listaarch64-binutils

Listing 4 Job configuration buildshbinshenv JFLAG=$BUILDER_JFLAG TARGET=arm64 TARGET_ARCH=aarch64 sh -x freebsd-ciscriptsbuildbuild-world-kernelsh

We also add a ldquoquarantine moderdquo of ldquobuild-in-jail script setsrdquo just specify ldquoQUARANTINE=1rdquo injailconf the the internet connection is removed after jailenvironment is setup and resource is also strictly limitedsuch as execution time

The other ciFreeBSDorg setup can be foundat httpswikifreebsdorgJenkinsSetup

6 Other Integrations

The best feature of Jenkins is its features can be extendedby many plugins we selected several plugins which con-sidered useful for us and integrated into our system

61 NotificationMake sure related developers can know the latest projectstatus is very important We use both Email and IRCnotifications The status for sending out notifications are

bull Build failed

bull Build unstable (compile successfully but some testcases failed)

bull Build back to stable

5

And for notification email we attach following infor-mation

bull Changes since last build (who amp what)

bull Tail of the console output

bull What are the test cases failed

bull Related URLs

Email notification is considered a ldquodouble-edgedswordrdquo once we send out too many mails and peo-ple will consider them as spam If nobody readsthe notification mails and takes action it is totallyno use So use carefully created a pre-send scriptwith ldquoEmail-ext pluginrdquo [7] for filter out the internalerrors of the jenkins cluster to reduce false alertThe script is available at httpsgithubcomfreebsdfreebsd-ciblobmasterscriptsemail-extpre-sendgroovy

62 Code Review SystemThe FreeBSD project owns a code review systemat httpsreviewsFreeBSDorg whichis setup with Phabricator [19] and jenkins has aldquoPhabricator Differential Pluginrdquo [11] plugin Itcurrently supports only git we patched it to supportsubversion httpsgithubcomlwhsuphabricator-jenkins-plugintreescmand are also preparing to upstream it

On ciFreeBSDorg we have two jobs

bull FreeBSD-head-amd64-build-phabric

bull FreeBSD-doc-head-igor-phabric

They are using ldquoquarantine moderdquo of the ldquobuild-in-jailscript setsrdquo and are triggered when a new patch is up-loaded to reviewsFreeBSDorg (fig 5)

And reports the build result back to reviewsFreeBSDorg (fig 6)

63 Clang Static AnalyzerWe created a ldquoclang scan-buildrdquo job with ClangStatic Analyzer [3] and the build script leveraged theldquobsdclang-analyzedmkrdquo from NetBSD [23] simplybuild the whole world and kernel with command in list-ing 5

Listing 5 Command to perform clang scan-buildmake analyze CLANG_ANALYZE_OUTPUT_DIR=clangScanBuildReports CLANG_ANALYZE_OUTPUT=html

The result is taken cared by ldquoClang Scan-Build Plu-ginrdquo [6] which generates good web interface and historytracking (fig 7 8 9)

64 Igor and CheckstyleFor document build we use igor [26] by Warren Blockand Checkstyle [2] for checking the style of documentXML We worked with Warren and added a new option(-X) to igor to output checkstyle XML The respectivejob is FreeBSD-doc-head-igor ldquoCheckstyle Pluginrdquo [5]of Jenkins also provides good web interface and historytracking (fig 10 11 12)

65 Compiler WarningsWe also collect compiler warnings with ldquoWarnings Plu-ginrdquo [16] For showing the trend of compiler warningsand this pushes developers produce higher quality code(fig 13 14)

66 Build Status BadgeWe provide build status badge for embedding in externalweb pages through ldquoEmbeddable Build Status Pluginrdquo[8] For example on httpswikiFreeBSDorgarm64 we have badge as fig 15

To show the latest status of each branch The badgeURLs are

bull httpsltjob-urlgtbadgeicon

bull httpsltjoburlgtlastCompletedBuildbadgeicon

67 OthersWe also use some other plugins for improving the inter-face and helping the maintenance task

bull SCM Sync configuration [14]

ndash Use post-commit hook to notify administra-tors

bull SafeRestart [15]

ndash Schedule restart jenkins when all current jobscomplete

ndash Save and resume the build queue

bull Green balls [9]

ndash To replace blue status ball with green ones tofulfill the convention for most culture

7 Conclusion

In this paper we described about the history and howwe setup the latest CI system of the FreeBSD projectCI does help the development of the FreeBSD projectand we hope these experience and codes can also helpdownstream projects

6

Figure 5 Build triggerd by a patch

Figure 6 Rsult of a build triggerd by a patch

7

Figure 7 Clang scan-build result

8 Future Work

The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas

bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo

bull Check for reproducible build

bull More tests even some are not enabled by defaultDtrace

bull Build for project branchesMake testing feature branches easier

bull Performance tests

bull Work with other projects

We hope these work can make the quality of FreeBSDeven better

AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware

bull jenkins-adminFreeBSDorg

bull The FreeBSD Foundation

bull clusteradminFreeBSDorg

bull phabic-adminFreeBSDorg

bull People on -testing -current -stable mailing lists

References

[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991

[2] Checkstylehttpcheckstylesourceforgenet

[3] Clang Static Analyzerhttpsclang-analyzerllvmorg

[4] JUnithttpjunitorg

[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin

8

Figure 7 Clang scan-build result

8 Future Work

The most important future work is have more people towork on testing FreeBSD providing more build and testscripts And there are some other ideas

bull CI for portsCollaboration with ldquoredportsrdquoAutomatically ldquoexp-runrdquo

bull Check for reproducible build

bull More tests even some are not enabled by defaultDtrace

bull Build for project branchesMake testing feature branches easier

bull Performance tests

bull Work with other projects

We hope these work can make the quality of FreeBSDeven better

AcknowledgmentsWe want to thank following people for sponsoring en-ergy idea and hardware

bull jenkins-adminFreeBSDorg

bull The FreeBSD Foundation

bull clusteradminFreeBSDorg

bull phabic-adminFreeBSDorg

bull People on -testing -current -stable mailing lists

References

[1] Benjamin Cummings Booch Grady Object Ori-ented Design With Applications1991

[2] Checkstylehttpcheckstylesourceforgenet

[3] Clang Static Analyzerhttpsclang-analyzerllvmorg

[4] JUnithttpjunitorg

[5] Jenkins Checkstyle Pluginhttpswikijenkins-ciorgdisplayJENKINSCheckstyle+Plugin

8

Figure 8 Clang scan-build result explanation

Figure 9 Clang scan-build trend

9

Figure 10 Igor + Checksytle - result

Figure 11 Igor + Checkstyle - comparing with previous build

Figure 12 Igor + Checkstyle - trend

10

Figure 13 Compiler Warnings - result

Figure 14 Compiler Warnings - trend

Figure 15 Build satus badge

11

[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin

[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin

[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin

[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls

[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder

[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin

[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin

[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin

[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin

[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin

[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin

[17] Jenkinshttpsjenkinsio

[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua

[19] Phabricatorhttpswwwphacilitycomphabricator

[20] Pure-FTPdhttpswwwpureftpdorg

[21] Sudohttpwwwsudows

[22] The FreeBSD ProjecthttpswwwFreeBSDorg

[23] The NetBSD Projecthttpwwwnetbsdorg

[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml

[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox

[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor

[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve

[28] nginxhttpsnginxorg

12

Figure 10 Igor + Checksytle - result

Figure 11 Igor + Checkstyle - comparing with previous build

Figure 12 Igor + Checkstyle - trend

10

Figure 13 Compiler Warnings - result

Figure 14 Compiler Warnings - trend

Figure 15 Build satus badge

11

[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin

[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin

[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin

[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls

[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder

[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin

[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin

[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin

[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin

[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin

[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin

[17] Jenkinshttpsjenkinsio

[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua

[19] Phabricatorhttpswwwphacilitycomphabricator

[20] Pure-FTPdhttpswwwpureftpdorg

[21] Sudohttpwwwsudows

[22] The FreeBSD ProjecthttpswwwFreeBSDorg

[23] The NetBSD Projecthttpwwwnetbsdorg

[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml

[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox

[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor

[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve

[28] nginxhttpsnginxorg

12

Figure 13 Compiler Warnings - result

Figure 14 Compiler Warnings - trend

Figure 15 Build satus badge

11

[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin

[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin

[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin

[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls

[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder

[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin

[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin

[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin

[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin

[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin

[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin

[17] Jenkinshttpsjenkinsio

[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua

[19] Phabricatorhttpswwwphacilitycomphabricator

[20] Pure-FTPdhttpswwwpureftpdorg

[21] Sudohttpwwwsudows

[22] The FreeBSD ProjecthttpswwwFreeBSDorg

[23] The NetBSD Projecthttpwwwnetbsdorg

[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml

[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox

[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor

[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve

[28] nginxhttpsnginxorg

12

[6] Jenkins Clang Scan-Build Pluginhttpswikijenkins-ciorgdisplayJENKINSClang+Scan-Build+Plugin

[7] Jenkins Email-ext pluginhttpswikijenkins-ciorgdisplayJENKINSEmail-ext+plugin

[8] Jenkins Embeddable Build Status Pluginhttpswikijenkins-ciorgdisplayJENKINSEmbeddable+Build+Status+Plugin

[9] Jenkins Green Balls Pluginhttpswikijenkins-ciorgdisplayJENKINSGreen+Balls

[10] Jenkins Job Builderhttpdocsopenstackorginfrajenkins-job-builder

[11] Jenkins Phabricator Differential Pluginhttpswikijenkins-ciorgdisplayJENKINSPhabricator+Differential+Plugin

[12] Jenkins Publish Over FTP Pluginhttpswikijenkins-ciorgdisplayJENKINSPublish+Over+FTP+Plugin

[13] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin

[14] Jenkins SCM Sync configuration pluginhttpswikijenkins-ciorgdisplayJENKINSSCM+Sync+configuration+plugin

[15] Jenkins SafeRestart Pluginhttpswikijenkins-ciorgdisplayJENKINSSafeRestart+Plugin

[16] Jenkins Warnings Pluginhttpswikijenkins-ciorgdisplayJENKINSWarnings+Plugin

[17] Jenkinshttpsjenkinsio

[18] Merino Julio Kyua Testing framework for in-frastructure softwarehttpsgithubcomjmmvkyua

[19] Phabricatorhttpswwwphacilitycomphabricator

[20] Pure-FTPdhttpswwwpureftpdorg

[21] Sudohttpwwwsudows

[22] The FreeBSD ProjecthttpswwwFreeBSDorg

[23] The NetBSD Projecthttpwwwnetbsdorg

[24] Tinderbox FreeBSD Developersrsquo Handbookhttpswwwfreebsdorgdocenbooksdevelopers-handbooktesting-tinderboxhtml

[25] Tinderbox FreeBSD WikihttpswikifreebsdorgTinderbox

[26] Warren Block igor FreeBSD DocumentationProject sanity check scripthttpwwwwonkitycom˜wblockigor

[27] bhyve(8) FreeBSD System Managerrsquos Manualhttpsmanfreebsdorgbhyve

[28] nginxhttpsnginxorg

12