Post on 22-May-2020
transcript
Jenkins Shared LibraryKai-Martin Dittkrist, andrena objects
Ralf Schmitt-Roquette, SAP
and Kubernetes
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 2Internal
Overview
• Kubernetes Introduction
• Our Application
• A pipeline run on Jenkins
• Introduction to Shared Libraries
• Two steps further
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 3Internal
Containers are awesome !
But why?
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 4Internal
VM vs Docker Container
Host OS
HypervisorHost OS
Docker Runtime
dockerd
Libs
nginx
Libs
MTA
Tools
Hardware
Libs
MTA
Tools
Hardware
VM
OS
Libraries
nginx
virtual HW
VM
OS
Libraries
nginx
virtual HW
VM
OS
Libraries
nginx
virtual HW
Libs
nginx
Libs
nginx
Libs
nginx
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 5Internal
We have isolated containers!
https://i.kym-cdn.com/photos/images/original/001/142/233/897.gif
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 6Internal
Containers and …
Do you remember, how fun it is to …
▪ start and stop every container?
▪ map ports and don’t get confused?
▪ check the health of a container?
▪ miss a volume on a different host?
▪ …
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 7Internal
Orchestration needed: who takes care ?
Scheduling
Network
Service Discovery
Storage
Scaling
Observability
Security
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 8Internal
Tell me, what you want…
All it takes, is a description and some bricks…
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 9Internal
Host OS
Docker Runtime
do
cke
rd
Hardware
Host OS
Docker Runtime
do
cke
rd
Hardware
Host OS
Docker Runtime
do
cke
rd
HardwareL
ibs
DB
Lib
sD
B
Lib
sD
B
Lib
sIM
Lib
sIM
Lib
sIM
Lb
Web
To
Lb
Web
To
Lb
Web
To
Lb
Web
To
Lb
Web
To
Lib
sM
TA
Lib
sM
TA
Lib
sM
TA
Lib
sM
TA
Lib
sM
TA
Lib
sM
TA
MTA 6 Web 5 IM 3DB 3
Kubernetes does it for you
KubernetesKubernetes
Lib
sM
TA
Lib
sM
TA
Lib
sM
TA
Lib
sM
TA
Lib
sM
TA
Lib
sM
TA
Lb
Web
To
Lb
Web
To
Lb
Web
To
Lb
Web
To
Lb
Web
To
Lib
sD
B
Lib
sD
B
Lib
sD
B
Lib
sIM
Lib
sIM
Lib
sIM
Lb
Web
To
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 10Internal
DemoKubernetes/ Sample Application
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 11Internal
Sample Application: Bulletinboard
• Platform for trading goods
• Everybody can browse the catalog
• Only premium users can create
advertisements
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 12Internal
Bulletinboard in K8s: http://bulletinboard--part-78e2cea9.ingress.testcw43.k8s-train.shoot.canary.k8s-
hana.ondemand.com/ads/
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 13Internal
Bulletinboard in K8s:https://api.testcw43.k8s-train.shoot.canary.k8s-hana.ondemand.com/api/v1/namespaces/kube-
system/services/https:kubernetes-dashboard:/proxy/#!/overview?namespace=part-78e2cea9
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 14Internal
Some facts about Kubernetes (K8s)
▪ Kubernetes = Greek for “helmsman” or “pilot”
▪ Based on Google’s Borg – a cluster manager for container orchestration
▪ Open sourced by Google and firstly announced in 2014
▪ v1.0 was released in July 2015
▪ By now K8s is governed by the Cloud Native Computing Foundation (CNCF)
▪ Vanilla Kubernetes is the basis for commercial products like RedHat’s Openshift or CoreOS’
tectonic
▪ New minor releases roughly every 3 month
▪ Extensive list of beta features
▪ Documentation: https://kubernetes.io/docs/home/ - concepts, tasks, API reference
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 15Internal
Pod
(incomplete) Kubernetes Concepts Map
Container
(your code)
DaemonSet ReplicaSet StatefulSet
Deployment
Job
CronJob
a resource for
every purpose
Volume
ConfigMapPersistent
VolumeClaimSecret
data / persistence
uses / controls
ServiceIngress
networking
Endpoint ServiceAccount
(Cluster) Role
(Cluster)
Rolebinding
IAM
ResourceQuota
LimitRange
Resource
management
Node
PodSecurity
Policy
Network Policy
Policies
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 16Internal
What happens if we run nginx?
Master
API Server
Scheduler
Controller Manager
kubectl
Nodes
Docker
kubelet
Container Container
proxy
Run nginx
etcd
(distributed)
key-value
store
Nodes
Docker
kubelet
Container Container
proxy
Pod Pod
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 17Internal
What happens if we run nginx?
• Control loops
• Check API for changes
Observe
• Current state
• Desired state
• Detect delta
Analyze• Enforce the
desired state
Act
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 18Internal
YAML: “YAML Ain't Markup Language”
▪ YAML is a human friendly data
serialization standard for all
programming languages.
▪ Indentation based
▪ Supports key:value maps and lists
▪ Supports nesting - a value can also
contain another key:value map or a list
YAML
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 19Internal
Sample Application Bulletinboard in K8s
HTTPS/
REST
HTTP/ REST
postgresql
bulletinboard-
ads
bulletinboard-
users
postgresql
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 20Internal
General considerations: Scaling
postgresql postgresql
bulletinboard-
adsbulletinboard-
ads
bulletinboard-
users
✓
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 21Internal
Bulletinboard in K8s: Target picture overall
statefulset
svc
HTTPS/
REST
ingr/
svc
nwp
nwp
nwp
nwp
deployment
statefulset
deployment
bulletinboard-
users
pod
bulletinboard-
ads
pod
HTTP/
REST
postgresql
pod
svc
postgresql
svc
pod
cm
sec
cm
sec
cm
sec
cm
sechttps://github.wdf.sap.corp/slvi/docker-k8s-training/blob/k8s-
bulletinboard/kubernetes/k8s-bulletinboard/README.md
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 22Internal
Bulletinboard in K8s: “Ads DB“
statefulset
svc
HTTPS/
REST
ingr/
svc
nwp
nwp
nwp
nwp
deployment
statefulset
deployment
bulletinboard-
users
pod
bulletinboard-
ads
pod
HTTP/
REST
postgresql
pod
svc
pod
cm
sec
cm
sec
cm
sec
cm
sec
svc
postgresql
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 23Internal
Bulletinboard in K8s: “Ads DB”
statefulset: ads-db-statefulset
pod: ads-db-statefulset-0
Docker
container:
postgres:9.6
configmap:
ads-db-configmap
secret:
ads-db-secret
▪ Configmap:
- postgres db files path
▪ Secret:
- initdb.sql script
- postgres superuser pw
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 24Internal
Bulletinboard in K8s: “Ads DB”
statefulset: ads-db-statefulset
‘headless’ service:
ads-db-service
pod: ads-db-statefulset-0
Docker
container:
postgres:9.6
pod DNS-name: ads-db-statefulset-0.ads-db-service
secret:
ads-db-secret
configmap:
ads-db-configmap
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 25Internal
Bulletinboard in K8s: “Ads App”
statefulset
svc
HTTPS/
REST
ingr/
svc
nwp
nwp
nwp
nwp
deployment
statefulset
deployment
bulletinboard-
users
pod
bulletinboard-
ads
pod
HTTP/
REST
postgresql
pod
svc
postgresql
svc
pod
cm
sec
cm
sec
cm
sec
cm
sec
✓
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 26Internal
Bulletinboard in K8s: “Ads app”
deployment: ads-app-deployment
configmap:
ads-app-
configmap
secret:
ads-app-
secret
bulletinboard-
ads
pod: ads-app…ent-xx-yy
▪ Configmap:
- environment variables:
user_route,
spring_profile_active,
post_user_check
▪ Secret:
- application-k8s.yml
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 27Internal
Bulletinboard in K8s: “Ads app”
deployment: ads-app-deployment
…ingress.<cluster>.k8s-train.shoot.canary.k8s-hana.ondemand.com/ads
configmap:
ads-app-
configmap
configmap:
ads-app-
secret
bulletinboard-
ads
pod: ads-app…ent-xx-yy
ingress:
ads-app-ingress
service:
ads-app-service
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 28Internal
Source Code Repositories
Commit Stage Integration Stage Acceptance Stage Production Stage
Jenkins pipeline + Docker + Kubernetes
Application
source codeDockerfileJenkinsfile pod.yaml
ads:app
ads:db
users:
db
users:
appads:app
ads:db
users:
db
users:
appads:app
ads:db
users:
db
users:
app
Cluster Integration Cluster Acceptance Cluster Production
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 29Internal
Source Code Repositories
Commit Stage Integration Stage Acceptance Stage Production Stage
Jenkins pipeline + Docker + Kubernetes
Application
source codeDockerfileJenkinsfile pod.yaml
ads:app
ads:db
users:
db
users:
appads:app
ads:db
users:
db
users:
appads:app
ads:db
users:
db
users:
app
Cluster A
Namespace Acceptance Namespace ProductionNamespace Integration✓
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 31Internal
Jenkins
• Open source automation server
• Written in Java
• Platform independent
• Standalone or in Application Server
• Extensively extendable via plugins.
• Similar tools: Bamboo, Team City, Travis, CircleCI, …
• ~ 70% market share (from datanyze)
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 32Internal
Deployment Pipeline with Jenkins 2.x
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 33Internal
Pipeline as Code, Jenkinsfiles
• Build and delivery description coded in GroovyDSL
• Typically in a Jenkinsfile in root of project/repository.
• Functions allow modularization of code blocks.
• Steps allow interaction with plugins.
• Either scripted or declarative syntax.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 34Internal
Pipeline as Code, Jenkinsfiles
• Build and deploy description is part of project/repository.• Developers can control how their software is build
• Source code and delivery information connected
• Changes are part of a Version Control System (e.g. git)• Reproduceable how process looked in the past.
• Traceable who changed the process.
• Can easier migrate from one Jenkins server to another.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 35Internal
How a Jenkinsfile looks like
• Scripted syntax pipeline
• Stages as natural blocks
of a pipeline
• Node as worker
• Functions used are
defined further down.
• Can be unit tested.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 38Internal
Motivation
• Used multiple times in one Pipeline!
• What if you have more than one Pipeline?
• Store code at one place and retrieve it.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 39Internal
Overview on Jenkins Shared Libraries
• Storage of Code• Central Repository of common pipeline code
• Modularization
• Less duplicated code
• Supported by Jenkins itself since ~2017
• Easy to maintain, can be tested
• Can have versions.
• Global trusted vs job configured untrusted version.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 40Internal
How to use Shared Libraries
• Stored e.g. in Github
• Configured in Jenkins
• Accessed in Jenkinsfile
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 41Internal
How to write Shared Libraries
• Github repository with at least one src or vars folder
• Jenkins GroovyDSL code (groovy files) like your code in Jenkinsfiles
• You can write object oriented
• ! Do not make complicated constructs Outsource/Externalize !
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 42Internal
Structure of JSLs
• src folder: contains ‘local’ classes• Things you could have more than once
• vars folder: contains pipeline steps/global variables• Will be automatically set up/instantiated at the start
• Singletons (exists only once)
• resources folder: contains files used by your code
• Accessible via libraryResource()
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 43Internal
Example how to create a step I
• Original:
Code as
part of the
Jenkinsfile
• A Jenkins step is
called in here:
‘echo’
• Examples of further steps that could be called ‘withCredentials’,
‘usernamePassword’ and ‘sh’
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 44Internal
Example how to create a step II
• As Step:
File in vars with name
generatePassword.groovy
• Added option to return
password base64 encoded
turned of by default
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 45Internal
How to use custom step generatePassword in Jenkinsfile
• Parameters are passed as normal arguments, so the call does not
change:
• Before: generatePassword(12)
• Now: generatePassword(12)
• But we could also do:generatePassword(12,true)
and get the password already base64 encoded
• In our custom step other Jenkins steps like ‘echo’ are still directly
callable!
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 46Internal
DemoPipeline with shared Library
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 48Internal
Piper / Project Piper
• Lots of different teams need same functionality
• Template Jenkinsfile as a start for CI in teams.
• Teams can use some steps or a complete template pipeline.
• Problems: • Evolution of template Jenkinsfile Piper 3.0
• Backwards compatibility of steps
• How much can be provided centrally
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 49Internal
Hierarchy of shared Libraries
Multiple shared libraries in SAP:
• On team level: e.g. `bulletinboard-JSL`
• SAP wide internal shared library: `piper-lib`
• World wide open source shared library: `piper-lib-os`
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 50Internal
SAP Project Piper
• Shared Library open source project from SAP
Library: https://github.com/SAP/jenkins-library/
Documentation: https://sap.github.io/jenkins-library/
• Provides steps and classes used in and external of SAP.
• Step configuration either done via parameters or by values
defined in config.yml file.
• Also provides a template pipeline you only need to configure via a
config.yml but you can extend and customize.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 51Internal
An Example: dockerExecute
• Allows to execute some part of your pipeline inside of a docker
image and makes this easy to use:
• Wrapper around docker.image()
• Allows to run also a sidecar container e.g. for selenium Tests.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 52Internal
A whole pipeline in a shared library
• Jenkinsfile:
• config.yml:
• Needs multibranch job.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 54Internal
Summary and Thank you
• Kubernetes is awesome!
• Use Jenkinsfiles for your Jenkins Pipelines!
• Move reusable code into Shared Libraries!
• Check out Project Piper / Jenkins Library of SAP!
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 57Internal
Testing of Jenkinsfiles
JenkinsPipelineUnit framework from Lesfuretes
https://github.com/jenkinsci/JenkinsPipelineUnit
Can write unitlike tests for Jenkinsfiles. Allows testing of e.g:
• syntax
• execution order of steps
Will be executed with each build.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 58Internal
Setup of JenkinsPipelineUnit in Maven-Java-Project
Added 1 dependency and 2 plugins to pom.xml
Added JenkinsfileNoLibTest.groovy file to src/test/groovy
Extends BasePipelineTest class from framework
A lot of basic methods are already mocked in that class
Test class needs a @before call of super.setup()
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 59Internal
DemoJenkinsfile Unit Test
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 61Internal
Testing in Piper
• Best Framework for Testing JSLs is part of Piper / Piper-OS
• Maven pom.xml• Plugins: build-helper-maven-plugin used to adapt to JSL folder structure
maven-compiler-plugin to change compiler to groovy-eclipse-compiler
• Important dependency: com.lesfurets.jenkins-pipeline-unit
• Test Utils in test/groovy/utils:• BasePiperTest class extending BasePipelineTest
• BasePiperTestContext class setting basic mock beans
• Rules for more complex mocking.
• JenkinsStepRule used to call the step you want to test.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 64Internal
Overview appendix
• Theory on classes in src & example
• Theory on classes/global variables in vars & example
• More theory on steps in general
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 65Internal
src – classes
• Two kinds, with or without the scope of an enclosing class.The following UNOFFICAL classification names Type 1/2 are from us and used to distinguish them.
• Type 1: Without scope: • Allows directly calling steps and stateless global variables
• Collection of “small” functions
• No state/member variables
• Type 2: With scope:• Needs script to call steps and global variables
• Can have member variables.
• Need a package name and corresponding folder substructure
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 66Internal
Example of type 2 class in src I
• FileUtils: does string replacement in files
• Old:
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 67Internal
Example of class in src II
• New:
FileUtils.groovyin
src/com/sap/cc/Jenkins
• Constructor gets
Script scriptpassed so other steps and
global variables can be
called as script.xyz()e.g. script.readFile()
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 68Internal
How to use custom class
• Need to import class
into our Jenkinsfile:
• Create new instance of class FileUtils (L78) and use it (L82):
• ‘this’ holds the script with all global variables etc.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 69Internal
vars – global steps and variables
• Three kinds: steps and two kinds of classes
• Get instantiated globally at the start of the pipeline -> exist only once!
• Classes
• Same as before (with and without class scope)
• Type 2 can only have default constructor.
• No package declaration
• Custom steps:
• Callable like normal steps in your pipeline
• File must be named all lower-case or “camelCased”
• Also allows directly calling steps and using global variables
• Just a ‘special’ kind of Type 1 class.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 70Internal
• Move small function executeShell to a global Class
• Old code:
• New code: file shell.groovy in vars
• Use: executeShell changes to shell.execute
Example of global Class (Type 1)
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 71Internal
Step in general
• Groovy file in vars folder, name of file is the name of the step.
• Direct use of other steps and global variables is possible.
• Groovy provides two ways to handle function parameters.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 72Internal
Classic parameter declaration
Example: assume in step.groovy:
def call(String aString=‘defaultString’,int aNumber=4711){ … }
• Easier to comprehend for short list of parameters
• Calling the step gets easier when you know the order of parameters
• Can give default values for each parameter
• How to call such a step:• step(‘hi’,10) aString will be ‘hi’, aNumber will be 10
• step(‘hi’) aString will be ‘hi’, aNumber will be 4711
• step(10) throws MissingMethodException because first parameter has to be a String
• There is no way to only give the integer to this function without providing a string.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 73Internal
Named agruments/Parameters Map
Example: assume in step.groovy:
def call(Map parameters = [:]){
String aString=parameters.get(‘aString’,‘defaultString’)
int aNumber=parameters.get(‘aNumber’,4711)
…
}
• Easier for long list of parameters
• Maintaining them can get confusing
• Use parameters.<name of a parameter> to access without a defaultValue
• How to call such a step:• step(aString:’hi’,aNumber:’10) aString will be ‘hi’, aNumber will be 10
• atep(aString:’hi’) aString will be ‘hi’, aNumber will be 4711
• step(aNumber:’10’) aString will be ‘defaultString’, aNumber will be 10
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 74Internal
Remarks and more theory to look up for yourself
• Jenkins GroovyDSL is less then Groovy(e.g. calls of Jenkins steps in class constructors not possible)
• If possible make classes (with members) serializable
• Uses continuation-passing style transformation(can be ‘overcome’ with @NonCPS annotation)
• If you want to use more then one JSL the names in vars should be
unique between the two.
© 2019 SAP SE or an SAP affiliate company. All rights reserved. 75Internal
References
• Groovy Documentation: http://docs.groovy-lang.org/docs/groovy-
2.4.12/html/documentation/
• NonCPS: https://github.com/jenkinsci/workflow-cps-plugin
• Shared Library: https://jenkins.io/doc/book/pipeline/shared-libraries/
• Examples: • Piper OS: https://github.com/SAP/jenkins-library