Jenkins Shared Library - andrena...Jenkins pipeline + Docker + Kubernetes Application source code...

Jenkins Shared LibraryKai-Martin Dittkrist, andrena objects

Ralf Schmitt-Roquette, SAP

and Kubernetes

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 2Internal

Overview

• Kubernetes Introduction

• Our Application

• A pipeline run on Jenkins

• Introduction to Shared Libraries

• Two steps further


Containers are awesome !

But why?


VM vs Docker Container

Host OS

HypervisorHost OS

Docker Runtime

dockerd

Libs

nginx

Libs

MTA

Tools

Hardware

Libs

MTA

Tools

Hardware

VM

OS

Libraries

nginx

virtual HW

VM

OS

Libraries

nginx

virtual HW

VM

OS

Libraries

nginx

virtual HW

Libs

nginx

Libs

nginx

Libs

nginx


We have isolated containers!

https://i.kym-cdn.com/photos/images/original/001/142/233/897.gif

https://i.kym-cdn.com/photos/images/original/001/142/233/897.gif


Containers and …

Do you remember, how fun it is to …

▪ start and stop every container?

▪ map ports and don’t get confused?

▪ check the health of a container?

▪ miss a volume on a different host?

▪ …


Orchestration needed: who takes care ?

Scheduling

Network

Service Discovery

Storage

Scaling

Observability

Security


Tell me, what you want…

All it takes, is a description and some bricks…


Host OS

Docker Runtime

do

cke

rd

Hardware

Host OS

Docker Runtime

do

cke

rd

Hardware

Host OS

Docker Runtime

do

cke

rd

HardwareL

ibs

DB

Lib

sD

B

Lib

sD

B

Lib

sIM

Lib

sIM

Lib

sIM

Lb

Web

To

Lb

Web

To

Lb

Web

To

Lb

Web

To

Lb

Web

To

Lib

sM

TA

Lib

sM

TA

Lib

sM

TA

Lib

sM

TA

Lib

sM

TA

Lib

sM

TA

MTA 6 Web 5 IM 3DB 3

Kubernetes does it for you

KubernetesKubernetes

Lib

sM

TA

Lib

sM

TA

Lib

sM

TA

Lib

sM

TA

Lib

sM

TA

Lib

sM

TA

Lb

Web

To

Lb

Web

To

Lb

Web

To

Lb

Web

To

Lb

Web

To

Lib

sD

B

Lib

sD

B

Lib

sD

B

Lib

sIM

Lib

sIM

Lib

sIM

Lb

Web

To


DemoKubernetes/ Sample Application


Sample Application: Bulletinboard

• Platform for trading goods

• Everybody can browse the catalog

• Only premium users can create

advertisements


Bulletinboard in K8s: http://bulletinboard--part-78e2cea9.ingress.testcw43.k8s-train.shoot.canary.k8s-

hana.ondemand.com/ads/

http://bulletinboard--part-78e2cea9.ingress.testcw43.k8s-train.shoot.canary.k8s-hana.ondemand.com/ads/


Bulletinboard in K8s:https://api.testcw43.k8s-train.shoot.canary.k8s-hana.ondemand.com/api/v1/namespaces/kube-

system/services/https:kubernetes-dashboard:/proxy/#!/overview?namespace=part-78e2cea9

https://api.testcw43.k8s-train.shoot.canary.k8s-hana.ondemand.com/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/overview?namespace=part-78e2cea9


Some facts about Kubernetes (K8s)

▪ Kubernetes = Greek for “helmsman” or “pilot”

▪ Based on Google’s Borg – a cluster manager for container orchestration

▪ Open sourced by Google and firstly announced in 2014

▪ v1.0 was released in July 2015

▪ By now K8s is governed by the Cloud Native Computing Foundation (CNCF)

▪ Vanilla Kubernetes is the basis for commercial products like RedHat’s Openshift or CoreOS’

tectonic

▪ New minor releases roughly every 3 month

▪ Extensive list of beta features

▪ Documentation: https://kubernetes.io/docs/home/ - concepts, tasks, API reference

https://kubernetes.io/docs/home/


Pod

(incomplete) Kubernetes Concepts Map

Container

(your code)

DaemonSet ReplicaSet StatefulSet

Deployment

Job

CronJob

a resource for

every purpose

Volume

ConfigMapPersistent

VolumeClaimSecret

data / persistence

uses / controls

ServiceIngress

networking

Endpoint ServiceAccount

(Cluster) Role

(Cluster)

Rolebinding

IAM

ResourceQuota

LimitRange

Resource

management

Node

PodSecurity

Policy

Network Policy

Policies


What happens if we run nginx?

Master

API Server

Scheduler

Controller Manager

kubectl

Nodes

Docker

kubelet

Container Container

proxy

Run nginx

etcd

(distributed)

key-value

store

Nodes

Docker

kubelet

Container Container

proxy

Pod Pod


What happens if we run nginx?

• Control loops

• Check API for changes

Observe

• Current state

• Desired state

• Detect delta

Analyze• Enforce the

desired state

Act


YAML: “YAML Ain't Markup Language”

▪ YAML is a human friendly data

serialization standard for all

programming languages.

▪ Indentation based

▪ Supports key:value maps and lists

▪ Supports nesting - a value can also

contain another key:value map or a list

YAML


Sample Application Bulletinboard in K8s

HTTPS/

REST

HTTP/ REST

postgresql

bulletinboard-

ads

bulletinboard-

users

postgresql


General considerations: Scaling

postgresql postgresql

bulletinboard-

adsbulletinboard-

ads

bulletinboard-

users

✓


Bulletinboard in K8s: Target picture overall

statefulset

svc

HTTPS/

REST

ingr/

svc

nwp

nwp

nwp

nwp

deployment

statefulset

deployment

bulletinboard-

users

pod

bulletinboard-

ads

pod

HTTP/

REST

postgresql

pod

svc

postgresql

svc

pod

cm

sec

cm

sec

cm

sec

cm

sechttps://github.wdf.sap.corp/slvi/docker-k8s-training/blob/k8s-

bulletinboard/kubernetes/k8s-bulletinboard/README.md

https://github.wdf.sap.corp/slvi/docker-k8s-training/blob/k8s-bulletinboard/kubernetes/k8s-bulletinboard/README.md


Bulletinboard in K8s: “Ads DB“

statefulset

svc

HTTPS/

REST

ingr/

svc

nwp

nwp

nwp

nwp

deployment

statefulset

deployment

bulletinboard-

users

pod

bulletinboard-

ads

pod

HTTP/

REST

postgresql

pod

svc

pod

cm

sec

cm

sec

cm

sec

cm

sec

svc

postgresql


Bulletinboard in K8s: “Ads DB”

statefulset: ads-db-statefulset

pod: ads-db-statefulset-0

Docker

container:

postgres:9.6

configmap:

ads-db-configmap

secret:

ads-db-secret

▪ Configmap:

- postgres db files path

▪ Secret:

- initdb.sql script

- postgres superuser pw


Bulletinboard in K8s: “Ads DB”

statefulset: ads-db-statefulset

‘headless’ service:

ads-db-service

pod: ads-db-statefulset-0

Docker

container:

postgres:9.6

pod DNS-name: ads-db-statefulset-0.ads-db-service

secret:

ads-db-secret

configmap:

ads-db-configmap


Bulletinboard in K8s: “Ads App”

statefulset

svc

HTTPS/

REST

ingr/

svc

nwp

nwp

nwp

nwp

deployment

statefulset

deployment

bulletinboard-

users

pod

bulletinboard-

ads

pod

HTTP/

REST

postgresql

pod

svc

postgresql

svc

pod

cm

sec

cm

sec

cm

sec

cm

sec

✓


Bulletinboard in K8s: “Ads app”

deployment: ads-app-deployment

configmap:

ads-app-

configmap

secret:

ads-app-

secret

bulletinboard-

ads

pod: ads-app…ent-xx-yy

▪ Configmap:

- environment variables:

user_route,

spring_profile_active,

post_user_check

▪ Secret:

- application-k8s.yml


Bulletinboard in K8s: “Ads app”

deployment: ads-app-deployment

…ingress.<cluster>.k8s-train.shoot.canary.k8s-hana.ondemand.com/ads

configmap:

ads-app-

configmap

configmap:

ads-app-

secret

bulletinboard-

ads

pod: ads-app…ent-xx-yy

ingress:

ads-app-ingress

service:

ads-app-service


Source Code Repositories

Commit Stage Integration Stage Acceptance Stage Production Stage

Jenkins pipeline + Docker + Kubernetes

Application

source codeDockerfileJenkinsfile pod.yaml

ads:app

ads:db

users:

db

users:

appads:app

ads:db

users:

db

users:

appads:app

ads:db

users:

db

users:

app

Cluster Integration Cluster Acceptance Cluster Production


Source Code Repositories

Commit Stage Integration Stage Acceptance Stage Production Stage

Jenkins pipeline + Docker + Kubernetes

Application

source codeDockerfileJenkinsfile pod.yaml

ads:app

ads:db

users:

db

users:

appads:app

ads:db

users:

db

users:

appads:app

ads:db

users:

db

users:

app

Cluster A

Namespace Acceptance Namespace ProductionNamespace Integration✓

A pipeline run on Jenkins


Jenkins

• Open source automation server

• Written in Java

• Platform independent

• Standalone or in Application Server

• Extensively extendable via plugins.

• Similar tools: Bamboo, Team City, Travis, CircleCI, …

• ~ 70% market share (from datanyze)

https://jenkins.io/


Deployment Pipeline with Jenkins 2.x


Pipeline as Code, Jenkinsfiles

• Build and delivery description coded in GroovyDSL

• Typically in a Jenkinsfile in root of project/repository.

• Functions allow modularization of code blocks.

• Steps allow interaction with plugins.

• Either scripted or declarative syntax.


Pipeline as Code, Jenkinsfiles

• Build and deploy description is part of project/repository.• Developers can control how their software is build

• Source code and delivery information connected

• Changes are part of a Version Control System (e.g. git)• Reproduceable how process looked in the past.

• Traceable who changed the process.

• Can easier migrate from one Jenkins server to another.


How a Jenkinsfile looks like

• Scripted syntax pipeline

• Stages as natural blocks

of a pipeline

• Node as worker

• Functions used are

defined further down.

• Can be unit tested.


DemoNo Library Pipeline

Introduction Shared Libraries


Motivation

• Used multiple times in one Pipeline!

• What if you have more than one Pipeline?

• Store code at one place and retrieve it.


Overview on Jenkins Shared Libraries

• Storage of Code• Central Repository of common pipeline code

• Modularization

• Less duplicated code

• Supported by Jenkins itself since ~2017

• Easy to maintain, can be tested

• Can have versions.

• Global trusted vs job configured untrusted version.


How to use Shared Libraries

• Stored e.g. in Github

• Configured in Jenkins

• Accessed in Jenkinsfile


How to write Shared Libraries

• Github repository with at least one src or vars folder

• Jenkins GroovyDSL code (groovy files) like your code in Jenkinsfiles

• You can write object oriented

• ! Do not make complicated constructs Outsource/Externalize !


Structure of JSLs

• src folder: contains ‘local’ classes• Things you could have more than once

• vars folder: contains pipeline steps/global variables• Will be automatically set up/instantiated at the start

• Singletons (exists only once)

• resources folder: contains files used by your code

• Accessible via libraryResource()


Example how to create a step I

• Original:

Code as

part of the

Jenkinsfile

• A Jenkins step is

called in here:

‘echo’

• Examples of further steps that could be called ‘withCredentials’,

‘usernamePassword’ and ‘sh’


Example how to create a step II

• As Step:

File in vars with name

generatePassword.groovy

• Added option to return

password base64 encoded

turned of by default


How to use custom step generatePassword in Jenkinsfile

• Parameters are passed as normal arguments, so the call does not

change:

• Before: generatePassword(12)

• Now: generatePassword(12)

• But we could also do:generatePassword(12,true)

and get the password already base64 encoded

• In our custom step other Jenkins steps like ‘echo’ are still directly

callable!


DemoPipeline with shared Library

Two steps further


Piper / Project Piper

• Lots of different teams need same functionality

• Template Jenkinsfile as a start for CI in teams.

• Teams can use some steps or a complete template pipeline.

• Problems: • Evolution of template Jenkinsfile Piper 3.0

• Backwards compatibility of steps

• How much can be provided centrally


Hierarchy of shared Libraries

Multiple shared libraries in SAP:

• On team level: e.g. `bulletinboard-JSL`

• SAP wide internal shared library: `piper-lib`

• World wide open source shared library: `piper-lib-os`


SAP Project Piper

• Shared Library open source project from SAP

Library: https://github.com/SAP/jenkins-library/

Documentation: https://sap.github.io/jenkins-library/

• Provides steps and classes used in and external of SAP.

• Step configuration either done via parameters or by values

defined in config.yml file.

• Also provides a template pipeline you only need to configure via a

config.yml but you can extend and customize.

https://github.com/SAP/jenkins-library/

https://sap.github.io/jenkins-library/


An Example: dockerExecute

• Allows to execute some part of your pipeline inside of a docker

image and makes this easy to use:

• Wrapper around docker.image()

• Allows to run also a sidecar container e.g. for selenium Tests.


A whole pipeline in a shared library

• Jenkinsfile:

• config.yml:

• Needs multibranch job.


DemoUsers Pipeline


Summary and Thank you

• Kubernetes is awesome!

• Use Jenkinsfiles for your Jenkins Pipelines!

• Move reusable code into Shared Libraries!

• Check out Project Piper / Jenkins Library of SAP!

Extended Part

Testing of Jenkinsfiles


Testing of Jenkinsfiles

JenkinsPipelineUnit framework from Lesfuretes

https://github.com/jenkinsci/JenkinsPipelineUnit

Can write unitlike tests for Jenkinsfiles. Allows testing of e.g:

• syntax

• execution order of steps

Will be executed with each build.

https://github.com/jenkinsci/JenkinsPipelineUnit


Setup of JenkinsPipelineUnit in Maven-Java-Project

Added 1 dependency and 2 plugins to pom.xml

Added JenkinsfileNoLibTest.groovy file to src/test/groovy

Extends BasePipelineTest class from framework

A lot of basic methods are already mocked in that class

Test class needs a @before call of super.setup()


DemoJenkinsfile Unit Test

Testing of Shared Libraries


Testing in Piper

• Best Framework for Testing JSLs is part of Piper / Piper-OS

• Maven pom.xml• Plugins: build-helper-maven-plugin used to adapt to JSL folder structure

maven-compiler-plugin to change compiler to groovy-eclipse-compiler

• Important dependency: com.lesfurets.jenkins-pipeline-unit

• Test Utils in test/groovy/utils:• BasePiperTest class extending BasePipelineTest

• BasePiperTestContext class setting basic mock beans

• Rules for more complex mocking.

• JenkinsStepRule used to call the step you want to test.


DemoPiper Tests

Appendix Shared Library


Overview appendix

• Theory on classes in src & example

• Theory on classes/global variables in vars & example

• More theory on steps in general


src – classes

• Two kinds, with or without the scope of an enclosing class.The following UNOFFICAL classification names Type 1/2 are from us and used to distinguish them.

• Type 1: Without scope: • Allows directly calling steps and stateless global variables

• Collection of “small” functions

• No state/member variables

• Type 2: With scope:• Needs script to call steps and global variables

• Can have member variables.

• Need a package name and corresponding folder substructure


Example of type 2 class in src I

• FileUtils: does string replacement in files

• Old:


Example of class in src II

• New:

FileUtils.groovyin

src/com/sap/cc/Jenkins

• Constructor gets

Script scriptpassed so other steps and

global variables can be

called as script.xyz()e.g. script.readFile()


How to use custom class

• Need to import class

into our Jenkinsfile:

• Create new instance of class FileUtils (L78) and use it (L82):

• ‘this’ holds the script with all global variables etc.


vars – global steps and variables

• Three kinds: steps and two kinds of classes

• Get instantiated globally at the start of the pipeline -> exist only once!

• Classes

• Same as before (with and without class scope)

• Type 2 can only have default constructor.

• No package declaration

• Custom steps:

• Callable like normal steps in your pipeline

• File must be named all lower-case or “camelCased”

• Also allows directly calling steps and using global variables

• Just a ‘special’ kind of Type 1 class.


• Move small function executeShell to a global Class

• Old code:

• New code: file shell.groovy in vars

• Use: executeShell changes to shell.execute

Example of global Class (Type 1)


Step in general

• Groovy file in vars folder, name of file is the name of the step.

• Direct use of other steps and global variables is possible.

• Groovy provides two ways to handle function parameters.


Classic parameter declaration

Example: assume in step.groovy:

def call(String aString=‘defaultString’,int aNumber=4711){ … }

• Easier to comprehend for short list of parameters

• Calling the step gets easier when you know the order of parameters

• Can give default values for each parameter

• How to call such a step:• step(‘hi’,10) aString will be ‘hi’, aNumber will be 10

• step(‘hi’) aString will be ‘hi’, aNumber will be 4711

• step(10) throws MissingMethodException because first parameter has to be a String

• There is no way to only give the integer to this function without providing a string.


Named agruments/Parameters Map

Example: assume in step.groovy:

def call(Map parameters = [:]){

String aString=parameters.get(‘aString’,‘defaultString’)

int aNumber=parameters.get(‘aNumber’,4711)

…

}

• Easier for long list of parameters

• Maintaining them can get confusing

• Use parameters.<name of a parameter> to access without a defaultValue

• How to call such a step:• step(aString:’hi’,aNumber:’10) aString will be ‘hi’, aNumber will be 10

• atep(aString:’hi’) aString will be ‘hi’, aNumber will be 4711

• step(aNumber:’10’) aString will be ‘defaultString’, aNumber will be 10


Remarks and more theory to look up for yourself

• Jenkins GroovyDSL is less then Groovy(e.g. calls of Jenkins steps in class constructors not possible)

• If possible make classes (with members) serializable

• Uses continuation-passing style transformation(can be ‘overcome’ with @NonCPS annotation)

• If you want to use more then one JSL the names in vars should be

unique between the two.

https://en.wikipedia.org/wiki/Continuation-passing_style


References

• Groovy Documentation: http://docs.groovy-lang.org/docs/groovy-

2.4.12/html/documentation/

• NonCPS: https://github.com/jenkinsci/workflow-cps-plugin

• Shared Library: https://jenkins.io/doc/book/pipeline/shared-libraries/

• Examples: • Piper OS: https://github.com/SAP/jenkins-library

http://docs.groovy-lang.org/docs/groovy-2.4.12/html/documentation/

https://github.com/jenkinsci/workflow-cps-plugin

https://jenkins.io/doc/book/pipeline/shared-libraries/

https://github.com/SAP/jenkins-library

Date post:	22-May-2020
Category:	Documents
Upload:	others
View:	40 times
Download:	1 times

Jenkins Shared Library - andrena...Jenkins pipeline + Docker + Kubernetes Application source code...

Documents