Wojciech Sliwinski [email protected] Beams Department, Controls Group CERN

CONTINUOUS INTEGRATION AND QUALITY ASSURANCE FOR THE ACCELERATOR

CONTROLS CODEBASE AT CERNJINR/CERN/MEPHI COMPUTING SCHOOL

DUBNA, 24TH OCTOBER 2011

Wojciech [email protected] Department, Controls GroupCERN

Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 2

Outline

Defining Software Quality

Continuous Improvement

Context – Accelerator Controls Codebase

Applying QA & CI for the Controls Codebase

Conclusions

24th October 2011


Outline





Conclusions

24th October 2011


Software Quality (SWQ)…

We know it’s “a good thing” ™

People think…“I know it when I see it”“I know the value of it”

But what are the actual characteristics of SWQ? Do we have a common understanding of SWQ?

The lack of it normally appears as consequences…

24th October 2011


1985: Therac-25 – Software issues…

Radiation Deaths linked to Software ErrorsKilled 2 peopleInjured dozens of othershttp://www.ccnr.org/fatal_dose.html

24th October 2011


1996: Ariane 5 – floating point conversion error

24th October 2011


2007: Airbus A340 – Software issues …

24th October 2011


80s – The Kano Model (Marketing)C

usto

mer

Sat

isfa

ctio

n

Product Features

Basic Needs MetLinear P

erformance

Delight, Surprise,

Innovate

24th October 2011

Satisfied

Dissatisfied

Needs wellfulfilled

Needs notfulfilled


Early and Other Quality Definitions

Zen and the art of motorcycle maintenance, Robert Pirsig, 1974:“If quality exists in an object, then you must explain why scientific

instruments are unable to detect it”“On the other hand, if quality is subjective, [existing only in the eye of

the observer] then this Quality is just a fancy name for whatever you'd like.”

“Quality is not objective. It doesn't reside in the material world [...] Quality is not subjective. It doesn't reside merely in the mind.”

McCall et al, 1977 and Boehm et al, 1978 First elaborate studies on the notion of 'software quality’Quality factors – only indirectly measurable like “reliability”Quality criteria – objective aspects that support the factors, like

“uptime”.

24th October 2011


IEEE SWQ Standards

IEEE Glossary of Software Engineering Terminology defines it as“the degree to which a system, component, or process meets

customer or user needs or expectations” IEEE Std 1061-1992 IEEE Standard for a Software Quality

Metrics MethodologyDoes not actually prescribe any metrics!Defines method to evaluate metrics that could be applicable to

a particular case Plenty more – eg. IEEE Std 730 and friends

24th October 2011


ISO Quality Standards

ISO 9000 series of standards for quality management systems:Process based“the degree to which a set of inherent characteristics fulfils the

requirements”

ISO 9126 Software Quality Characteristics and Metrics, 2001Comprehensive but … large & complex6 key quality attributes:

○ Functionality, Reliability, Usability○ Efficiency, Maintainability, Portability

24th October 2011


SWQ – Huge Set of CharacteristicsFunctionality Suitability Accuracy Interoperability Security Functionality compliance Reliability Maturity Fault tolerance Recoverability Reliability compliance Usability Understandability Learnability Operability Attractiveness Usability compliance Efficiency Time behavior Resource utilization Efficiency compliance Maintainability Analyzability Changeability Stability Testability Maintainability compliance Portability Adaptability Installability Co-existence Replaceability Portability compliance Availability Maintainability Efficiency Portability Flexibility Reusability Integrity Testability Interoperability Reliability Robustness Usability Effectiveness Safety Productivity Satisfaction

• Only a subset applies to all projects• Choose those that suit the project goals

24th October 2011


McCalls’ categories of SWQ factors

Source: J.A. McCall, P.K. Richards and G.F. Walters, Factors in Software Quality,RADC-TR-77-369, 1977, US Dep. of Commerce.

Operation: Correctness Does it do what I want?Reliability Does it do it accurately all of the time?Efficiency Will it run on my hardware as well as it can?Integrity Is it secure?Usability Can I run it?Revision: Maintainability Can I fix it?Testability Can I test it?Flexibility Can I change it?Transition: Portability Will I be able to use it on another machine?Reusability Will I be able to reuse some of the software?Interoperability Will I be able to interface it with another system?

24th October 2011


Internal & External Quality Attributes

Important Primarily to Users Important Primarily to DevelopersAvailability Maintainability

Efficiency Portability

Flexibility ReusabilityIntegrity Testability

Interoperability Reliability

Robustness

Usability

24th October 2011


The SW Quality “Iceberg”

24th October 2011


The Maintainability Index (MI)

Parameter Name Measures

aveV Average Halstead complexity Computational density

aveV (g )′ Average extended cyclomatic complexity Logical complexity

aveLOC Average count of lines of code Code size

PerCM Average percent of lines of comments Human insight

MI = 171 – 5.2 x ln(aveV) - 0.23 x aveV(g’) - 16.2 x ln(aveLOC) + 50 x sin √2.4PerCM

24th October 2011


MI applied to FreeBSD codebase

24th October 2011

Kernel Codebase User Programs

Modules with MI < 40 are rejected !


Applying Quality

Computers don’t care about quality!

Metrics are tests for quality characteristics Quality “clusters”

Testing is ensuring a quality of conformanceBugs occur in clusters too!

Like testing, if quality is low, builds should fail

Quality characteristics must be part of the whole development lifecycleQuality products only come from quality requirements, quality design, quality

code, …24th October 2011


Applying Quality (Assurance)

Software Quality consists of:1. Engineering Methods (proper analysis, design, …)2. Standards and Procedures (common to all)3. Technical Reviews (eg. code reviews)4. Configuration management (repositories, versioning)5. Measurement (code, metrics)6. Testing (unit, system, integration, acceptance)7. Improvement (of the items above)

24th October 2011

• Quality is a way of life• If you think Quality is expensive, try an accident…


Capability Maturity Model (CMM) Levels*

1. Initial (chaotic, ad hoc, individual heroics)○ the starting point for use of a new process.

2. Repeatable○ some processes are repeatable, possibly with consistent results

3. Defined ○ the process is defined/confirmed as a standard business process, and

decomposed to levels 1 and 2

4. Managed (quantatively)○ using process metrics, management can identify ways to adjust and adapt

the process to particular projects

5. Optimized○ process management includes deliberate process

optimization/improvement.

* Capability Maturity Model from SE Institute.24th October 2011


Outline





Conclusions

24th October 2011

Continuous Improvement (CI) – Origins

Deming’s Plan-Do-Check-Act (PDCA) The Toyota Production System

“Stop the line” quality control About eliminating “waste” (obstacles) A learning, non-judgmental whole team approach Adopted by the Agile movement for SWD Caveat: Development is an “empirical process” Introduced when an organization recognizes it has

arrived at some impasse

24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 22

CI - How it works

“People (at all levels) freely discuss the information, issues, ideas, evaluate them, choose, plan and execute improvements”

Focuses on how things get done:Standardized (and ing) common tasksOnly fixing possible issuese.g.: repeated mistakes, time sinks, quality holes

Very suitable where tools, technology, external outcomes and requirements change often


CI - What it relies on

Objective information:To evaluate current situation and processesAssess applied improvementsJudge outcome as worthwhile, or retryDemonstrate added value of CI against overhead

Improvement culture:“Free speech” – everyone’s ideas are welcome A real desire to try it and improveAccepting there will not be total agreementReplacing skeptical inaction with real experimentation


CI – It can fail!

No one “owns” responsibility for implementation No charter to make changes No time allocated for improvement No follow up by team or sponsors


Improvement Practices

Change AgentsAKA: Coaches, Monitors, Consultants, ChampionsOne or more floating people searching for improvements

Kaizen time aka “renovation days”Team plans some improvementsAll actions executed by all at same time

Reflection workshopsRegular meetings to decide on improvementsApply one or more ideasKeep what works


CI - Summary

Team driven incremental changes Team evolves and improves together Drives increase in quality, efficiency and satisfaction Not a tool or technique – it is an attitude Not leader oriented, instead peer based consensus Keeps us ahead in best practice and professionalism



Bibliography

Software Requirements

K. Wiegers, Microsoft Press, 2003

Code Complete: A Practical Handbook of Software Construction S. McConnell, Microsoft Press, 2004

Software Engineering: Principles and Practice H. van Vliet, John Wiley & Sons, 2008

Wikipedia page on SWQ ISO standard 9126.

24th October 2011

Bibliography

The Toyota Way, J. K. Liker, 2004 Agile and Iterative Development, C. Larman, 2004 Sustainable Software Development, Kevin Tate, 2006 Implementing Lean Software Development, M. & T. Poppendick,

2007 Rapid Development, S. McConnell, 1996 The Enterprise and Scrum, K. Schwaber, 2007 Collaboration Explained, J. Tabaka, 2006 Peopleware Papers: The Human side of Software, L. Constantine,

2001



Outline





Conclusions

24th October 2011


Context - CERN Accelerator Complex

24th October 2011


Context - The CERN Control Centre

24th October 2011


TCP/IP communication services


PUBLIC ETHERNET NETWORK

TIMING GENERATION

TT

T

T

T

T T T T

OPERATORCONSOLES

OPERATORCONSOLES

FIXEDDISPLAYS

CE

RN

GIG

AB

IT E

THER

NE

T

TEC

HN

ICA

L N

ETW

OR

K

FILE SERVERS

APPLICATION SERVERS

SCADA SERVERS

RT Lynx/OSVME Front Ends

WORLDFIPFront Ends PLC

LHC MACHINE

ACTUATORS AND SENSORSCRYOGENICS, VACUUM, ETC…

QUENCH PROTECTION AGENTS,POWER CONVERTERS FUNCTIONSGENERATORS, …

BEAM POSITION MONITORS,BEAM LOSS MONITORS,BEAM INTERLOCKS,RF SYSTEMS, ETC…

Wor

ldFI

P

SE

GM

EN

T (1

, 2.5

MB

its/s

ec)

PR

OFI

BU

S

FIP

/IO


OP

TIC

AL

FIB

ER

S

Controls Software Infrastructure

Client tier

Server tier

Resource tier

CTRL CTRL

DB

Business Layer

Device Layer

Presentation Layer

CMW

A 3-tier architecture• Presentation Layer

– Graphical interactive applications

– Fixed Displays• Business Layer

– General purpose and specific Application servers

• Device Layer– Front End

Device servers• Communication to the

equipment goes through Controls MiddleWare CMW

24th October 2011


Controls Software Codebase - Situation

Complex Accelerator domain

~6 million LOC Java/J2EE

~200 developers (CERN + other labs) ~1000 projects/modules

C/C++ ~35 developers (CERN + other labs) +20 major projects

Oracle database Eclipse IDE + plugins SVN + Maven + CommonBuild( Atlassian Suite (JIRA, Fisheye, Crucible, etc.)

Nowadays encourage Scrum Focused on Software Quality

GUI Applications

Control Logic

Middleware

Control System

24th October 2011

Controls Software Codebase - Situation

Lots of Code~6 million lines, 20’000+ classes.… and still growing



Situation – Self Assessment and Outlook

Unknown qualityQuality uncheckedNo mentoring or guidance?Limited standards

High Staff turnoverStudents, Project Associates, Fellows …

More projects coming (PS renovation, …) More service provision (looking after running services) Will this cause future problems like maintenance?

24th October 2011


Outline





Conclusions

24th October 2011


Objectives – improve Software Quality

Introduce quality improvement as an integral part of the everyday development work

Leverage tools to automate the process as much as possible

Establish guidelines and metrics to measure progress

24th October 2011


Quality in the Development Cycle

For each stageAgreed mandatory activities and deliverablesTools identified and integrated with IDE (Eclipse), giving

immediate feedback

Design

Implement, Test,

Document

Deploy, Maintain

24th October 2011


Design Reviews

Design meetings with external members

To verify the soundness of design in an early stage

To identify overlapping functionality

Promote a set of common components and libraries

At the level of sub-components, main classes and design patterns

UML: class and sequence diagrams

Design

Implement, Test,

Document

Deploy, Maintain

24th October 2011


Code Reviews & Code Analysis

Goal: Identify defects Ensure maintainable code Verify conventions are followed

Static code analysis tools to identify common mistakes and bug patterns: FindBugs Checkstyle Eclipse warnings

Person-to-person time consuming Only for critical code Mentoring of junior

developers Light-weight person-to-

person code reviews with FishEye + Crucible

Design

Implement, Test,

Document

Deploy, Maintain

24th October 2011


Files being reviewed

Author and

reviewers Comments inline

24th October 2011

43

Reviewing Code (Crucible)


A list of ‘bugs’

The ‘bug’ line indicated

‘Bug’ explained

24th October 2011


Testing & Continuous Integration

General agreement on benefits of testing and CI

To increase level of testing, unit tests mandatory deliverable of each project >30% test coverage for non-trivial

classes, measured with Clover

To discover inter-project issues early, Continuous Builds with Bamboo: CI server from Atlassian Compiles and runs unit tests Builds fail on:

○ Low test coverage○ Basic PMD rules

Design

Implement, Test,

Document

Deploy, Maintain

24th October 2011

46

Continuous Integration and Test (Bamboo)

Test coverage for project Code metrics

Classes with highest risk


Red = not covered

Green= covered

24th October 2011


Triggered by changes in a dependencyaccsoft-

commons-core

accsoft-common

s-util

accsoft-commons-

concentration

accsoft-commons-

io

Top/flop list generated (Work in progress)

24th October 2011


Integration and System Testing with CO Testbed

CO Testbed - mini-accelerator labHardware and SoftwareCore components from the Accelerators Controls duplicated

into this Test Environment

System and Integration tests are executed by Bamboo CI serverAutomatic execution of tests with “real” systemAutomatic deployment of Release Candidates

24th October 2011

Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

50

CO Testbed Hardware in placeTIMING

FEC03

FEC01 FEC02

FEC04FEC05

SERVER06SERVER07

24th October 2011

51

Integration Tests done through Client APIsBamboo CI Server

CMW Proxy CMW Directory Svc

RBAC A1 Service

ConfigDB

Client APIs (JAPC, CMW, RBAC)

Timing

CMW, RBAC A2, FESA

CMW, RBAC A2, FESA

PPC/LynxOS Linux/i386

TESTS

Woj

ciec

h S

liwin

ski:

CI a

nd Q

A fo

r the

Acc

eler

ator

C

ontro

ls C

odeb

ase

at C

ER

N

JMS

24th October 2011


CO Testbed controlled by Bamboo CI server

24th October 2011


1.

24th October 2011


1.

2.

24th October 2011

Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 5524th October 2011


Challenges

Think of and organize us as one big team, not many small ones

The code belongs to the group, not to a project or an individual developer

The guidelines and standards established and agreed by everyone

Chal

leng

esAgree on standards and configurations

24th October 2011


Encourage developers to invest time on quality

Quality-related tasks part of the yearly objectives for a project

Progress measured against the level a project adheres to guidelines – top/flop lists

Focus the effort where the effect is highest, rely on tools as much as possible

‘CI days’ – common days with a common goal

Chal

leng

es

24th October 2011


Conclusions

Integrated quality improvement in the development cycle

Introduced guidelines/standards and supporting tools

Increased developer motivation through Immediate feedback when developingOfficial tracking of progress and top/flop lists

Increased awareness of benefits, application in individual projects and confidence when making changes

Agile approach reflect, identify ”waste” and improve24th October 2011

Date post:	24-Feb-2016
Category:	Documents
Upload:	zeno
View:	22 times
Download:	1 times

Wojciech Sliwinski [email protected] Beams Department, Controls Group CERN

Documents