Date post: | 24-Dec-2015 |
Category: |
Documents |
Upload: | nathan-kelly |
View: | 215 times |
Download: | 1 times |
Quality Assurance and Testing in LCG
CHEP 2004 Interlaken, Switzerland 30 September 2004
Manuel Gallas, Jakub MOSCICKI
CERN
http://[email protected]
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 2
Outline:
Quality Assurance and Testing in LCG
•LCG SPI Context
•SPI Services Overview
•Sw-Testing:•Test Frameworks •Test Documents•User support
•Quality Assurance:•QA activities•Bug tracker statistics•Test coverage
•LCG App.Area policies
•Summary
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 3
LCG SPI context
LCG Infrastructure•Common services
•Similar ways of working (process)
•Tools, templates, training
•General QA, tests, integration, release
LCG Application Area software projects
•POOL: Persistency
•SEAL: Core common software
•PI: Physics Interfaces
•SIMU: Simulation
• …etc…
LHC grid software applications
(LHC experiments, projects, etc)
LCG SPI project
LCG Application Area
SIM
ULA
TIO
N
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 4
SPI Services Overview
Provide General Services needed by each project• CVS repository, Web Site, Software Library• Mailing Lists, Bug Reports, Task Management, Collaborative Facilities
Provide solutions specific to the Software Development phases• Tools, Templates, Policies, Support, Documentation, Examples• Software Testing and QA
Coding Analysis and Design Development
Release
SpecificationsTesting
Build systemsDeployment and
Installation Documentation
Quality Assurance
Software Development
General ServicesCVS service
Collaborative Facilities Task Management Mailing Lists
Web Portal External Software
SP
I p
roje
ct
The rest of the talk describes QA and Sw-Testing services
SPI Web Site - http://spi.cern.ch
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 5
•Software testing should be an integral part of the software development in the LCG App Area.•All level of software testing should be run as part of an automatic process.
SW-Testing Overview - http://spi.cern.ch/testing
Software testing
Integration test
System test
Acceptance test
Unit test
Sw-testing team
IntegrationTests
System Tests
ExamplesLHC experiments
Use in Exp.Test FrameWorks
SW-testing doc
SW- testing support
SPI SW-Testing
SW-testing polices
Code Documentation
Software developer
CODE
Work PackageTest
Automated testing
Nightly Building
System
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 6
LCG-SPI SW-Testing
Aim: to help developers: to help projects:
• to produce test code suitable to be run in automatic way • to document their tests
• to plan their test phase• to run all their test in automatic way • to browse and scan easily the test results and failures
Our inputs: • Contacts within HEP-community.• What is available as free open source code.
Our constraints: • Avoid commercial software and licensing problems. • Avoid “do it yourself solutions”•Try to adopt commonly used open-source software.
Requirements from the users : • Work with different languages (C++, Python, …) and in different environments (platforms & compilers)• Allow two ways of testing: (1) “check the output of the test-code” (2) “check the test results inside the test-code”.• Provide a easy way to integrate existing tests.• Allow the creation of dependencies among tests.• Easily establish a criteria if a test should pass or fail in a given release• Organize the tests by components or packages / integration / system /acceptance .• Regression testing and check respect to the previous sw-release.• Easy integration with the Nightly Building System.• Provide also a graphical interface to run the tests and examine the test results.
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 7
Test Frameworks: Global picture
X-Unit familyJUnit QtUnit
CppUnit PyUnit
Oval
Test Scripts
Old tests
SW Product Examples
Unit testing
Acceptanc
etesting
QA activity
End User Installation
checks
Nightly Building
Project Release
Sw-Testing
QMTest• Uses a GUI for creating and running tests (also in batch).• Can run tests in parallel, supports execution of a single test or many at once (test-cases & test-suites) • Organizes tests hierarchically• Records dependencies among tests
Top layer
Bottom layer
- Adaptable to theprogramming language and developer- Prepared to be run in automatic way
- Integrates different ways to test.
- Common environment to run the tests and to access the test results.
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 8
•Main Scope: Unit–testing
•The same “assertion style” in different languages.
•Provides:- The name of the test case that failed.
-The name of the source file that contains the test.
-The line number where the failure occurred
• Different platforms/compilers: (Linux/Solaris/Windows)
Test Frameworks: X-Unit family
X-Unit family
JUnit
QtUnit
CppUnit
PyUnit
A simple test:
1. Subclass the TestCase CppUnit or PyUnit class
2. Override the method runTest().
3. When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 9
Test Frameworks: Oval
Oval:• validation and regression • developed & used within CMS experiment
•Can be use for Unit-test to Validation tests.
•Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”)
• It is possible to set different run environments.
• Can run external scripts and external binaries.
Ovalfile(configuration)
ExecutableTest
Source Code
Referencefile
Log file
Oval run
Ovaldiff
Oval build
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 10
Test Frameworks: QMTest
QMTest
•Uses a graphical interface for creating and running tests
•The configuration files are in XML and can be created from the GUI. We provide also script to do it
• Runs tests in parallel
• Organizes tests hierarchically
• Supports execution of a single test or many at once
• Records dependencies among tests
• Can be run in batch mode -> easy integration with the Nightly-Building systems
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 11
Test Documents
• For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities.• We use a template•Test Case template collects information about the existing tests and how they cover the expected functionality of the software.
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 12
User support: Web information
http://spi.cern.ch/
Test Frameworks
HowTo
Test doc
Policies
http://spi.cern.ch/testing
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 13
User support: How-To
*
*
3 Follow sw-testing policies
Automatic testing requires some standardization
1
2
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 14
Quality Assurance Service - http://spi.cern.ch/qa
• The main goal of QA activity is help LCG projects
• assess and improve the quality of the software
• provide tools to collect useful metrics/statistics which help toasses quality;
• generate reports;• verify if project setup is correct with LCG
policies.
• QA Tools and Focus • Automatic reports
• Development/integration of automatic tools
• LCG Policies• agreed and defined by AF• SPI supports them in the tools and
procedures and only helps to work them out
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 15
Quality Assurance Activities
QA Checklist on each Release• Build the release • Run automatic tests• Metrics
• Test Inventory & Metrics• Documentation/Examples Inventory • Bug tracker Statistics • Code Inventory • Rule Checker , Logiscope
• LCG Policies • Configuration of a build system • CVS directory structure • Software testing polices
Automatic QA Reports
• Generated at every release
• Published on the SPI web site
• External tools• Test coverage
• SLOC, Valgrind, ignominy
• QA procedure: well-defined, transparent, open- clear rules and checklist of assessed items- anybody at anytime may see statistics - anybody may create reports themselves- anybody may contribute
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 16
QA Reports
Bug tracker statisticsTesting inventory by packageResults of the automatic testingTest coverageSource code statistic (n files, n lines, sloc, .. )CVS structure
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 17
QA reports: Bugtracker Statistics
• Defect Analysis in Time Defect Reparation Trends Activity of the project / users Flexible category selection
subsystems, severity,... Caveat:
- not all bug reports are“real” bugs
• Correlation of defects and testsTraceability
oEvery bug fix should be accompanied with a test case
oTest case should contain a bug reference #Avoiding reappearing bugs
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 18
QA reports: Test Coverage
• Coverage of• Unit tests
• Integration tests
• Validation tests
• At the level of• Project
• Package
• File
• Source Lines
• Tools• gcc3.2 –gcov
• LTP scripts
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 19
LCG Policies
• LCG Policies• CVS and Build Directory Policy • Software Testing Policies • Version Numbers, Tagging and Release
Procedure • Installation Directory Structure • Platform string, binary names,
debug flags and more
• They are a needed by the LCG• They are defined by the LCG projects (AF),
collected by SPI• SPI supports them in the tools and
procedures and only helps to work them out
• If everything is different is too difficult to use and to automate
• compromising on our habits, for project needs
• tell when they are not followed
• First time that this exists at this extent, and that is checked
QA tools verify the compliance of software to the Policies
M.Gallas, J.Moscicki CERN PH-SFT
SPI - Software Process & Infrastructure 20
• QA activities• designed to match the HEP environment
- focused more on testing and bug tracking- less on sophisticated source code metrics
• integrated and automated into the LCG development environment
• QA infrastructure will be reused in EGEE
Summary
Quality Assurance and Testing in LCG
•LCG SPI Context
•SPI Services Overview
•Sw-Testing:•Test Frameworks •Test Documents•User support
•Quality Assurance:•QA activities•Bug tracker statistics•Test coverage
•LCG App.Area policies
•Summary
• Software testing activities• a complete and coherent automatic testing
infrastructure was deployed based on existing tools in the open source community. It covers covers from unit to validation tests.
• a unique entry point (QMTest) can be use to run all the tests at the release time, in the nightly building system, for the QA reports or by the developer.