A. Aimar - EP/SFT LCG - Software Process & Infrastructure1 SPI Infrastructure for LCG Software...

A. Aimar - EP/SFT LCG - Software Process & Infrastructure 1

SPI Infrastructure

for LCG Software Projects

Overview

A.Aimar EP/SFTCERN

LCG Software Process & Infrastructure


Project context of LCG SPI

LCG Application Area

LCG Infrastructure•Common services

•Similar ways of working (process)

•Tools, templates, training

•General QA, tests, integration, release

LCG Application Area software projects

•POOL: Persistency

•SEAL: Core common software

•PI: Physics Interfaces

•Simulation

•Detector Description

• …etc…

LHC grid software applications

(LHC experiments, projects, etc)

LCG SPI project


Project context of the LCG SPI

• “Software Management Process RTAG”• General recommendations

- All LCG projects must adopt the same set of tools, standards and procedures

- Adopt commonly used open-source or commercial software where available

- Avoid “do it yourself solutions”- Avoid commercial software that may give licensing

problems

• If each project needs an infrastructure, many projects need it even more…

- Tools, standards and procedures- Try to avoid complexity


Infrastructure Software Development

Software development Support

• Keep it “simple and modular” approacha. Provide general services needed by each project

- CVS repository, Web Site, Software Library- Mailing Lists, Bug Reports, Collaborative Facilities

b. Provide solutions specific to the software phases- Tools, Templates, Training, Examples, etc.

CodingAnalysis and Design

Development

Release

SpecificationsTesting

…..Deployment and

Installation…..

Planning


Project guidelines

• No specific software development• The project should avoid

any new development, and future maintenance, of any specific software package needed for the LCG software infrastructure.

• Use HEP or free software• All software used should be

already available, or become available, in the HEP community or in the free software community

• The goal of the project is to define an infrastructure and later a simple process• Future maintenance will need

separate planning and resources.

• Everything is done in collaboration• LCG and LCG projects (Pool)• IT division and LHC

experiments• Big projects (G4, Root, etc)

• Using as the existing IT services• The project should use all

existing IT services• Make sure they match the

LCG needs


http://spi.cern.ch



OUTLINE: SPI Services

• Services specific to LCG projects• General overview• Very useful to LCG project but not really to

other projects

• Services for LCG projects but available• External software• Testing framework

• General service for LCG and others• Project Portal


>>> CVS repository and Delivery Areas

CVS repository• A central CVS repository

managed and available to all projects

• Any project just needs to ask for it, and declare its users permissions

• Managing mirroring and backups

• Users access controlled• Tools for automatic clean

up of locks, etc• Systems to browse the

repositories

Delivery areas• The AFS and DFS

delivery areas 1. an area to install software

created by projects in the LCG application area

2. an area where external and third party software

3. an area where software is installed which is provided by people for evaluation within a project

• Pre-installed software shall be in directories specifying the package name, the package version and the “OS_Compiler”


:pserver:<anoncvs>@lcgapp.cern.ch:/cvs/…

\\cern.ch\dfs\projects\lcg

/afs/cern.ch/sw/lcg


>>> Coding guidelines

Rule Checker

(on the way)

Human Reviews from SPI


>>> CVS organization


>>>Scram

• Used by Pool and Seal• Improvements

• restructuring the build system to improve performance• possible to use automake/autoconf within a scram project

• Porting to Windows • Execution on perl on Windows and usage of nmake

(standard Windows installation without cygwin)• Import projects into Visual Studio

• Organization• a single SPI-managed toolbox service for LCG projects

• Organizing a tutorial and training for LCG developers and users


>>> LCG Workbook

http://lcgapp.cern.ch/project/workbook/


Other components available

• Memory Leaks

• Automatic Builds (on the way)

• Software distribution (investigating “pacman”)

• Software documentation, templates, etc


>>> Code Documentation

• Features of interest:• Code browsing• Code searching• Code information• Design diagrams

• Any project will have them available as part of the infrastructure• Doxygen • LXR

ViewCVS


Code documentation: Doxygen


Code documentation : LXR


Code documentation: ViewCVS

• CVSgraph: Displays the tree of revisions and branches graphically

• Enscript: Colorize files in the CVS repository.


>>> External Software Service


Service: Software Library

• All software needed by more than one LCG projects shall be installed in common areas

• Available for several compilers and operating systems

• Maintained centrally with support and help to the user

• External documentation available• Contact with the authors if the software is

developed in HEP or LCG• Distribution and pre-installation to all LCG

and HEP users


/afs/cern.ch/sw/lcg/external


Current status

• Central installation of HEP or open source software needed by LCG related projects or following their requests in term of version and platforms (linux, MSWindows and Solaris when needed).

• Standard installation & location• Availability

• AFS: /afs/cern.ch/sw/lcg/external/• DFS: \\cern.ch\dfs\project\lcg\external\ (just started)

sundev005 /afs/cern.ch/user/l/lcgspi [87]> ls /afs/cern.ch/sw/lcg/external/Ant/ JACE/ Python/ edg-rls-client/ png-lib/ valgrind/Boost/ JDK/ TJDO/ expat/ root/ wxPython/CMake/ JDORI/ XalanJ/ gccxml/ scripts/Colt/ Log4J/ XercesC/ mysql/ tarFiles/CppUnit/ MMMySQL/ XercesJ/ mysql++/ test.html*GSL/ Oval/ doxygen/ packages/ uuid/






>>> Software TestingTests of classes

• individual class• small groups of collaborating classes• work package

Software developer

CODE

Testprograms

Automatic software test

run processCode Documentation

Tools•CppUnit

•PyUnit

•Oval



Test frameworks: CppUnit

A simple test1. Subclass the TestCase CppUnit class2. Override the method runTest().3. call CPPUNIT_ASSERT

Similar tools: Junit, PerlUnit, PyUnit, QtUnit

Output in XML, compiler or text

Windows version for MVC++6.0


Test frameworks: CppUnit


Test frameworks: Oval

Ovalfile(configuration)

ExecutableTest

Source Code

Referencefile

Log file

Oval run

Ovaldiff

Oval build

Can be use for Unit-test

It is possible to set different run environments

Can run external scripts and external binaries

Authors modified it to run it with or with any environment

Oval:• validation• regression


Test frameworks: Oval


Test: structure and documentation

SPI-TESTING component

SPI-CVS_structurecomponent

• CVS testing structure•Test-naming

Will help to have a automatic way to runtest code

Test documentation•Test cases specification template

src doc test

testA testBOvalfile Nametest.ccpNametest.refNametest.log[Nametest.in][Mametest.out]


Testing

• Available CppUnit, PyUnit and Oval• Can also be used stand alone, integrated

with Scram and with CMT

• Can also be used by projects NOT using Scram


>>> Project Portal

Bug Tracker• A central installation so that

each project has an available bug tracker system

• Centrally managed but customizable for different needs

• Easier to post bug if projects use similar systems

• No work of maintenance and upgrade on the projects

• Web based

http://lcgappdev.cern.ch(for a few test projects)

Project Portal• The development front-

end of every project• With all features needed,

configured and integrated• News and Faq• Task management• Mailing lists access• Bug tracking access• Download area• Access to the CVS

repository• We have customized

“Savannah” from GNU



Benefits

• Totally web based

• Single entry point to project(s)

• Uniform access to project information

• Set up common web infrastructure for a project without coding



Bug tracker

• Fields customizable for each project

• File uploads

• Audit trail

• Full text searches

• Cc forwarding



News system

• Messages can be displayed in threads

• Monitoring (automatic sending of email)

• Link detection• Transforms URLs to links

• References to bugs/tasks get linked



Task manager

• Sub-project/task hierarchy

• Tasks can be assigned to developers

• Tasks carry a deadline, etc.

• Tasks/bugs interdependencies



Personalize web page

• A logged in user gets• a customizable personal entry page

• display of current work status- User's bugs, tasks, ...

- Monitored news threads

- Bookmarks




And much more…

• Consolidate project links• ViewCVS (, Doxygen, LXR, Bonzai, ...)

• Project homepage

• Mailing lists (subscribe/unsubscribe)

• FAQ system (user/admin documents)

• Support manager

• Download area

• Patch manager

• Projects administrators can choose what to use for each project


Admin: Project types

• Name and description

• Preferences common to one type of projects• Homepage, CVS information

• List of status descriptions (e.g. alpha, beta, stable)

• Download/upload area (faq) information

• Allow use of other subsystems (patch manager, task manager, news, support request manager, bug tracker, FAQ)


Admin: For each project

• Public information- Name, description, development status

- Enable subsystems (bug tracker, patch manager, task manager, CVS link, news system, support tracker, FAQ system, download/upload area)

• Members• Task manager, patch manager, support manager:

none, tech, admin tech&admin

• Project administrator



Admin: Bug tracking configuration

• Bugs- Fields, values

- Email notification

- Global: List of 'external' email addresses who get all new bug submissions

- Personal (techs+admins): List of 'Users to watch'

- Personal(techs+admins): Detailed list of events that trigger email to you

• News• Delete postings




Specific implementations

• General bug fixing and improvements• Implemented bulk user registration

• Get a list of users and add/remove them all without registering them one by one

• Integration with AFS authentication• Added a PAM module to verify a login against the AFS

authentication• The user can still define a password not using the AFS

password

• Sending back to Savannah our changes, patches and fixes

• Run a stable service • http://lcgappdev.cern.ch for now


Proposed release schedule

• December 2002 - Beta Versions • Many components will be available before

• February 2003 - Version 1.0• All components and services mentioned above except

Software Library (but basic), Project portal (but basic)• Monitoring if resources will need to be injected

• May 2003 - LCG Test - Version 1.1• Software Library, Project portal and all features• Standard templates and tools available

• September 2003 - Version 1.2 • LCG software process• Standards for user specifications• Design guidelines and specifications

• end of 2003 until 2005 • Development Version 2, support and maintenance


Conclusion

• Homogeneity is important for the LCG and for its projects

• All work is done seeing the experts and work in LHC experiments and LCG projects• We all need to compromise and NOT focus on

infrastructure and individual taste

• The reference URL is http://spi.cern.ch

the others in this presentation may change.


http://spi.cern.ch

Date post:	14-Jan-2016
Category:	Documents
Upload:	christiana-goodman
View:	223 times
Download:	0 times

A. Aimar - EP/SFT LCG - Software Process & Infrastructure1 SPI Infrastructure for LCG Software...

Documents