Date post: | 29-Dec-2015 |
Category: |
Documents |
Upload: | frank-fowler |
View: | 213 times |
Download: | 0 times |
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
EMI Quality Assurance Processes (PS06-4-499)
Alberto Aimar (CERN)CERN IT-GT-SL Section Leader
EMI SA2 QA Activity Leader
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• EMI context and QA objectives
• Guidelines and Metrics• Tools and Testbeds• Reports and Reviews
• Conclusions
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 2
Outline
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
EMI Mission Statement
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 3
The European Middleware Initiative (EMI) project represents a close collaboration of the major European middleware providers - ARC, gLite, UNICORE and dCache - to establish a sustainable model to support, harmonise and evolve the grid middleware for deployment in EGI, PRACE and other distributed e-Infrastructures
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
EMI Project Structure
NA1 - Administrative and Technical Management
NA2 – Outreach and Collaborations
SA1 - Maintenance and Support
JRA1 - Development, Integration and Evolution
4EMI QA Activities - A.Aimar (CERN)CHEP 2010, Taipei
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 5
The Product Teams
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Four large MW projects developing several products – Many development teams > 25 Product Teams
• Request from the grid infrastructures and the EU funding – Have more uniform releases and interoperable middleware distributions– Provide a common degree of verification & validation– Have metrics and objective criteria to compare product quality and
evolution over time
• No resources to have independent QA efforts– Dev teams focus on development (JRA1) and maintenance (SA1)– No time to set up report on metrics, install tools, maintain testbeds
• Provide homogeneous packaging, reporting and reviewing of the products – Have limited impact on the way development teams work– Let them still use their build systems, dev tools, bug trackers, etc
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 6
The Challenge
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Quality Assurance Process Definition and Monitoring – Standards-compliant software engineering process. – Continual activity of monitoring its application
• Metrics Definition and Reporting – Definition, collection and reporting of software quality metrics. – Reports information on the status of the software to take corrective actions
• QA Implementation Review and Support – Review activities of the QA, test and certification implementations. – Sample review of test plans, compliance, porting guidelines, documentation, etc– Supporting the Product Teams in implementation of tests and use of testing tools
• Tools and Repositories, Maintenance and Integration – Definition and maintenance of tools required to support QA process– Supporting activity to software providers to integrate external tools– Repositories for the EMI software packages, tests, build and reports
• Testbeds Setup, Maintenance and Coordination – Setup and maintenance of distributed testbeds for continuous integration testing– Coordination and provision of larger-scale testbeds from collaborating providers
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 7
EMI QA (SA2) Tasks and Objectives
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 8
Software Quality Plan
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Software Quality Assurance Plan (SQAP) – Document that specifies the tasks needed to define and measure quality,
responsibilities for quality monitoring, documentation required and procedures – Plan that will be followed to manage the QA process
• SQAP specifies the manner in which the EMI project is going to achieve its software quality goals. – Metrics and measurements are associated in order to evaluate the quality of
the EMI software and lifecycle
• SQA tasks, roles and responsibilities– EMI technical activities (SA1, SA2 and JRA1) – EMI technical bodies (PTB and EMT)– Even of specific individuals/roles in EMI
• The SQAP covers documentation, reporting and reviewing tasks– Describes the metrics that will be used for the QA reporting and reviews
• Will be updated regularly, based on experience and needs• It is complemented by a set of guidelines (on dev and doc)
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 9
Software Quality Plan
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• A set of very practical Guidelines for QA and Software Development is available:– Configuration and Integration – Packaging and Releasing – Change Management (bugs, patches, etc) – Certification and Testing – Metrics Generation
• Move towards a uniform setup and common definitions and conventions in the project– Releases, patches, versions– Packaging, repositories, distributions– User support, documentation
• Advantages for e-infrastructures are obvious but it requires some work and little changes by the dev teams in EMI– The guidelines were defined taking the best practices from the 4
middlewares and are mostly agreed by all dev teams
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 10
Development Guidelines
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 11
Development Guidelines
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Definition of the Minimum Required Documentation for each software component. – Installation Guide and Troubleshooting guide – User Documentation – Software Requirements Specifications – Software Design Description – Software Verification and Validation Plan and Report
• General Project-wide Required Documentation– Technical Development Plan– Software Release Plan and Schedule– Software Maintenance and Support Plan– QA Tools Documentation– Continuous Integration and Certification Test beds Documentation– Security Assessments
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 12
Documentation Guidelines
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Metrics are needed to quantify and qualify the status of the software components
• Use as much as possible numerical metrics – All automated and extracted in the same exact way , by the same tool(s)
• Starting with a selection of useful and simple metrics– Tools available give a common environment to extract metrics and test– Same metrics for all components, in order to have fair reports
• Types of SQA metrics– Metrics on code, process, support, documentation – Internal and external metrics (code and process)– Language dependent (Java, C++, Python)
• Examples of metrics– Reaction time to critical bugs, delays in releases– Complexity, bug density, test coverage
• We refer to QA standards, but use what is realistically applicable– ISO /IEC 9126-1,-2,-3,-4 and ISO/IEC 25000
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 13
Quality Metrics
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 14
Metrics
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Metric, Testing and Packaging Tools– Compliance and interoperability of the software products– Integration builds of all middleware components– Same identical platforms for all builds, use standard packages on platforms– Automatic deployment and distributed testing of software products
• Integration of data coming from different sources and tools– Common report of metrics from different static and dynamic software QA tools– Collection of data from several req and bug trackers used by dev teams – Using same tool for packaging and testing tools– Use of an exchange format for different sources
• Support for repositories and distribution– Common software repository for all EMI middleware– Use the standard RHEL/SL and Debian repositories and formats
• Generation of QA reports– Metrics extraction, storage and archival– Graphs and reports at all levels of detail – Comparison tables and plot trends
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 15
Tools
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 16
Tools
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• EMI SA2 provides a distributed testbed – Real hw resources (+ obviously using virtualization) – For integration testing in EMI project – For the product teams testing– Distributed over the sites of the middleware partners
• Testbed available to Dev teams for testing their software– The teams can easily test their components with what is in production
or will soon be (RC)– Production and RC available for all components, servers, etc– Product Team can configure the services needed for its specific tests
• Provide support for the typical scenarios– Integration testing within a minor release– Integration testing for a major release – Cross middlewares tests across the network– Large-scale tests, real usage scenarios
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 17
Testbeds
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 18
Testbeds
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• QA reports objectively describe the quality of the product– Only clear metrics are included, e.g. number of bugs/SLOC, reaction
time, trends over time, successful/failed releases, etc – Reports both on the product but also on how the team works– Not the goal of the QA activity, a mean to see strengths and weaknesses
• The same type report for all components– Allows comparisons among components – Trends over time of the same components
• Fully automated. Dev teams can have their report weekly– See the results and execute corrective actions
• SA2 reviews of the QA reports and supports the teams– Provides the general reviewing every Month and formally every Quarter – Reports to the EMI mgmt in case of issues– Supports the dev team that need help with metrics, tools, testbeds
CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 19
QA Reports and Reviews
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 20
QA Reports
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
• Dedicated QA is a useful activity in large projects, but guidelines, procedures should not overload the developers
• Challenge of implementing QA in a distributed and heterogeneous environment – Different kind of sw products, different tools, distributed teams, etc
• Collected experience from the developers in order to define realistic and shared solutions– Never seen a top down or theoretical software approach working...
... so we try a (slightly) different one.
• Define metrics and automate report generation • Provide also practical services, not just procedures
– to developers: supported and automated tools, testbeds, packaging– to e-infrastructures: verified and homogeneous sw, doc, repositories
https://twiki.cern.ch/twiki/bin/view/EMI/SA2
CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 21
Conclusions
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Thank you
CHEP 2010, Taipei 22EMI QA Activities - A.Aimar (CERN)
EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611