20
EMI INFSO-RI- 261611 EMI INFSO-RI- 261611 SA2 - Quality Assurance Alberto Aimar (CERN) SA2 Leader 1 st EMI Periodic Review Brussels, 22 June 2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
SA2 - Quality Assurance
Alberto Aimar (CERN)SA2 Leader
1st EMI Periodic ReviewBrussels, 22 June 2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Outline
• Objectives• Achievements
• KPIs • Lessons Learned• Outlook
1st EMI Periodic Review22/06/2011 2
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
SA2 in EMI
3
Quality Assuranc
e
ProcessPolicies MetricsReviews
Tools InfrastructureSupport
SA2
1st EMI Periodic Review
NA1, JRA1
JRA1
JRA1SA1
SA1
SA2
Software & Services
Requirements
Defines
ImplementsCertifies
Release Candidate
Process definition
Process monitoring
NA1, NA2
22/06/2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
SA2 QA Objectives
Processes and Metrics• Establish common software quality assurance process and metrics Tools and Testbeds• Provide tools and resources for building and testing• Support a continuous integration and testing infrastructure Monitoring and Review• Monitor metrics trends, reviewing and improving software qualityCustomers QA Criteria• Allow passing customer acceptance criteriaUser Support on QA• Support and consultancy within the project 4
Define and establish a common software quality assurance process and metrics for all engineering activities.Enable a continuous integration and testing process by selecting and maintaining tools and resources for building and testing software either within the project of in collaboration with external resource providersAllow the EMI middleware to consistently pass the customer acceptance criteria and continually improve the software quality and the process itself by monitoring the metrics value trends, reviewing quality control activities and related tests, providing support and consultancy in QA matters. [DoW]
Define and establish a common software quality assurance process and metrics for all engineering activities.Enable a continuous integration and testing process by selecting and maintaining tools and resources for building and testing software either within the project of in collaboration with external resource providersAllow the EMI middleware to consistently pass the customer acceptance criteria and continually improve the software quality and the process itself by monitoring the metrics value trends, reviewing quality control activities and related tests, providing support and consultancy in QA matters. [DoW]
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
SA2 QA Strategy
EMI is merging 4 established Middleware projects, each with its own QA practices
• Work with a 3-years vision Y1: explain, define and implement Y2: review and automateY3: consolidate and optimize
• Use established QA standards ISO /IEC 9126-1-4 and 25000
• Benefit from existing QA practices Use existing QA tools, resources and expertise
1st EMI Periodic Review 522/06/2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611SA2 Management Strategy
Strategy applied for all SA2 activities and tasks
• Involve PTs, SA1, JRA1 - Merging 4 products, tools, platforms, etc is a challenge, we involved all teams.
• Provide early solutions - We wanted to provide early policies, tools, repositories, etc because teams were waiting for them
• Explain QA constraints - Make aware all developers and teams that QA will requires changes for converging on common solutions
• Be transparent – Work closely with other activities (SA1, JRA1, EMI EMT Product teams, EGI QA, PO), all SA2 material public
• Aim at automated solutions – They provide exponential benefits compared to any specific or manual solutions
• Support tools and testbed – Not just policies and rules to follow, but also give useful infrastructure and services to the developers
1st EMI Periodic Review 622/06/2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
SA2 Information
1st EMI Periodic Review 7
https://cern.ch/twiki/bin/view/EMI/SA2 (bit.ly/emisa2)
22/06/2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Objective: Common QA Process
Software Quality Assurance Plan defined early • SQA factors, SQA management• Roles and procedures• Standard Practices and conventions
Detailed QA Policies• Release and Change Management• Configuration and Integration• Packaging and Distribution• Testing and Certification• Documentation
8
SA2
Define and establish a common software quality assurance process and metrics for all engineering activities [DoW] SO 1.4Define and establish a common software quality assurance process and metrics for all engineering activities [DoW] SO 1.4
22/06/2011 1st EMI Periodic Review
EMI QA policies are used by all Product Teams to achieve a uniform process and presentation of EMI software, documentation, etc
EMI QA policies are used by all Product Teams to achieve a uniform process and presentation of EMI software, documentation, etc
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Some QA Policies
22/06/2011 1st EMI Periodic Review 9
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Objective: Common QA Metrics
Selected Metrics to objectively qualify status of software • Automated numerical metrics • Cover different requirements of EMT, SA1, PT, SA2 • Provide reports, trends and dashboards
Generate a wide range of metrics to cover several QA aspects• Metrics on code, process, documentation, support • Internal and external metrics (code and process)• Language dependent (Java, C++, Python), object-oriented
Examples of metrics • Reaction time to critical bugs, delays in releases• SLOC, code complexity, bug density, test coverage
22/06/2011 1st EMI Periodic Review 10
Continually improve the software quality and the process itself by monitoring the metrics value trends [DoW] SO 1.4Continually improve the software quality and the process itself by monitoring the metrics value trends [DoW] SO 1.4
Automated tools to collect the same QA metrics across all EMI products. QA data generated to be analysed & key metrics selected
Automated tools to collect the same QA metrics across all EMI products. QA data generated to be analysed & key metrics selected
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
11
Objective: Provide EMI QA Tools
Initial situation: All different tools, dev environments, trackers, etc
Final Goals: Move towards a single tool set, workflow and data
Approach Followed• Define common software engineering workflow - Uniform build,
testing, packaging trying not to affect the developers’ efficiency• Provide build and test infrastructure - Provide a comprehensive
infrastructure able to support the policies (build, packaging, releasing, etc). This infrastructure is ETICS.
• Introduce progressively testing, QA metrics – provide automated test execution, metrics generation and reporting
• Focus on Support - Support integrated in GGUS, 50% of users were new to the tools
22/06/2011
Enable a continuous integration and testing process selecting and maintaining tools and resources for building and testing software [DoW] SO 1.5Enable a continuous integration and testing process selecting and maintaining tools and resources for building and testing software [DoW] SO 1.5
1st EMI Periodic Review
All EMI software is built, packaged and distributed with the same tools and is all following same policiesAll EMI software is built, packaged and distributed
with the same tools and is all following same policies
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Middleware Languages ReleaseIntegration
DependencyManagement
Packaging Testing Trackers
Tools - Initial Situation
22/06/2011 1st EMI Periodic Review 12
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Middleware Languages ReleaseIntegration
DependencyManagement
Packaging Testing Trackers
Tools - Current Status
22/06/2011 1st EMI Periodic Review 13
XML Exchange Format
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611Objective: Common QA Testbed
Created the EMI Testbed for integration and testing process• Single Testing Infrastructure within Quality Assurance • Product Team – centric model for their individual certification
Support Internal Inter-Component Testing• Focus on interaction among different clients and components provided
by other Product Teams and for release candidates verifications
Defined Large Scale Testing infrastructure• Scalability and interoperability testing on real environment• Involved external customers, resources and their verifications (EGI QA)
Installed all EMI Services both production and release candidatesAgreed participation of EGI sites to large scale testing
14
Enable a continuous integration and testing process either within the project or in collaboration with external resource providers [DoW] SO 1.4Enable a continuous integration and testing process either within the project or in collaboration with external resource providers [DoW] SO 1.4
22/06/2011 1st EMI Periodic Review
All EMI Software (~90 installations) production and release candidates on EMI testbed (7 sites) for testing & certification
All EMI Software (~90 installations) production and release candidates on EMI testbed (7 sites) for testing & certification
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
SA2 – Year 1
22/06/2011 1st EMI Periodic Review 15
Quality Assuranc
e
ProcessPolicies MetricsReviews
Tools InfrastructureSupport
SA2NA1, JRA1
JRA1
JRA1SA1
SA1
SA2
Software & Services
Requirements
Defines
ImplementsCertifies
Release Candidate
Process definition
Process monitoring
In Y1 SA2 established the foundations of a measurable QA process based on common policies, tools and infrastructureIn Y1 SA2 established the foundations of a measurable QA process based on common policies, tools and infrastructure
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Outline
• Objectives• Achievements
• KPIs • Lessons Learned• Outlook
1st EMI Periodic Review22/06/2011 16
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
SA2 KPIs
1st EMI Periodic Review 17
KPI Description Target Q1 Q2 Q3 Q4 P1
KSA2.2 - Services AvailabilityETICS 97% - 99.7 % 95.6 % 98.6 % 98.0 %
Testbed 97% - 97.5 % 98.7 % 98.8 % 98.3 %KSA2.3 - Testbed Size
50 CPUs - 70 60 90 73KSA2.6 - No of Requests
ETICS - - 39 27 75 47Testbed - - - 8 22 15
KSA2.7 - Average Response TimeETICS - - 8.2 h 5.2 h 5.9 h 6.4 h
Testbed - - - 6.0 h 4.3 h 5.2 hKSA2.8 - Average Solution Time
ETICS - - 17.4 h 10.8 h 37.6 h 21.9 hTestbed - - - 11.2 h 29.5 h 20.3 h
22/06/2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
Lessons Learned
• Merging QA for four products all with different processes, tools is really complicated, technically and humanly
• Strong support of all partners was crucial for adoption • Combine top-down QA constraints & bottom-up needs• Involve teams experts and whoever want to contribute.
Constantly explain the goods of shared common solutions • Start with simple metrics, same for all, explain and discuss
details. Select metrics suitable for different audiences• Provide early tools and testbed infrastructure with useful
services in order to support the policies and the process• Periodically review and adapt keeping in mind the overall
QA goals of the project and developers needs
1st EMI Periodic Review 1822/06/2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
SA2 in EMI Year 2
Main QA Focus for Year 2 (in addition to Y1 activities)
• Review and adapt QA policies, tools, testbed after Year 1• Select Key Metrics and work with PTs to improve on them• Distribute standard QA reports automatically generated• Provide SL6 and Debian 6 new platforms on ETICS, Testbed• Improve on Unit Testing PTs should add tests in ETICS • Add Tests on testbed PTs provide automated tests• Progress with Large Scale Testbed with volunteer EGI Sites
Management Approach for Year 2• Follow the QA strategy with same approach used in Y1
Worked very well. Y2: review and automate. Involve PTs in QA matters in order to have their support
1st EMI Periodic Review 1922/06/2011
EMI I
NFS
O-R
I-261
611
EMI I
NFS
O-R
I-261
611
1st EMI Periodic Review 20
Thank you
EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611
22/06/2011
