Date post: | 05-Dec-2014 |
Category: |
Technology |
Upload: | jean-louis-letouzey |
View: | 3,696 times |
Download: | 0 times |
Source Code Quality Evaluation:The SQALE method
December 2011
Author: Jean-Louis Letouzey
SQALE: Software Quality Assessment based on Lifecycle Expectations
Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
2
The SQALE Method: Summary
Has been developed by experts, independent of any tool vendor
Focus on the diagnostic objectivity (precision, no false-positive)
Easy to understand, to implement and to deploy
Avoids practices that damage measurement results
Aggregation with averages that generates compensation effects
Notation on a delimited interval that generates threshold effect
Promotes simple principles
Source code quality is a non functional requirement that should be specified, then verified
Evaluating quality is measuring the remaining needed workload to fix all non-conformities
Is a robust method for identifying and managing the Technical Debt
Software Quality Assessment based on Lifecycle Expectations
5
inspearit and source code evaluation
inspearit is not a tool vendor, inspearit is an independantcompany
inspearit thinks that the method come first, then the toolsto support it
inspearit assist large accounts to implement source code analysis with SQALE
Our customers reported us issues with current methods and tools
Difficulties to understand the meaning and usage of indicators
Too much false positive
No support for remediation decision (what are the priorities?)
6
The needs for a source code evaluation method
Ability to objectively evaluate and monitor software development products in order to anticipate issues
Aligned with best measurement practices
Ability to compare
Source code versions
Different products with different usage/history
Development teams or subcontractors performance
Capacity to provide useful inputs to an improvement plan
Capacity to support decisions: Ex. Two teams working on two similar projects
The first one, delivered 3 weeks in advance but with 100 coding practice issues
The second, delivered 1 week in advance but with 15 coding practice issues
Which project is the most efficient and effective?
7
High level requirements for an evaluation method
The SQALE method has been developped as a solution to all these requirements
Quantified, Objective, Precise, Sensitive
Implementable by automated static analysis tools
Reproducible by the implementation of one tool to another (produce the same findings based on non ambigous definitions, rules…)
Provide guidance for tailoring this standard model to:
Any language
Different severity levels (business critical, life critical etc.)
Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
8
9
A Hierarchy of Qualitychar. and sub-char.
= Quality Model
QH
Q
Measure/rule
Sub-characteristic
CharacteristicMaintainability
A Hierarchyof Artifacts
A
Component
AppliA
AH
Summary of the challenge for an evaluation method
Challenge / Need:Provide a quality related “measure” or “score” for each couple {A,Q} of the 2 hierarchies
Two hierarchies:
File Get a measure, a score which characterizes, represe nts the evaluated concept
Portfolio
Domain
Quality
10
Measurement basics: The representation condition
“The condition that, if one software entity is less than another entity in terms of a selected attribute, then any software measure for that attribute must associate a smaller number to the first
entity than it does to the second entity” [1]
Real World Mathematical World
Aa
Ab
Mes(Aa)
Mes(Ab)
Measurement function
Impact on :- Measure/rule choice- Normalization functions- Aggregation rules
Ac
Attribute
Mes(Ac)Art
ifact
s
Mes(Aa) > Mes(Ab) Mes(Ab) > Mes(Ac )
A representative measure keep the
relationship established in the
real world Aa >> Ab >> Ac
[1] N.E. Fenton and S. L. Pfleeger, Software Metric s: A rigourous & Practical Approach, second edition, ISBN 053495425-1,PWS Publishing Comp any, Boston, 1997
11
The representation condition applied to aggregation
Issues:
The aggregation should represent the basic findings
Issues should be reported up to the highest level of the hierarchy
Aggregationrule
?
ImprovementVersion a Version b
AgScore_a AgScore_b The aggregate score should report the improvement
Version b >> Version aAgScore_b > AgScore_a
We have identified some effects that violate this c ondition
12
The masking effect
The masking effect appears when the aggregate value is not sensitive to the variation of one of the base values
Example:
File 1
File 2
File 3
File 4
File 5
Min Max Media
n
MyAppli Va A A C E E A E C
MyAppli Vb A A C D E A E C
An improvement from Va to Vb on file 4 does not impact the aggregate score
���� Aggregations by Min, Max and Median violate the rep resentation condition
13
The compensation effect
The compensation effect appears on aggregation functions such as: mean, weighted mean, median
Example:
Comment ratio (target for the project : >30% per file)
File 1
File
2File
3File
4File
5
File 6
File 7
File 8
File 9
File 10
MyAppli Va 5% 34% 48% 47% 31% 37% 33% 35% 4% 39%
Avera
ge
31%
While 2 files do have “maintainability” issue, the average is OK
In real life, lack of comment in files 1 & 9 won’t be compensated by abundance of comments in file 3 & 4
That ‘s one reason why most Quality Dashboards are not precise
���� Aggregations by average (weighted or not) violate t he representation condition
14
The type of scale and allowed aggregations
The measurement theory is precise about allowed aggregation [1]
Scale Valid Transformation Main Valid AgregationNominal 1 to 1 mapping NoneOrdinal Monotonic increasing
functionMin, Max, Median
Interval M' = aM + B (a>0) Min, Max, Median, Average
Ratio M' = aM (a>0) Min, Max, Median, Mean, Average, Sum, Distance (Euclidian or other)
Absolute M' = M All
Due to the representation condition, some combinati ons (scale type,aggregation) should be rejected
[1] N.E. Fenton and S. L. Pfleeger, Software Metric s: A rigourous & Practical Approach, second edition, ISBN 053495425-1,PWS Publishing Comp any, Boston, 1997
More C
hoice
15
Synthesis of our analysis
Synthesis of allowed operations and aggregation issues depending on scale type
Within SQALE, we choose to normalize all measures o n a ratio scale andto aggregate the normalized values by summation
Scale Min, Max, medianAverage, Weighted
averageSum, Distance
NominalNot allowed Not allowed Not allowed
OrdinalPotentially not representative
Not allowed Not allowed
IntervalPotentially not representative
Potentially not representative
Not allowed
RatioPotentially not representative
Potentially not representative
Representative
AbsolutePotentially not representative
Potentially not representative
Representative
Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
16
17
The SQALE method: Structure
Implementation/Tools
8 Fundamental Principles
4 concepts
Tailoring
Measurement theory and representativity
18
The SQALE 8 Fundamental Principles
1. The quality of the source code is a non-functional requirement
2. The requirements in relation to the quality of the source code have to be formalised according to the same quality criteria such as any other functional requirement
3. Assessing the quality of a source code is in essence assessing the distance between its state and its expected quality objective
4. The SQALE Method assesses the distance to the conformity with the requirements by considering the necessary remediation cost of bringing the source code to conformity
5. The SQALE Method respects the representation condition
6. The SQALE Method uses addition for aggregating the remediation costs and for calculating its quality indicators
7. The SQALE Method’s Quality Model is orthogonal
8. The SQALE Method’s Quality Model takes the software’s lifecycle into account
19
The 4 main concepts of the SQALE method
Source Code
Static analysis tools
FindingsTable
Remediationcosts table Aggregation
rules
SQI
STISRI…
SQID…
1 Quality Model 2 Analysis Model Indicators43 Indices
List of source code related requirements
Rem
ediationfunctions
Testabilité Fiabilité Evolutivité Efficacité Maintenabi lité
Maintenabilité 589
Efficacité 248 248
Evolutivité 1 480 1 480 1 480
Fiabilité 548 548 548 548
Testabilité 6 535 6 535 6 535 6 535 6 535
6 535 7 083 8 563 8 811 9 400
Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
20
21
Back to the fundamentals of Quality
You buy a new car
• How will you feel, if the delivered car has only 5 cylinders and 290 hp?
22
Back to the fundamentals of Quality
In 1979, Philip Crosby in his famous book “Quality is free” established the 4 principles of Quality:
the definition of quality is conformance to requirements
the system of quality is prevention
the performance standard is zero defects
the measurement of quality is the price of nonconformance
Since that time, the vision and definition of quality has been extended to a much wider scope including customer satisfaction
But anyway, quality is still at least “Conformance to requirements”
23
The SQALE Quality Model: source code requirements
An organized set of expectations (requirements)based on lifecycle needs
Reuse
Maintain
Deliver
Evolve
Test
Code
Testability
Reliability
Changeability
Efficiency
Maintainability
Reusability
Security
Portability
Architecture related reliabilityArchitecture related reliability
Fault toleranceFault tolerance
Logic related reliabilityLogic related reliability
Instruction related reliabilityInstruction related reliability
Data related reliabilityData related reliability
UnderstandabilityUnderstandability
ReadabilityReadability
Ram related efficiencyRam related efficiency
Rom related efficiencyRom related efficiency
CPU related efficiency CPU related efficiency
Archi. related changeabilityArchi. related changeability
Logic related changeabilityLogic related changeability
Data related changeabilityData related changeability
Unit Testing testabilityUnit Testing testability
Integration Testing testabilityIntegration Testing testability
TestabilityTestability
ReliabilityReliability
ChangeabilityChangeability
EfficiencyEfficiency
SecuritySecurity
MaintainabilityMaintainability
PortabilityPortability
ReusabilityReusability
Number of derived class <=10no public data
…
…
…
Characteristic
Sub-characteristic
Requirement
Requirements , appear only once within the Quality Model, when they are first needed. They are checked with relevant static analysis tools
Requirements: type of issues
It is important to use a SQALE Quality Model that covers all the types of code issues
24
Copy and Paste, internal structure of methodsCopy and Paste, internal structure of methods
Potential logic errors, exception management,
test coverage
Potential logic errors, exception management,
test coverage
Excessive coupling, Hard coded dataExcessive coupling, Hard coded data
Useless code,un-optimized codeUseless code,un-optimized code
Presentation, structurnessPresentation, structurness
When deployed, the SQALE Quality Model contains from 30 to 100+ requirements tailored to the organization context
TestabilityTestability
ReliabilityReliability
ChangeabilityChangeability
EfficiencyEfficiency
MaintainabilityMaintainability
Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
25
26
Rem
ediationfunction
Remediation cost table
The SQALE Analysis Model
Quality Indexes represent the remediation effort needed to refactor artifacts in order to comply with the Quality Model
Tool
Analysis
4.1“Understandability index” for the selected file
“Characteristic indexes” are aggregated by adding “ Subcharacteristic indexes”
Non conformity table
Part level indexesPart level indexes are aggregated by adding all fil e indexes
Σ
Sub characteristic
indexes4.1
Σ
27
Req_1 Req_2 Req_3 Req_4 Req_5 Req_6 … … … … Req_99File_1File_2
File_3File_4……………File_99999
Remediation costs table
The SQALE Analysis Model
For a given couple {A,Q}, SQALE provides a simple rule to calculate the associated score
A
Component
AppliA
File
Portfolio
Domain
Q
Measure/rule
Sub-characteristic
CharacteristicMaintainability
Quality
The positions into the 2 hierarchies define the per imeter of remediation costs to be added
Σ
28
The SQALE Analysis Model: remediation factors
How findings are transformed into costs?
• SQALE use « Remediation Functions » that are associated to types of Non Conformity. The standard SQALE Analysis Model contains 5 types which correspond to different « remediation lifecycles »
• These Types and values are proposed by default. I it is recommended to extend /taylor them at Organization/Project/Application level
Estimated cost for fixing one Non Conformity of Typ e4 is: 5 Work Units
Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
29
30
3° concept: SQALE indices
The SQALE characteristic indices:
- SQALE Testability Index: STI- SQALE Reliability Index: SRI- SQALE Changeability Index: SCI- SQALE Security Index: SSI- SQALE Efficiency Index: SEI- SQALE Maintainability Index: SMI- SQALE Portability Index: SPI- SQALE Reusability Index: SRuI
The global SQALE Quality Index: SQI
- This is the Technical Debt
SQALE index densities: SQID, STID etc
Artifact remediation workload for all requirements associated to testability
Example: Density by KLOC(1,000 lines of code)
31
4° concept: The 3 SQALE indicators
SQALE Indices
SQI
STISRI…
SQID…
Rating Grid
SQALE Kiviat
Testabilité Fiabilité Evolutivité Efficacité Maintenabi lité
Maintenabilité 589
Efficacité 248 248
Evolutivité 1 480 1 480 1 480
Fiabilité 548 548 548 548
Testabilité 6 535 6 535 6 535 6 535 6 535
6 535 7 083 8 563 8 811 9 400
Based on the ratio (in %)Remediation cost / Development cost
Example of Testability ratingRemediation cost (STI): 4.36 hoursDevelopment cost: 250 hoursRatio: 1.7 % ���� Rating: “C”
32
TestabilityTestability
ReliabilityReliability
ChangeabilityChangeability
EfficiencyEfficiency
MaintainabilityMaintainability
The SQALE Pyramid: A two points of view indicator
An analytic view provided by orthogonal characteristicsOne understands impact of each Non Conformity and improvement on quality characteristic and life cycle issues.
© Copyright inspearit 2007-20011
Σ
Σ
Σ
An external view that represents the percieved quality evaluated by consolidation of the hierarchy of characteristics
Σ
Interpreting the SQALE indices
33
The perceived benefits
As SQALE requires to specify the quality of the code, the quality measure is objective
The SQALE quality index represents a workload, a cost. It is the concrete “Technical debt” of the project
It is easy to monitor simultaneously:
the remaining workload associated to functionalities
the debt associated to code quality
and update accordingly the project‘s planning
Technical debt may be aggregated at any portfolio granularity
Technical debt density allows to compare versions, applications, subcontractors…
As SQALE does not violate the representation condit ion, SQALE thus provides usable meanings to source code measurement s
Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
34
35
Using SQALE: Tools
PRIVATE, METRIXWARE:
SQALE index distribution analysis
36
Using SQALE: Tools
SQUORE: Dashboard
37
Using SQALE: Tools
SONAR: Monitoring the source code quality(Technical Debt)
38
Using SQALE: Tools
SONAR: Tailored Dashboard
39
Using SQALE: Tools
SONAR: Portfolio Management
40
Using SQALE: Deployment
0 Initialization 1 Tailoring Deployment32 Implementation
Planification Stake holders Perimeter …
Method trainingDevelopment of Tailored SQALE models Specify the tool solution
Choose and Implementation ofthe solution
Tool Deployment, training coaching Monitor and improve
Identify the sponsor and stake holders, define the roadmap
Define the most usefull use cases of source code analysis:
Build the « Source code analysis » Project team
Perform awarenesssession
Coach and support the users
Monitor the solution and define an improvementplan (identification and implementation of new requirements)
Choose the solution to be implemented withinthe organization
Implementation of the tailored models withinthe selected solution
Implementation of the selected indicators and reports within the selected solution
Update process, associated deliverablesand training to preparedeployment
On day training for the team: The SQALE Method
Develop a tailoredSQALE Quality Model
Develop a tailoredSQALE Analysis Model
Validation of bothmodels trough a pilot project
Specify the toolsolution includingrecommandedindicators
Training,Workshop preparation and animation
Support SeminarsTrainingCoaching Support
Support
41
The SQALE Discovery Kit
Discover the fundamentals principles and benefits of the SQALE method with the“SQALE Discovery Kit”. This package contains:
A one day training session on the SQALE Method
The identification (through dedicated interviews) of your main “use cases” of source code analysis within your organisation’s context
The development (through dedicated Workshops) of your own quality and analysis models
These models will be tailored to your environment and will be the basis for defining and evaluating the quality of your source code (for one of the following language: Java, C, C++, Cobol)
The concrete assessment of one of your application using the SQALE method and your tailored quality and analysis models including a detailed assessment report
Workshop on how to interpret and use the results
At the end you will get:
Your tailored models for one of your development language
An evaluation report
Direction for actions
Total duration: about 20 days
Thanks
To know more about SQALE: http: /www.sqale.orgTo do more with SQALE, contact inspearit
Saturday, December 17, 2011
© inspearit - Author - Place
42
just sqale it