+
Presented by: Ali Idri
Evaluating Software Product Quality:
A systematic mapping study
Sofia Ouhbi, Ali Idri, José Luis Fernández-Alemán, Ambrosio Toval
IWSM MENSURA ConferenceOctober 6-8,2014, Rotterdam
Evaluating Software Product Quality: A systematic mapping study
2+Presentation Outline
1. Motivations and Objectives
2. Software Product Quality: Introduction
3. Research Methodology
4. Results
5. Principal findings
6. Implications
7. Conclusion and Future work
04/13/2023 Evaluating Software Product Quality: A systematic mapping study 3
+Motivations and Objectives
04/13/2023Evaluating Software Product Quality: A systematic mapping study
4+Motivations and Objectives
• Obtain the current approaches used to SPQE • No systematic mapping study of SPQE approaches has been
published
Motivations
• Summarize the existing SPQE approaches and whether or not these approaches are based on well-known SQ models
• Discover if SPQE is done through one or many SQ characteristics
• Identify the research types and empirical types that exist in SQPE literature
• List the data sets used in the evaluation of the SQPE approaches
Objectives
04/13/2023 Evaluating Software Product Quality: A systematic mapping study 5
+Software Product QualityIntroduction
Evaluating Software Product Quality: A systematic mapping study
6+Software Product Quality
“Although quality is pursued in order to satisfy users, it is important to note that the quality desired by users is not universal but rather subject to constant change”
SQuBOK, JUSE 2007
Software product quality could be defined as “the totality of characteristics of an entity that bear on it’s ability to satisfy stated and implied needs”
ISO/IEC 9126-1
Evaluating Software Product Quality: A systematic mapping study
7+Software Product Quality
Several SQ models and standards with which to improve Software Product Quality have been proposed such as:
These models have some common SQ characteristics, such as:
o McCallo Boehmo Dromeyo ISO/IEC 9126 standard, which has been
replaced with ISO/IEC 25010
o Efficiencyo Reliabilityo Portabilityo Maintainability
04/13/2023Evaluating Software Product Quality: A systematic mapping study
8+ISO/IEC 250nn (SQuaRE)
2501nQuality Model Division
2500nQuality Management
Division
2502nQuality Measurement
Division
2503n
Quality Requirement
Division
2504n
Quality Evaluation
Division
04/13/2023 Evaluating Software Product Quality: A systematic mapping study 9
+Research Methodology Systematic mapping study
04/13/2023Evaluating Software Product Quality: A systematic mapping study
10+Systematic mapping study
The systematic mapping study principal goal is to provide an overview of a research area, and identify the quantity and type of research and results available within it. This method focuses on classification, conducting a thematic analysis and identifying publication fora.
A mapping process consists of three activities:
1) the search for relevant publications,
2) the definition of a classification scheme,
3) the mapping of publications.
04/13/2023Evaluating Software Product Quality: A systematic mapping study
11+Systematic mapping process
Definition of Research Questions
Conduct Search
Screening of Papers
Key wording using
Abstracts
Data Extraction and Mapping
Process
Review Scope All papersRelevant Papers
Classification Scheme
Systematic Map
Process Steps
Outcomes
04/13/2023Evaluating Software Product Quality: A systematic mapping study
12+Systematic mapping questions
•Which publication channels are the main targets for SPQE research?
MQ1
•How has the frequency of SPQE research dissemination changed over time?
MQ2
•In which research types are SPQE papers classified?
MQ3
•Which approaches have been used for SPQE?
MQ4
•Are the SPQE selected studies empirically validated?
MQ5
•What are the datasets that were used in SPQE literature?
MQ6
•Which artifacts have been reported in SPQE selected studies?
MQ7
•What are the well-known SQ models that have been mentioned in SPQE literature?
MQ8
•Which characteristics were used to evaluate SPQ?
MQ9
04/13/2023Evaluating Software Product Quality: A systematic mapping study
13+Search string
“Software quality" AND (evaluat* OR measur* OR assess*) AND (technique* OR approach* OR method* OR model* OR framework* OR tool* OR data mining OR learning OR
artificial intelligence OR pattern recognition OR analogy OR case based reasoning OR nearest neighbo* OR decision tree* OR classification tree* OR neural net* OR genetic programming OR genetic algorithm* OR bayesian belief
network* OR bayesian net* OR association rule* OR support vector machine* OR regression OR fuzzy logic)
04/13/2023Evaluating Software Product Quality: A systematic mapping study
14+Study selection
IC1
•The studies that address evaluation, measurement or assessment of the quality of software product in overall or through SQ characteristics
EC1
•Papers that focus on system quality not on SPQ
EC2
•Papers whose subject was one or many SQ characteristics (e.g. maintainability, reliability) which were not used in the evaluation of SPQ
EC3
•Papers that were published before the nineties and after 2013
04/13/2023Evaluating Software Product Quality: A systematic mapping study
15+Study selection
04/13/2023Evaluating Software Product Quality: A systematic mapping study
16+Data extraction
Em
pir
ical ty
pe
Researc
h t
yp
e
Art
ifactSQ
PE A
pp
roach-
- Data mining technique- Process: A serie of actions, or functions leading to an SPQE result and performing operations on data. - Method: A regular and systematic means of accomplishing SPQE. - Tool-based technique: A technique based on a software tool to accomplish SPQE tasks. - Model: A system representation that allows SPQE to be investigated through a hierarchical structure. - Framework: A real or conceptual structure intended to serve as a support or guide for SPQE. - Other, e.g. guidelines.
- Evaluation research: Existing SPQE approaches are implemented in practice and an evaluation of them is conducted.
- Solution proposal: An SPQE solution is proposed. This solution may be a new SPQE approach or a significant extension of an existing approach.
- Other, e.g. experience paper, review.
- Case study: An empirical inquiry that investigates an SPQE approach within its real-life context. - Survey: A method for collecting quantitative information concerning an SPQE approach.- Experiment: An empirical method applied under controlled conditions, in order to evaluate a SPQE approach. - History-based evaluation: Studies evaluating SPQE approaches in previously completed software projects.- Other
- Documentation- Design module- Source code- Other
04/13/2023 Evaluating Software Product Quality: A systematic mapping study 17
+Results
04/13/2023Evaluating Software Product Quality: A systematic mapping study
18+MQ1. Publication channels
Publication Channel Total
Journal 33
Conference 12
Workshop 6
Symposium 3
Other 3
The Journal of Systems and Software is a recurrent publication source for SPQE techniques, followed by the Information and SoftwareTechnology Journal. • There is no obvious conference target that attracts publication concerning SPQE approaches
+
04/13/2023Evaluating Software Product Quality: A systematic mapping study
19
MQ2. Publication trend
+
04/13/2023Evaluating Software Product Quality: A systematic mapping study
20
MQ3,4,5. Research types, approaches and empirical types
04/13/2023Evaluating Software Product Quality: A systematic mapping study
21+MQ4. SPQE Data Mining Approaches
04/13/2023Evaluating Software Product Quality: A systematic mapping study
22+MQ6. Data Sets
04/13/2023Evaluating Software Product Quality: A systematic mapping study
23+MQ7. Artifacts
04/13/2023Evaluating Software Product Quality: A systematic mapping study
24+MQ8. SQ Models around 50% of SPQE papers do not cite any well-known SQ model
04/13/2023Evaluating Software Product Quality: A systematic mapping study
25+MQ9. SQ Characteristics
25% of SPQE papers did not mention any SQ characteristics
The main characteristic used in the evaluation of SPQ in literature is reliability including fault-proneness, followed by maintainability and functionality
23% of the selected studies had used all the ISO/IEC 9126 characteristics to evaluate SPQ
04/13/2023 Evaluating Software Product Quality: A systematic mapping study 26
+Principal findings
04/13/2023
27+Principal findings
SPQE is taken very seriously by researchers
The interest in evaluating SPQ has increased dramatically last year
Researchers are still searching for good approaches with which to
enhance SPQE
Few researchers chose to evaluate existing SPQE techniques
The majority of SPQE approaches that were found are data mining-
based techniques, principally fuzzy logic, regression and clustering
techniques
Data used was extracted from open source projects, domain specific
projects or from existing data sets such as NASA to evaluate SPQE
solutions
Evaluating Software Product Quality: A systematic mapping study
04/13/2023Evaluating Software Product Quality: A systematic mapping study
28+Principal findings
Case studies were used to evaluate SPQE approaches and a few
experiments were conducted in the SPQE papers selected
Source code was the main artifact concerned by SPQE approaches
Researchers are more interested to evaluate internal SQ
Half of the selected studies based their solutions on SQ models,
particularly the ISO/IEC 9126 standard
50% of the selected studies have not mentioned any well-known SQ
models, however they have evaluated SQ by using an SQ
characteristic
Reliability was the most frequently reported SQ characteristic in
SPQE literature, and was in most cases achieved using fault-
proneness
04/13/2023 Evaluating Software Product Quality: A systematic mapping study 29
+Implications
04/13/2023Evaluating Software Product Quality: A systematic mapping study
30+Implications
This study will allow researchers and practitioners to discover the
existing SPQE approaches and techniques in literature.
The empirical studies presented may provide an overview of the
efficiency of each approach.
More studies involving recent SQ models are needed to develop
approaches that will meet SQ standards, particularly the ISO/IEC
25010 standard.
The evaluation of SQ characteristics which have an impact on the
emerging market of mobile software applications, should receive
more attention from researchers.
04/13/2023 Evaluating Software Product Quality: A systematic mapping study 31
+Conclusions & Future work
04/13/2023Evaluating Software Product Quality: A systematic mapping study
32+Conclusions and Future work
Our study has summarized the existing knowledge as regards
SPQE approaches
Papers dealing with SPQE approaches between 1990 and
2013 were identified. 57 papers were selected
Ongoing research is based on proposing an empirical method
with which to evaluate SPQ in some specific environments
04/13/2023 Evaluating Software Product Quality: A systematic mapping study 33
+Thank you for your attention!Any questions?