Evaluating the impact of a MDWE Approach on the Productivity and
the Satisfaction of Software Development Teams
Yulkeidi Martínez, Cristina Cachero and Santiago Meliá
Universidad Máximo Gómez Báez de Ciego de Ávila, CubaUniversidad de Alicante, Spain
Summary
Introduction
Empirical Evidences
Productivity and Satisfaction ExperimentObjectiveSubjectsDesignAnalysis Data
Conclusions and Future works
Context
Web Engineering community advocates the use of models in order to improve software development processes for Web Applications
Many MDWE approaches have proposed the use of models to obtain automatically (MDD) or only to define (MBD) a Web Application (e.g. WebML, UWE, OOHDM, OOH4RIA, etc.)
ContextThese MDWE approaches claims the
following benefits of MDD:Short and long term productivity gains
(process)Improved the efficiency of maintainability tasks
(process)Defect and rework reduction (better quality of
the resulting product)
What explains the low level of adoption of modeling practices by the Web developer’s mainstream?
Context
The MDE (MDWE) research community have not provide a sufficient body of practical evidence that soundly backs these benefits of the modeling practices with respect to code-centric approaches
The percentage of empirical studies that illustrate the impact of MBD and MDD over productivity or satisfaction is still very low, which hampers the generalizability of the results
Empirical EvidencesA systematic review carried out in Dec 2010 on 600
papers [Martinez, 2011] that claimed to provide empirical evidence sustaining in any way MDD assertions about impact on product quality and process productivity produced the following results:
Empirical Evidences
Some experiments show how when projects grow larger, the use of MDD increases productivity (results ranging from 2 up to 9 or even 20 times)[Mellor 2003], [Alfonso, 2006], [Mellega, 2010]
An industrial experiment [Mohagheghi, 2008] contradicts previous works where MDD productivity gains and losses depending on the particular study. They consider a strong dependency of the maturity level of the tool
Objective
The aim of this study is to augment the repository of empirical data that contributes to giving a scientific answer to the following research question:
What is the impact of the development method on the productivity and satisfaction of junior developers while developing Web 2.0 applications?
Our experiment
The study analyses three representative development methods based on Fowler’s modeling madurity classification:Code-Centric: models are only used as sketches,
without tools, the modeling activity is performed in whiteboards or blank pages to discuss complex or unclear aspects of the system
Model-Based Development (MBD): diagrams show most of the details of a system in order to foster its understanding. They are usually based on standards as UML. We use the class diagram of UML with the Rational Software Modeler tool
Our experiment
Model-Driven Design (MDD): fully-fledged models that can be used to characterize partially or completely an application
To represent the business logic, the subjects have used the OOH4RIA Domain Model that permits to obtain the business logic and the persistence of a Web application using a Object-Relational Mapping (such as NHibernate)
OOH4RIA Model-Driven Development
OOH4RIA (Meliá et al. 2008) proposes a MDE process based of a set of functional models and transformations to obtain a RIA implementation
Navigation ModelUser Interface Models
Domain Model
Problem-Space Models
OOH4RIA IDE: http://suma2.dlsi.ua.es/ooh4ria
Domain Model: Business Logic and Persistence
Domain Model Characteristics
Extended Class Diagram notation (a small learning curve) based on a DSL language
Automatic synchronization of arguments of CRUD operations (reduced maintainability)
Model constraints based on Hibernate QL rules
Wizards to accelerate the modeling (creating the OID and CRUD operation by default)
Data types based on Hibernate types (generic for any database)
Synchronization with manual code (custom operations and customized operations)
Degree of automation of different paradigms
Discipline Code-Centric
(VS 2010)
Model-Based(RSM)
Model-Driven (OOH4RIA)
MODEL Sketch or Absent
Blueprint Fully-fledged (DSL)
IMPLEMENTATION Manual Semi-automatic Automatic
TEST Manual Manual Semiautomatic
Disciplines during the construction phase of the Agile UP [Ambler, 2005] development process
Subjects of the study 26 Mc S. Students of a Master in Web Software Development which
are divided into 6 teams from 4 to 6 people
75% of them had more than 2 years of professional experience as developers of Web applications
The mean age was 25.6 years old and all of them were Computer Engineering graduates
Subjects Background (prior knowledge questionnaire to the Master): 81% knew UML, and that 12% had a high-level of knowledge of it 76% had programmed with VS.NET and 12% had applied it in the
industry 0% had previous practical of MDD, although 56% of them were aware
of the existence of it
All of them had received more that 50 hours of training in the Master about the .NET programming, UML and OOH4RIA modeling
Projects
Each of the six groups developed a social network application for a different domain: Trips, Events, Hospitals, Academic, facework and Automobile
Each group implements the same three subsystems of their social networks: Events: people can invite their friends or colleagues to
attend to a place Groups: a community of users to create contents and
relationships among people Organizational Page: subjects (companies, celebrities,
etc., depending on the particular application) can publish content, photos, etc. in a unidirectional way to the social network
Experiment Design The experiment applies a factorial design eliminating
any possible order effect. Teams were randomly assigned to each treatment order.
Team/Module
Application
Group Events Organization
1 Travel Code-centric
MBD MDD
2 Events Code-centric
MDD MBD
3 Hospital MBD MDD Code-centric
4 Academics MDD MBD Code-centric
5 Facework MBD Code-Centric
MDD
6 Automobile
MDD Code-Centric
MBD
Experiment Design All projects must implement the same Architecture: an object-oriented business
logic based on a relational-mapping framework as Hibernate
Business Logicimplemented in the Experiment
Experiment Design
Independent Variables or Factors (IV) are:Meth: Method (code-centric, MBD, MDD)Mod: Module (Group, Events, Organization)
Dependent Measurable Variables (DV) are:P(Meth, Mod): a ratio variable that measures the
productivity of a team with a method and module
S(Meth, Mod): an interval variable, based on a 7-point Likert scale, that measures the satisfaction of developers with each method and module
Productivity Hypotheses
These measures have been used to test the following testable hypotheses, which are based on the research questions and the existing empirical evidence:Productivity Hypothesys (PH):
Prod (MDD) > Prod (MBD) > Prod (code-centric)Productivity-Module Interaction Hypothesys
(PMIH):
P(Module*Meth) < 0.05
The effect on Productivity of the particular module is insignificant compared to the effect of the method
Empirical Measures: Productivity
We measured both the development time and the size of the modules developed by each team. Development time: The student had to document
the time of each development activity through the JIRA tool. They report the development time of each task (modeling, programming and testing)
Module size: We measure the size of the code produced by students in source lines of code (SLOC). SLOCs come in to express the size of software among programmers with low levels of experience [Gollapudi]. We automated the obtainment of this measure through the Line Counter tool
Analysis of DataRQ1: Impact of the method on the team
productivityProductivity Means:• P(MDD) = 4.6• P(MBD) = 2.3• P(Code-Centric) = 0.23
MDD is 2 Times more productive than MDB and 20 Times more productive than Code-centric (Adhoc)
The productivity is significantly affected by the method used, and regardless of the particular module being developed
Empirical Measures: Satisfaction
Satisfaction Hypothesis (SH): S(MDD) > S(MBD) > S(Code-centric)
Satisfaction-Module Interaction Hypothesis (SMIH):
S(Module*Meth) < 0.05
The effect on Satisfaction of the particular module is insignificant compared to the effect of the method
Empirical Measures: Satisfaction
To measure Satisfaction we defined a satisfaction scale made up of eleven items, where each one was based on a 7-point Likert rating scale
Subjects responded to a Post-experiment questionnaire that includes a semantic-differential scale that required developers to judge each method describing the developer's overall satisfaction
Empirical Measures: Satisfaction
Example of Ease of Use:
Analysis of DataRQ2: Impact of method on developer's satisfaction
Productivity Means:• P(MDD) = 4,76• P(MBD) = 3,48• P(Code-Centric) = 4,17
MDD is 36 per cent more satisfactory than MDB and 14 per cent more satisfactory than Code-centric (Adhoc)
The satisfaction also is significantly affected by the method used, regardless of the particular module being developed
Satisfaction Results H1:Perceived usefulness of the development approaches
H2:Perceived ease of use of the development approaches
H3:Perceived compatibility of the development approaches
Final Results H4: Intention of Adoption (to develop the Final Master Project)
Conclusions
These results show that model-driven techniques improves the productivity and also the satisfaction among developers when they are accompanied by a generation environment
The satisfaction can decrease even below code-centric practices when the modeling activities are used exclusively as blueprints to improve the understanding, and the developers must implement manually almost all the final code
Future Work
This is the beginning of a family of experiments in which replicate this analysis with practitioners in industry and more complex Web application models (e.g. navigation and UI models)
Further experimentation is need to separate the effect of methods from their tools (VS, RSM and OOH4RIA Tool)
We must generalize to different population, methods and application types and sizes
Thanks!
Questions?
Contact:Yulkeidi Martínez [email protected] Cristina Cachero [email protected] Santiago Meliá [email protected]