Doctoral theses at NTNU, 2009:02 Sven Ziemer Trade-offs for web application development: understanding and improving current industrial practices ISBN 978-82-471-1372-1 (printed ver.) ISBN 978-82-471-1372-8 (electronic ver.) ISSN 1503-8181 NTNU Norwegian University of Science and Technology Thesis for the degree of doctor philosophiae Faculty of Information Technology, Mathematics and Electrical Engineering Department of Computer and Information Science Doctoral theses at NTNU, 2009:02 Sven Ziemer
Transcript
Doctoral theses at NTNU, 2009:02
Sven ZiemerTrade-offs for web applicationdevelopment: understanding andimproving current industrialpractices
ISBN 978-82-471-1372-1 (printed ver.)ISBN 978-82-471-1372-8 (electronic ver.)
ISSN 1503-8181
NTN
UN
orw
egia
n U
nive
rsity
of
Scie
nce
and
Tech
nolo
gyTh
esis
for
the
degr
ee o
fdo
ctor
phi
loso
phia
eFa
cult
y of
Info
rmat
ion
Tech
nolo
gy, M
athe
mat
ics
and
Elec
tric
al E
ngin
eeri
ngD
epar
tmen
t of C
ompu
ter
and
Info
rmat
ion
Scie
nceD
octoral theses at NTN
U, 2009:02
Sven Ziemer
Sven Ziemer
Trade-offs for web applicationdevelopment: understanding andimproving current industrialpractices
Thesis for the degree of doctor philosophiae
Trondheim, January 2009
Norwegian University ofScience and TechnologyFaculty of Information Technology, Mathematics and ElectricalEngineeringDepartment of Computer and Information Science
NTNUNorwegian University of Science and Technology
Thesis for the degree of doctor philosophiae
Faculty of Information Technology, Mathematics and Electrical EngineeringDepartment of Computer and Information Science
ISBN 978-82-471-1372-1 (printed ver.)ISBN 978-82-471-1372-8 (electronic ver.)ISSN 1503-8181
Doctoral Theses at NTNU, 2009:02
Printed by Tapir Uttrykk
If you don’t know where you are going,any road will do.
Chinese proverb
ii
Abstract
When web applications are developed with a strong emphasis on time-to-market,the resulting development and trade-off practices are informal and make useof uncertain and fragmented knowledge. Developers and other stakeholdershave to perform trade-offs using imprecise specifications, uncertain knowledge,and their beliefs, opinions and interests. In addition, this knowledge is frag-mented between the stakeholders, since each one holds his own views, opinionsand interests. Performing trade-offs therefore involves sharing the stakeholders’knowledge and interests before a balanced decision can be made.
The aim of this research is to understand how trade-offs are performed in webapplication development and to improve the current trade-off practices. In thisresearch we have studied the development and trade-off practices in industrialweb application development by interviewing eight Norwegian companies. Theinterviews with the companies identified real world problems that have beenused further in the research. An analysis of the collected data has resultedin the discovery of several improvement opportunities for the applied trade-offpractices. Working with these opportunities has resulted in a trade-off strategyand several trade-off tools. The trade-off strategy deals with the observationthat many companies are unaware that they are performing trade-offs and thatthe trade-off tools use qualitative assessments of the stakeholders’ belief andopinion to perform such trade-offs. Two trade-off tools are tested experimentallyby running a total of six experiments. The results of the experiments are usedto improve a release planning method. By using real world problems from theinterviewed companies the research has a realistic approach, yielding resultsthat are relevant for industrial web application development.
The following contributions are claimed: First, it describes developmentpractices found in eight small Norwegian Web Application development compa-nies, and focuses in particular on their trade-off practices. Second, a trade-offstrategy to create awareness for trade-offs, and to enable knowledge sharingbetween stakeholders, is developed. Third, qualitative assessments of trade-offoptions are applied and tested, thus enabling prioritisation of trade-off options.Fourth, trade-off methods for enabling and facilitating knowledge sharing aredeveloped. These include a release planning method that uses qualitative assess-ments to express the stakeholders’ beliefs and interests. Finally, a subprocessfor managing change in web application development is provided, implementinga learning loop for knowledge management.
iii
iv ABSTRACT
Preface
This thesis is submitted to the Norwegian University of Science and Technology(NTNU) for the doctoral degree – Doctor of Philosophy (PhD) The work hasbeen part of the Websys project, which was financed by the Norwegian Researchcouncil (NFR), and has been carried out at the Software Engineering group, De-partment of Computer and Information Science (IDI) under the supervision ofProfessor Tor Stalhane. I also had the opportunity to visit the IMSE group atVrije Universiteit in Amsterdam (The Netherlands) for 10 months.
After working in industry for some years it was an exciting step to go backto university to start this PhD research. My hope was to build on my expe-rience as project manager for a software process improvement project, and tofurther improve my insight into how software processes and practices are shapedwithin companies. This was the start of my travel through my PhD research,which took me to some unexpected situations and experiences. With time, myresearch was motivated by understanding the social relations between the in-volved stakeholders, how they communicate and how each stakeholder seeks thefulfilment of his interests.
This travel has opened a new view for me, one which combines several disci-plines, such as software engineering, organisational theory, decision making andcommunication theory. When submitting this PhD thesis I thus feel that I havenot come to an end but rather to a starting point for the continuation of thiswork.
Trondheim, 15 September 2008Sven Ziemer
v
vi PREFACE
Acknowledgements
This work would not have been possible without the help, support, guidance,and inspiration of my colleagues, friends and family, and I am indebted tomany. First of all, I am thankful to my supervisor Professor Tor Stalhane, whoencouraged me in my work, who was open to many questions and discussions,and who let me have the freedom to approach the research in my own way. Aspecial thanks also to my second supervisor, Professor Monica Divitini, who wasopen to discuss issues on research methods and the research design with me,and to Professor Reidar Conradi, who advised me on the structure of the thesis,and who gave me valuable feedback on my thesis.
While working on my PhD thesis I made two wonderful friends: Siv HildeHoumb and Xiaomeng Su. Most importantly, I am grateful for their friendship,which enriches my life. They have also supported me during my research work,and I would not have been able to complete this thesis without their support indifficult times. A special thanks to Xiaomeng for her help with the structure ofthis thesis and for her feedback.
A special thanks goes to Ilaria Canova Calori who helped me to design, runand analyse the experiments on the release planning method for web applicationdevelopment.
Part of my work was carried out at Vrije Universiteit in Amsterdam and Iam thankful to Professor Hans van Vliet for hosting my stay.
Many colleagues have made my time at the Software Engineering Groupat NTNU a memorable time, with many interesting discussions and social ac-tivities. My thanks go to Kirsti Elisabeth Berntsen, Birgit Krogstie, JenifferSampson, Sobah Abbas Petersen, Gry Seeland, Anita Gupta, Thomas Øster-lie, Glenn Munkvold, Jingye Li, Darijus Strasunskas, Christian Monch, ØyvindHauge, Gasparas Jarulaitis, and Geir Kjetil Hansen.
Thanks to the administrative staff, who helped me with many practicalissues, and to Sari Cunningham, who proofread my thesis and helped to preparethis manuscript.
Finally, I want to thank my wonderful wife Kari for her love and for herenduring support, understanding and encouragement. You convinced me toaccept this PhD position at NTNU while you were starting your own PhD workin Nijmegen (The Netherlands). I am indebted to you for all the experiencesI made during my PhD work and the view this work has given me on my ownfuture.
1.1 Research overview . . . . . . . . . . . . . . . . . . . . . . . . . . 71.2 The contribution of this work . . . . . . . . . . . . . . . . . . . . 121.3 The research activities of this research and their relation to the
3.1 Overview of companies involved in the research . . . . . . . . . . 323.2 The four research phases with the applied research methods . . . 333.3 Excerpt from a KJ-diagram – Negative experiences . . . . . . . . 403.4 An Ishikawa diagram for an experience from figure 3.3 . . . . . . 403.5 General model of an experiment . . . . . . . . . . . . . . . . . . 423.6 The experiments in this research at a glance . . . . . . . . . . . . 43
4.1 Web application development classification . . . . . . . . . . . . 504.2 Lifetime phases of a web application . . . . . . . . . . . . . . . . 514.3 Fragmented knowledge (part A) vs. shared knowledge (part B) . 544.4 Qualitative trade-off tool . . . . . . . . . . . . . . . . . . . . . . . 554.5 Five steps in qualitative trade-off assessment . . . . . . . . . . . 574.6 A combination of two SWOT analyses . . . . . . . . . . . . . . . 584.7 The configuration decision process . . . . . . . . . . . . . . . . . 604.8 The relation between the five experiments to evaluate the release
planning method . . . . . . . . . . . . . . . . . . . . . . . . . . . 624.9 The effect of introducing a new method . . . . . . . . . . . . . . 634.10 Subprocess for managing changes in the development environment 65
5.1 The contributions and their relation to the research approach andmethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
xiii
xiv LIST OF FIGURES
Abbreviations
AHP Analytical Hierarchy ProcessATAM Architectural Tradeoff Analysis MethodAWE Agile Web EngineeringCBD Component Based DevelopmentCMC Computer-mediated CommunicationCOVAP Cost-Value Approach for Prioritising RequirementsBBN Baysian Belief NetworkEF Experience FactoryFMEA Failure Mode and Effect AnalysisIFM Incremental Funding MethodKJ diagram Affinity diagram (devised by Jiro Kawakita)NRP Next Release Problem methodNTNU Norwegian University of Science and TechnologyOO Object OrientationOOHDM Object-Oriented Hypermedia Design MethodOVAC The Optimising Value and Cost in Requirement Analysis
methodQFD Quality Function DeploymentPG Planning GamePMA Postmortem AnalysisPSERM Planning Software Evolution with Risk ManagementPWC Pair-Wise ComparisonRCA Root Cause AnalysisSEI Software Engineering InstituteSWOT analysis Strengths, Weaknesses, Opportunities and Threats analysisTPWC Tool Supported Pair-Wise ComparisonTTM Time-to-marketXP eXtreme ProgrammingUML Unified Modelling Language
xv
xvi ABBREVIATIONS
Part I
Research contextand results
1
Chapter 1
Introduction
This chapter gives a short overview of the research work reported in this thesis.It starts with the problem outline for the research, and presents the researchbackground and context. The chapter also shows how the research questions,the research design, the claimed contributions, and the papers that have beenpublished as part of this research, are related.
1.1 Problem outline
Performing trade-offs in software development involves balancing several oppos-ing and interconnected factors in such a way that the outcome of each factor isacceptable for a given situation and for a given set of stakeholders. The taskof performing a trade-off requires an understanding of how the involved fac-tors are connected, and how they affect the development project. Performingtrade-offs under time pressure, and with incomplete and uncertain knowledgeof the involved factors and potential consequences of each factor, adds to thecomplexity and difficulty of this task. This research addresses how trade-offsare performed in web application development [81, 69], and how trade-off prac-tices can be improved, given the development practices applied in typical webapplication development projects.
This research focuses on web applications that are developed with a strongemphasis on time-to-market, and in competition with other web applications.While some web applications are developed in a traditional way, where webtechnology is only used for convenience, other web applications are developedwith a strong emphasis on time-to-market and within a competitive context. Inthe latter case, the use of web technology enables development practices thatfocus on short time-to-market, rapid changes and continuously deploying newversions of an application. Web applications that are developed this way facea strong competition, that comes from two directions: (1) the competitors are”only a mouse click away”, since their products are deployed on the Internet aswell, and (2) every competitor would like users to return to their web application,
3
4 CHAPTER 1. INTRODUCTION
since many web applications are competing for the same users. Thus, developingweb applications has two top priorities: to meet the users’ expectations and todevelop applications within the time constraints with respect to competing webapplications. The priorities, however, are in conflict with each other and leavethe developers with the task of performing trade-offs between time-to-marketand quality issues of an application. An informed trade-off depends on thedevelopers’ abilities to assess the consequences of their choices and to negotiatebetween the conflicting priorities.
Web application development is characterised by (1) small developmentteams, (2) many small software deployments, (3) involvement of several stake-holders from non-technical fields, (4) informal and anecdotal specifications orrequirements, and (5) short deadlines. Developers depend on their domainknowledge when developing web applications since these development practicesinvolve mainly tacit knowledge. These development practices may also be foundelsewhere. However, what makes web application development special is the in-tensity with which these practices are applied [81].
Typical trade-offs in web application development involve factors such astime-to-market, functional requirements and quality attributes. Examples in-clude application scalability (how many users can use an application simultane-ously), portability (how many browsers are supported) and performance (howlong does a user have to wait for a response). The outcome of such trade-offs hasan effect on when new functionality is deployed and to what degree the users’expectations are met. Both priorities – time-to-market and quality – have theirsupporters among the stakeholders of an application. Each stakeholder seeksthe fulfilment of his needs and wishes that corresponds to his interests and goals.Hence the conflict between these two priorities will also be found between thestakeholders.
The problem is to facilitate a resolution of the conflicting priorities betweenthe stakeholders and to negotiate a way to balance the factors. This has to bedone without a precise specification of the quality attributes involved, withoutexplicit knowledge about the consequences of the available options, and on thefly. Without a precise specification of requirements and the lack of explicitknowledge about the development activities, consequences of candidate choicescan only be described and compared imperfectly [117].
To address this problem this research deals with two subproblems. First,can the stakeholders’ tacit knowledge about the development activities be con-verted into explicit knowledge [76]? The stakeholders have their opinions, expertjudgement and experiences. How can this knowledge be expressed and used toassess different options of a trade-off? Second, can this knowledge be shared insuch a way that the total knowledge reflects the views of all stakeholders? Byreducing the expectation of one factor from optimal to something that is ac-ceptable, the outcome of other factors can rise to something that is acceptableto the other stakeholders. Can this ”give and take” between the stakeholders ontheir choices for each factor result in trade-offs with an outcome that in the endis more acceptable to a majority of stakeholders than the optimal fulfilment ofa single factor is [33, 106]? This research focuses thus on expressing the stake-
1.2. RESEARCH CONTEXT 5
holders’ tacit knowledge and on sharing all stakeholders knowledge to performa trade-off between two top priorities in web application development.
1.2 Research context
The research in this thesis is part of the WebSys project [100], a research projectfinanced by the Norwegian Research Council. The objective and the researchgoals of the WebSys project [100] are
”. . . to provide a better overall understanding of the trade-off be-tween time-to-market and reliability in developing Web-based sys-tems. This understanding will manifest itself as better methods andstandard processes that can be applied, tested and refined throughempirical work in Norwegian software companies. By this, we willgain insight into and improve the companies’ development processes”.
Main goal: To develop guidelines for the industrial development of timelyand reliable Web-based systems.
Sub-goals:
G1 Better understanding of the software process and related technologies onhow to make trade-offs between time-to-market (TTM) and reliability.
G2 Contribute to better methods for Web-based systems development wherereliability and TTM need to be considered together.
G3 Dissemination and exchange of the gained knowledge, with a strong inter-national component.
The project description calls for co-operation with Norwegian software com-panies. Contact was made with eight companies for either an interview or aworkshop on web application development. This provided this research with anunderstanding of industrial web development.
The results of this PhD research have been published at international work-shops and conferences. In addition, an international workshop on trade-offs wasorganised by the Websys project in 2006.
1.3 Research questions
The research questions for this research have been formulated incrementally.An initial set of research questions was formulated before the research started.During the course of the work they have been revised and influenced by theresults and feedback of this research. The aim of this research is twofold: tounderstand how web applications actually are developed (and in particular how
6 CHAPTER 1. INTRODUCTION
trade-offs are performed), and to improve the development practices in perform-ing trade-offs of this type of software development. This has led to four researchquestions:
RQ 1: What characterises Web Application Development?
RQ 2: How are trade-offs performed in web application development, and howare trade-offs performed that involve time-to-market as a factor?
RQ 3: How can trade-offs in web application development be improved?
RQ 4: How is knowledge transferred within web development teams?
1.4 Research design
The research questions are dealt with using several research phases and method-ologies. The research is done in co-operation with companies that are developingweb applications, and therefore web development within these companies canbe studied and real world problems identified.
This led to a research design with four phases. Each phase addresses at leastone of the research questions, and contributes to the results of this research. Anoverview is given in figures 1.1 and 1.3, and the research approach and researchphases are described in sections 3.1 and 3.2. This research is iterative, in thesense that the phases are used a bit ”back and forth”, and not executed in asequential manner. The results from one phase may affect not only the nextphase but also the previous phase. A brief description of the research phases isgiven below.
1. Data collection phase – Here, data are collected on industrial web de-velopment, and trade-off practices used for time-to-market are collected.Another objective for this phase is to identify problems and opportunitiesfor co-operation with the companies. The problems have to be relatedto the companies’ development and trade-off practices in web applicationdevelopment, and should be feasible for finding improvements.Relevant research methods: The qualitative research interview and Post-mortem Analysis.
2. Analysis phase – Here, we build up an understanding of industrial webapplication development and of the factors that influence the performedtrade-offs. Other relevant issues are to understand how development prac-tices emerge, and on what assumptions trade-offs are made.Relevant research methods: Grounded theory, and applying a frameworkto understand how development practices emerge [98, 68].
3. Construction phase – Here, some of the identified problems and co-operation opportunities are selected for further work. The objective is toimprove the development and trade-off practices and to solve the problems,
1.4. RESEARCH DESIGN 7
Res
earc
h Ph
ases
1. D
ata
Col
lect
ion
2. A
naly
sis
3. P
robl
em S
olvi
ng4.
Val
idat
ion
Res
earc
h qu
estio
nsR
Q 1
,R
Q 2
RQ
1,
RQ
2R
Q 3
, R
Q 4
RQ
3,
RQ
4C
ontr
ibut
ion
C1
C1
C1,
C2,
C3,
C4,
C5
C4
Res
earc
h m
etho
dsIn
terv
iew
Post
Mor
tem
Ana
lysi
sLi
tera
ture
stu
dy
Gro
unde
d Th
eory
R
oot c
ause
ana
lysi
sEx
perim
enta
tion
Inte
rvie
w
Publ
icat
ions
Non
e.P3
: Web
App
licat
ion
Dev
elop
men
t and
Qua
lity
– O
bser
vatio
ns fr
om in
-te
rvie
ws
with
com
pani
es
in N
orw
ay
P1: T
he u
se o
f Tra
de-o
ffs in
the
Dev
elop
men
t of W
eb A
pplic
a-tio
nsP4
: A d
ecis
ion
mod
ellin
g ap
-pr
oach
for a
naly
sing
requ
ire-
men
ts c
onfig
urat
ion
trade
-offs
in
time-
cons
train
ed W
eb A
pplic
a-tio
n D
evel
opm
ent
P5: T
rade
-off’
s in
Web
Ap
plic
atio
n D
evel
opm
ent –
How
to
ass
ess
a tra
de-o
ff op
portu
nity
P8: M
anag
ing
chan
ge in
sof
t-w
are
deve
lopm
ent
P2: T
rade
-off
Anal
ysis
in
Web
Dev
elop
men
tP6
: Kno
wle
dge
shar
ing
thro
ugh
a si
mpl
e re
leas
e pl
anni
ng m
etho
d fo
r web
ap
plic
atio
n de
velo
pmen
tP7
: An
expe
rimen
t with
a
rele
ase
plan
ning
met
hod
for
web
app
licat
ion
deve
lop-
men
t
Oth
er–
Initi
al re
sear
ch
ques
tions
– Id
entifi
catio
n of
“p
robl
em s
ituat
ion”
– R
esea
rch
ques
tion
revi
sed
– An
alys
is o
f “pr
oble
m
situ
atio
ns”
– M
ore
gene
ral p
robl
ems
are
sele
cted
, not
spe
cific
com
pany
pr
oble
ms
– St
uden
t exp
erim
ents
, us-
ing
scen
ario
s co
nstru
cted
fro
m “p
robl
em s
ituat
ions
”
Figure 1.1: Research overview
8 CHAPTER 1. INTRODUCTION
by providing methods and guidelines. More general problems are selected,not specific company problems.
The construction of new methods and guidelines is inspired by the ana-lysis from the previous phase, as well from theories, concepts and othermethods, such as quality function deployment [3], value-based softwareengineering [13] and the Delphi method [65]. A goal for method construc-tion is that the method must be easy and intuitive to use by developers,and that there is no need for intensive training.
4. Validation phase – Here, the above constructed methods and guidelinesare validated. This is done by means of student experiments. In eachexperiment the subject has to solve a task using one of the methods.Each task is part of a scenario constructed using the collected data fromreal life projects.
1.5 Papers
The following eight papers have been published as part of this research. Figure1.1 shows how these papers relate to the overall research design and the researchquestions. Papers P1 – P8 are included in part II of this thesis. A shortdescription of the papers is shown below:
P1: Sven Ziemer and Tor Stalhane: The use of Trade-offs in the Devel-opment of Web Applications. In Proc. First International Workshopon Web Quality (WebQuality 04) at 4th International Conference on WebEngineering (ICWE 2004), 27-28 July 2004, Munich, pp. 269-281. Alsoprinted in Maristelle Matera and Sara Comai (Eds.): ”Engineering Ad-vanced Web Applications”, In Proc. Workshops in Connection with the4th International Conference on Web Engineering (ICWE 2004), Munich,Germany, 28-30 July, 2004. Published by Rinton Press, ISBN 1-58949-046-0.
This paper presents a trade-off tool inspired by the Quality Function De-ployment method (QFD) [3]. The tool uses a simple qualitative assess-ment to evaluate the consequences of an option in a decision, with respectto other requirements and quality attributes. This makes it possible tocompare different options and to perform a trade-off, knowing the conse-quences.
P2: Sven Ziemer, Tor Stalhane and Magnar Sveen: Trade-off Analysisin Web Development. International Conference on Software Engineer-ing, 3-WoSQ: Proceedings of the third workshop on Software quality, pp.70–75, ACM Press, 2005. In Xiaoqing (Frank) Liu, Yan Sun, GautamKane, Yuji Kyoya, and Kunio Noguchi (Eds.): Proc. Third Workshop onSoftware Quality, held at the 27th International Conference on SoftwareEngineering (ICSE’05), St Louis, Missouri, May 17, 2005. Published byACM, ISBN 1-59593-122-8, pp. 70–75.
1.5. PAPERS 9
This paper reports on a student experiment to evaluate the trade-off toolpresented in paper P1 [124]. The experiment is a one factor experimentwith two treatment design, comparing the trade-off tool with an ad-hocapproach to select an option for a decision. The results of the experimentshowed that the groups using the trade-off tool experienced fewer diffi-culties in making the selection, fewer conflicts between the stakeholders,and a better consensus on the final decision. However, the groups usingthe trade-off tool also experienced their communication as less construc-tive. Taken together, the experiment showed that performing a trade-offbecame easier by using the provided trade-off tool.
P3: Sven Ziemer and Tor Stalhane: Web Application Development andQuality – Observations from interviews with companies in Nor-way. In Jose A. Moinhos Cordeiro, Vitor Pedrosa, Bruno Encarnacao,and Joaquim Filipe (Eds.): Proc. Second International Conference onWeb Information Systems and Technologies: Internet Technology / WebInterface and Applications (WEBIST 2006), Setubal, Portugal, April 11–13, 2006, Vol. 1. Published by INSTICC Press, ISBN 978-972-8865-46-7,pp. 495–498.In this paper, results from interviews on development practices with sevencompanies that develop web applications are reported. The results pre-sented here are on development practices for requirements, quality issuesand development process. The interviews revealed that most companiesdid not have a clear trade-off strategy, and that the most important qual-ity issues for the developers were related to the user’s experience of usingan application.
P4: Sven Ziemer, Pedro R.F. Sampaio and Tor Stalhane: A decision mod-elling approach for analysing requirements configuration trade-offs in time-constrained Web Application Development. In Proc.Eighteenth International Conference on Software Engineering and Knowl-edge Engineering (SEKE 2006), San Francisco, 5-7 July 2006. Publishedby Knowledge System Institute Graduate School, Skokie, IL, USA, ISBN1-891706-18-7, pp. 144–149.This paper presents a release planning method for web application devel-opment. The method uses a value based assessment of requirements. Itgives guidance on the construction of candidate releases that fit withinthe time constraint and supports the decision on the final release. Themethod is consensus driven, and facilitates knowledge sharing between theinvolved stakeholders.
P5: Sven Ziemer and Tor Stalhane: Trade-off’s in Web applicationdevelopment – How to assess a trade-off opportunity. In JoseCordeiro and Slimane Hammoudi (Eds.): Proc. Third International Con-ference on Web Information System and Technologies (WEBIST 2007),Barcelona, Spain, March 3–6, 2007. Published by INSTICC Press, ISBN978-972-8865-78-8, 8p.
10 CHAPTER 1. INTRODUCTION
Trade-offs with respect to time-to-market often involve quality attributes.With no specification it is the developers’ task to assess what are accept-able levels for important quality attributes of a web application. However,these acceptance levels will change, and with the informal developmentpractices found in most companies, it is hard to be aware of the change inthe end-users’ expectations. The trade-off tool presented here uses a seriesof SWOT analyses [45] to assess the options in a trade-off, and uses a crossanalysis to compare them with the current context of an application.
P6: Sven Ziemer and Ilaria Canova Calori: Knowledge sharing througha simple release planning method for web application develop-ment. In Daniel Cooke et al. (Eds.): Proc. Nineteenth International Con-ference on Software Engineering & Knowledge Engineering (SEKE 2007),9–11 July 2007, Boston, USA. Published by Knowledge System InstituteGraduate School, Skokie, IL, USA. ISBN 1-891706-20-9, pp. 686-691.
This paper reports on a student experiment to evaluate the release plan-ning method for web application development presented in P3 [123]. Theexperiment uses a one factor with two treatment design, comparing therelease planning method with an ad-hoc release planning approach. Af-ter the experiment, the students fill in a post-experiment questionnaire,with questions regarding knowledge sharing, understanding, consensus,requirement prioritisation and stakeholder satisfaction. The results showthat the control groups using the ad-hoc approach performed better onknowledge sharing and understanding, while the treatment group usingthe release planning method performed better on the remaining factors.
P7: Sven Ziemer and Ilaria Canova Calori: An experiment with a re-lease planning method for web application development. In PekkaAbrahamsson, Nathan Baddo, Tiziana Margaria, and Richard Messnarz(Eds.): Proc. 14th European Conference on Software Process Improve-ment (EuroSPI’2007), 26–28 September 2007, Potsdam, Germany. Pub-lished in Springer LNCS 4764, ISBN 978-3-540-74765-9, ISSN 0302-9743,pp. 106–117.
This paper reports the results from four experiments with the releaseplanning method presented in P3 [123]. The experiments are plannedand operated as the first experiment in P6 [122]. The experiments areplanned to test how experience in using the release planning method af-fects knowledge sharing and understanding, and what effect an improvedcommunication style in using the release planning method has on knowl-edge sharing and understanding. The results show that in both cases thetreatment group using the release planning method improved in knowl-edge sharing and understanding, compared to the control group that usedthe ad-hoc approach.
P8: Sven Ziemer: Managing change in software development. In:Lech Madeyski, Miroslav Ochodek, Dawid Weiss and Jaroslav Zendulka
1.5. PAPERS 11
(ed.): Software Engineering in Progress. In conjunction with 2nd IFIPCentral and East European Conference on Software Engineering Tech-niques (CEE-SET 2007), Work-in-progress session, 10-12 October 2007,Poznan, Poland. Published by Wydawnictwo Nakom, Poznan, Poland,ISBN 978-83-89529-44-2, pp. 184–198.
In this paper, a process to manage the change that occurs in the context ofweb development is presented. The process presented in this paper aims atimproving the trade-off practices, by facilitating a deeper understanding ofthe involved factors over time. This understanding is achieved by offeringsimple activities that plan, perform and evaluate a trade-off. A knowledgebase for the project is created, and lessons learned are made available forother stakeholders.
In addition, this research has contributed to two other publications. PapersP9 and P10 are not included in this thesis, since they do not report on researchresults that answer any of the research questions. For reference, we include asmall summary of the two papers here.
P9: Glenn Munkvold, Gry Seland, Siv Hilde Houmb and Sven Ziemer: Em-pirical assessment in converging space of users and profession-als. In Proceedings of the 26th Information Systems Research Seminar inScandinavia (IRIS’26), Helsinki, August 9-12, 2003, 14 p.
Implementing generic standardized systems in local practices often impliesa process of change involving people from various domains and commu-nities. In this paper we put particular emphasis on the new emergingcommunity of people this implies. Based on experience from the SoftwareEngineering and HCI domain, we discuss a methodology supporting thesenew challenges. We use Communities of Practice as a possible conceptualframework, and action research as a methodological approach to betterunderstand the ongoing situated activities and processes in practice.
P10: Ilaria Canova Calori, Tor Stalhane and Sven Ziemer: Robustnessanalysis using FMEA and BBN – Case study for a web-basedapplication. In Jose Cordeiro and Slimane Hammoudi (Eds.): Proc.Third International Conference on Web Information System and Tech-nologies (WEBIST 2007), Barcelona, Spain, March 3–6, 2007. Publishedby INSTICC Press, ISBN 978-972-8865-78-8, pp. 164-170.
Time pressure and quality issues represent important challenges for thosewho develop web-based systems. The ability to analyse a system’s qualityand implement improvements early in the development life cycle is of greatpractical importance. For our study we consider robustness as a criticalquality issue. Our objective is to propose a general framework for conduct-ing robustness analysis of web-based systems at an early stage of softwaredevelopment, providing a tool for evaluating failure impact severity andsupporting trade-off decisions during the development process.
12 CHAPTER 1. INTRODUCTION
1.6 Contributions
The contributions of this research are assigned in three groups: Understandingweb development practices, trade-off methods for web application development,and facilitation of knowledge sharing between stakeholders in web applicationdevelopment. An overview of the contributions and their relation with thepublished papers is shown in figure 1.2. Figure 1.3 shows in what order thedifferent activities have been performed and how they relate to the publicationsand contributions.
Contribution Development practices
Trade-off methods
Knowledgesharing Paper
C1: Development practices X P3C2: Trade-off strategy X P1C3: Qualitative assessment
of trade-off optionsX X P1, P2,
P4, P5C4: Enabling knowledge
sharingX P6, P7
C5: Knowledge management X P8, P5
Figure 1.2: The contribution of this work
C 1 Web Development Practices A description of such practices in eightNorwegian companies, based on interviews and Postmortem Analyses with thedevelopers and managers in the companies. The development practices includetrade-offs between time-to-market and quality attributes.
C 2 Trade-off strategy for web application development Strategy basedon the principles of (a) creating awareness for trade-offs, (b) balancing short andlong term wishes, (c) sharing stakeholders’ knowledge, and (d) using qualitativeassessment of options in trade-offs. The strategy is to utilise trade-offs by bal-ancing stakeholders’ interests, focusing on the opportunities, and by using thefreedom of choice to find the means to achieve the opportunities.C 2.1 Subprocess for change management (same as C 5.1) The researchprovides a subprocess for managing the change in web application development,using the trade-off strategy mentioned above.
C 3 Qualitative assessment of trade-off options and decision supportHere, several ways of assessing the options of a trade-off qualitatively are used.This enables prioritisation of trade-off options and gives valuable support fordecision making in situations with mainly tacit and uncertain knowledge.C 3.1 Trade-off tools for assessing options Tools used to assess trade-offoptions under consideration qualitatively.
1.7. THESIS STRUCTURE 13
C 4 Trade-off methods for enabling and facilitating knowledge shar-ing Several factors that contribute to enable such sharing are tested. Knowl-edge sharing builds up a common understanding between stakeholders, and con-tributes to an assessment of trade-off options reflecting all stakeholders’ views.C 4.1 Release planning method Enables knowledge sharing between stake-holders, using qualitative assessments and a consensus driven decision process.
C 5 Knowledge management for web application development Alearning loop to facilitate learning and to create a knowledge base to assistin trade-offs. The accumulated knowledge is used to manage web applicationprojects in a rapidly changing context. Over time this will increase the confi-dence in the available knowledge.C 5.1 Subprocess for change management A subprocess for managingchange in web application development, which implements a learning loop andaccumulates a knowledge base for web development projects.
1.7 Thesis structure
This thesis is divided into two parts:
• Part I presents the research questions, research results, and the evaluationof the research, as well as an introduction to the field, and puts the researchinto context. It covers chapter 1 – 6.
• Part II contains papers P1 –P8 listed in section 1.5. They provide detailedresults and discussion of the individual research activities.
An overview of the chapters in part I is given below:
• Chapter 2 – Related Works This chapter covers related works in WebEngineering, release planning, knowledge sharing, trade-off methods anddecision making.
• Chapter 3 – Research Methods This chapter presents the researchapproach, research phases and research methods that have been appliedin this research.
• Chapter 4 – Results The results of this research are presented, coveringa description of development practices, a release planning method for webdevelopment, and a process for managing change in web development.
• Chapter 5 – Evaluation The research in this work is evaluated, byrevisiting the research questions and by linking the research to the researchgoals.
• Chapter 6 – Conclusion and further work The final chapter of partI concludes the work done and points out directions for future research.
14 CHAPTER 1. INTRODUCTION
Tim
elin
e =
Influ
ence
bet
ween
two
stud
ies=
Activ
ity 1
:In
terv
iew
IPM
A I
Activ
ity 4
:In
terv
iew
IIPM
A II
Activ
ity 2
:An
alys
is I
Activ
ity 3
:Q
ualita
tive
trade
-off
tool
Activ
ity 5
:St
uden
t ex
perim
ent
(E 0
)
P2P1
Activ
ity 6
:An
alys
is II
P3
C1
C1C2
C3
Activ
ity 8
:Tr
ade-
off
asse
ssm
ent
P5C2
C3
Activ
ity 7
:Re
leas
e pl
anni
ng
met
hod
P4C4
C3
Activ
ity 1
1:Ch
ange
pr
oces
s
P8
C5
C3 C4
Activ
ity 9
:St
uden
t ex
perim
ent
(E 1
)
P6C4
Activ
ity 1
0:St
uden
t Ex
perim
ent
(E 2
.1–E
3.2
)
P7C4
Data
Col
lect
ion
Phas
eCo
nstru
ctio
n Ph
ase
Anal
ysis
Pha
seVa
lidat
ion
Phas
e
Activ
ity 1
: Int
ervie
ws w
ith 4
co
mpa
nies
and
a P
MA
with
1
com
pany
Activ
ity 4
: Int
ervie
ws w
ith 8
co
mpa
nies
and
a P
MA
with
1
com
pany
Expl
anat
ion:
Px
= P
aper
(see
sec
tion
1.5)
Cx =
Con
tribu
tion
(see
sec
tion
1.6)
Figure 1.3: The research activities of this research and their relation to thecontributions and publications
Chapter 2
Related work
This chapter gives a view on the related works for web engineering, trade-offanalysis methods, release planning methods, decision making and knowledgesharing, and thus shows the research context that this work took place in.These fields are related to work in this thesis and have inspired and guided thework done for this thesis. An evaluation of this work’s results with respect tothe related work will be given in section 5.4.
2.1 Web Engineering
The aim of studying the literature on web engineering is to get an understand-ing of how web applications are developed and what development practices areapplied to do so. Another point of interest is to understand how developmentmethods for web applications are constructed to support the distinct character-istics of web applications and development practices to develop them.
Web engineering is a subdiscipline of software engineering and studies thedevelopment of web applications and systems. A definition of this discipline wasgiven by Murugesan et al. [73]:
Web engineering is the establishment and use of sound scientific,engineering and management principles and disciplined and system-atic approaches to the . . . development, deployment and maintenanceof high quality Web-based systems and applications.
Not all researchers view the development of web applications as a sepa-rate discipline, but rather as a new application domain of software engineering[52]. However, all agree that web application development has some distinc-tive characteristics, and that a systematic and disciplined approach to developweb applications is needed. All experts agree to avoid ad-hoc development ap-proaches which result in a crisis of web application development [7]. Withouta disciplined approach, web applications will neither deliver the desired quality
15
16 CHAPTER 2. RELATED WORK
nor be developed on time. Web application development calls for continuous up-dates and refinements, which constitute an evolution of the application withoutspecific releases (as opposed to traditional software development). This makesweb application development behave like gardening. The gardening metaphorraises the question if an engineering approach is appropriate for web applicationdevelopment. Murugesan et al. [73] believe that engineering principles need tobe adapted to the web environment. Developing web applications is differentfrom developing other software applications, for a number of reasons. First, theweb is both an application and a delivery medium. Web applications have agreater focus on the look and feel, and incorporate multimedia at various degreesin presentation and interface. This results in a greater bond between art andscience. Finally, web applications are developed within a short time frame, thusmaking it difficult to apply the same level of formal planning used in traditionalsoftware development.
Web engineering is a multidisciplinary field, combining traditional softwareengineering disciplines such as requirement engineering, evolutionary systemdevelopment and security, with disciplines such as graphical design, copyrightand intellectual property rights, multimedia and hypermedia [28, 73].
2.1.1 Web development practices
Development practices in web application development are of particular interestto this research. Several studies on web development practices are found in theliterature and are briefly presented here.
In their study of web application development Ramesh et al. [81] focus onunderstanding the features that characterise web application development. Thestudy involves nine companies developing web applications in the US. The mainfinding of this study is that there are special web application development prac-tices and that they are caused by three factors: demand for rush to market,operating in a different kind of market environment, and the lack of experiencein developing such products. The resulting web application development prac-tices are characterised by parallel development activities, release orientation,tool dependence, customer involvement, prototyping, fixed architecture, com-ponents, ignored maintenance, and tailored methodology. Some of the practicesare further described and analysed in [9]. The characteristics of web applicationdevelopment practices are not necessarily unique for web application develop-ment, and may be found in traditional software development as well. However,the intensity with which they are applied is unique for web application devel-opment. Another issue raised by this study is that of a different developmentculture. The culture of web application development ”appreciates less structure,smaller teams size and diverse team compositions”. The development cultureis, however, evolving together with the product, and when the market demandsquality and maturity over speed, the development culture begins to resembletraditional software development.
A survey of web engineering in practice by McDonald et al. [70] identifiessimilar practices. In this survey, web application development practices are
2.1. WEB ENGINEERING 17
identified and described as a first step to developing a web application develop-ment process. Such a process has to address practices such as short developmentlife-cycle times, multidisciplinary teams and small development teams workingin parallel on similar tasks.
Yusop et al. [117] performed a study on five web application developmentprojects with companies from Singapore. In this study they addressed the im-pact of non-functional requirements in web application development projects,and found that non-functional requirements are not specified sufficiently. Thisis often due to uncertainty about the customer’s needs, lack of time and lackof knowledge of the importance of non-functional requirements. Functional re-quirements are, however, specified in a structured manner. Two non-functionalrequirements that are important to the companies in this study are security andintegration with other systems. As a result of the insufficient specification ofnon-functional requirements, developers have to rely on previous experience indeveloping similar systems.
In a survey of Jordanian web companies conducted by El Sheikh et al. [95],it is shown that small web application development companies find it hard toapply web engineering practices such as web metrics, standards and procedures,control of the development process and organisation issues. The survey wascarried out by using the Software Best Practice Questionnaire from ESI [48].No explanation other than a difference between European companies and smallweb application development companies in Jordan is given. However, this studystresses the same points as the studies cited above, such as small developmentteams, tight time constraints and lacking experience in developing productsunder these circumstances.
The development of web application has some distinctive characteristics, asshown by these surveys. These include small development teams, tight timeconstraints, short development cycles with frequent updates, a multidisciplinarynature of development activities, and a development culture that appreciates lessstructure. None of these characteristics are exclusive for web application devel-opment, and can be found in other application domains as well. However, whatmakes web application development different from other application domains isthe intensity with which these practices apply [81].
2.1.2 Web development methods and tools
A number of software development methods for web application developmenthave been constructed to address the special characteristics of web applications.This is the case for web design methods that address the special ways of de-signing web applications. One such method is the Object-Oriented HypermediaDesign Method (OOHDM) [93, 85] that uses a five-step process to build a webapplication. This method applies models for both the conceptual design, navi-gational design and the interface design, using UML with some small extensions.Other design methods are HDM [37], W2000 [6], WebML [18] and Autoweb [35].A commonality is that web applications are generated from platform indepen-dent models. These models focus on the web interface of the applications and
18 CHAPTER 2. RELATED WORK
on the structure of the information that is presented.Another field where specialised web methods are found is quality modelling.
The WebQEM quality model for web applications has been developed by Olsinaet al. [79, 78]. This model addresses the quality of some distinct features ofweb applications, such as the fact that web applications are content-driven,document-oriented and user-centred and that searching and browsing are im-portant functionalities for web applications. Important quality requirementsfor web applications are therefore information accuracy, information suitability,accessibility and legal compliance.
Several development processes for Web application development have beenproposed, such as Web OPEN [44] and Agile Web Engineering (AWE) [71]. TheWeb OPEN process is an extension of an established object-oriented develop-ment process, that is also well suited for component-based development (CBD).To support web development, it is extended with new activities, tasks, tech-niques and roles. Among the distinct characteristics of web application that areaddressed are architecture, content management and requirements engineering.The AWE process focuses on the multidisciplinary nature of web engineering,and stresses the many roles involved in web application development and howthey have to work together. AWE includes roles that reflect both the business,domain and creative design models. In addition, traditional software develop-ment roles are included.
All development methods constructed for web application development aimat supporting several distinct characteristics of web application development.They do so, however, in different ways. The methods described in this sectioncan be divided into two different approaches: One approach is the model-basedmethods, focusing on the content and navigational sides of web applications.These methods assume the use of a disciplined development approach with de-velopers that can model applications using a language like UML. A secondapproach is taken by the development processes, which focuses more on thesoftware development practices and aims at supporting them. The Web OPENprocess focuses on the contents level, as do the model-based approaches. How-ever, the Web OPEN process does so by including new roles and tasks andextends a traditional development team. The model-based approaches intro-duce a model-driven paradigm to software development, which has not beenfound in use among web application development practices as discussed in theprevious section.
2.2 Trade-off analysis methods
Trade-off analysis methods are found in several fields where conflicting aspectsneed to be balanced. In the Oxford Dictionary of English, a trade-off is de-fined as a balance achieved between two desirable but incompatiblefeatures. By studying trade-offs for software architecture, requirements en-gineering, and software processes and development practices, we search for anunderstanding of how trade-off methods are functioning. In particular, the fol-
2.2. TRADE-OFF ANALYSIS METHODS 19
lowing four questions are of interest to this research: (1) How are conflictingaspects of a decision identified? (2) How are conflicting aspects of a decisionanalysed? (3) How are candidate options for a decision analysed? and (4) Howare trade-offs settled and what is the stakeholders’ role in this process?
Trade-off methods for architectural trade-off analysis and requirement priori-tisation trade-off analysis are presented, in order to answer the aforementionedfour questions. In addition, five of the presented trade-off methods are sum-marised and compared to each other in figure 2.1.
2.2.1 Architectural trade-off analysis
In an architectural trade-off analysis, we assess to what extent quality re-quirements are met by a software architecture [10]. One of the best knowntrade-off methods is the Architecture Tradeoff Analysis Method (ATAM)[22, 58, 57]. This method has been developed by the Software Engineering In-stitute (SEI) for the analysis of software architectures with respect to multiplequality attributes. It aims at locating and analysing sensitivity points and trade-off points. To do so, ATAM uses scenarios that cover both the planned use of thesystem and future growth of the system. ATAM consists of 10 steps groupedinto four phases, and is organised as a workshop with up to 40 stakeholders.Sensitivity points and trade-off points are identified for conflicts between thequality attribute utility tree, architectural approach and scenarios about howthe system will be used.
In ATAM, conflicts between quality attributes, architectural approaches andplanned and future use of the system are identified by a qualitative analysis.This evaluates if the architectural approaches answer the needs of the qualityattributes and scenarios. This identifies the sensitivity and trade-off points.The focus is thus not on analysing the conflicting aspects, but on identifyingand locating them. Finding options for trade-offs and settling trade-offs are notpart of ATAM. However, ATAM is developing the architecture in an iterativeapproach, and the architecture is improved by the architect in an incrementalfashion [119]. Stakeholders are involved in this evaluation both by providingscenarios and by contribution to the analysis.
ATAM can be used together with the Cost Benefit Analysis Method(CBAM), which is another method from the Software Engineering Institute[56, 77]. This method helps architects to evaluate the cost and benefits ofpotential solutions to the trade-offs identified by the ATAM method.
In an approach called Tradeoff and Sensitivity Analysis in SoftwareArchitecture Evaluation by Zhu et al. [119], the Analytical Hierarchy Pro-cess (AHP) is used to evaluate potential software architecture designs. Thisapproach starts with trade-offs that are already identified and takes into con-sideration all quality attributes, priority weights of design alternatives for in-dividual quality attributes, and priority weights among the quality attributesthemselves. Using AHP makes it possible to consider trade-offs in relative termswhen there exist no precise specifications of quality attributes. This approachalso aims at making the decision consequences more explicit, something that is
20 CHAPTER 2. RELATED WORK
Dom
ain
Arch
itect
ure
eval
uatio
nR
equi
rmen
t prio
ritis
atio
nM
etho
dA
rchi
tect
ure
Trad
e-of
f Ana
lysi
s M
etho
dBy
: Kaz
man
et a
l. (1
998)
and
C
lem
ents
et a
l. (2
002)
Trad
eoff
and
Sens
i ti vi
ty
Ana
lysi
s in
Sof
t war
e A
rchi
tec t
ure
Eval
uatio
nBy
Zhu
et a
l. [1
17]
Goa
l-orie
nted
Ana
ly-
sis
Met
hod
By: Y
amam
oto
et a
l. (2
008)
Cos
t-val
ue re
quire
-m
ent p
riorit
isat
ion
By: K
arls
son
et a
l. (1
997)
Trad
e-of
f ana
lysi
s fo
r re
quire
men
ts s
elec
tion
By: R
uhe
et a
l. (2
003)
Iden
tifica
tion
of c
onfli
ctin
g as
pect
s
Gro
up e
valu
atio
n of
sc
enar
ios,
qua
lity
attri
bute
util
ity tr
ee
and
arch
itect
ural
ap
proa
ches
Iden
tified
as
trade
-off
poin
ts (e
.g. b
y us
ing
ATAM
)
Proj
ect c
onst
rain
ts:
requ
irem
ents
and
tim
e-to
-mar
ket
mod
elle
d in
goa
l-gr
aph
with
ann
otat
ed
attri
bute
s
Proj
ect c
onst
rain
ts:
requ
irem
ents
and
tim
e-to
-mar
ket
Proj
ect c
onst
rain
ts:
requ
irem
ents
and
tim
e-to
-mar
ket
Anal
ysis
of c
on-
flict
ing
aspe
cts
Defi
ned
as s
ensi
tivity
po
ints
and
trad
e-of
f po
ints
; han
dled
indi
-vi
dual
ly
Con
si de
rs a
ll qu
ality
at-
tribu
tes,
prio
rity
wei
ghts
of
des
ign
alte
rnat
ives
for
indi
vidu
al q
ualit
y at
tri-
bute
s an
d pr
iorit
y w
eigh
ts
amon
g th
e qu
ality
at-
tribu
tes
them
selv
es
Wei
ghtin
g of
an-
nota
ted
attri
bute
s in
go
al-g
raph
; mul
tiple
cr
iteria
are
pos
sibl
e
Pairw
ise
com
paris
on
of b
oth
cost
and
val
ue
usin
g AH
P
Req
uire
men
ts a
re d
ivid
-ed
into
sev
eral
cla
sses
.Pa
irwis
e co
mpa
rison
of
stak
ehol
der p
refe
renc
e be
twee
n cl
asse
s, a
nd fo
r al
l req
uire
men
ts w
ithin
a
clas
sAn
alys
is o
f can
-di
date
opt
ions
Base
d on
incr
emen
tal
impr
ovem
ent,
thus
no
con
side
ratio
n of
se
vera
l opt
ions
Cal
cula
tion
of re
lativ
e pr
iorit
y of
des
ign
alte
rna-
tives
usi
ng A
HP
All p
ossi
ble
cand
idat
e op
tions
are
ana
lyse
d,
usin
g TO
PSIS
Use
s co
st-v
alue
dia
-gr
am to
com
pare
can
-di
date
requ
irem
ents
Sele
ctio
n of
can
dida
te
requ
irem
ents
Settl
emen
t of
trade
-offs
Incr
emen
tal p
roce
ss
that
dev
elop
s th
e sy
s-te
m a
rchi
tect
ure
Opt
imis
atio
n ba
sed
Opt
imis
atio
n of
bes
t tra
de-o
ff; fi
nal d
eci-
sion
s m
ade
by s
take
-ho
lder
s
Dis
cuss
ion
invo
lvin
g st
akeh
olde
rs, d
ecis
ion
by p
roje
ct m
anag
er
Trad
e-of
f ana
lysi
s fo
r ca
ndid
ate
requ
irem
ents
se
lect
ion
usin
g he
uris
tic
appr
oach
Stak
ehol
der
invo
lvem
ent
Eval
uatio
n ca
n in
volv
e al
l sta
keho
lder
s;
invo
lves
usu
ally
arc
hi-
tect
and
sev
eral
oth
er
stak
ehol
ders
Prio
ritis
atio
n of
des
ign
alte
rnat
ives
is p
erfo
rmed
by
a s
ingl
e st
akeh
olde
r
The
appr
oach
use
s m
ultip
le a
ttrib
utes
that
ca
n be
eva
luat
ed b
y a
sing
le s
take
hold
er
each
Invo
lves
cus
tom
er a
nd
user
s fo
r val
ue, a
nd
deve
lope
rs fo
r cos
t pr
iorit
isat
ion;
dec
isio
n by
pro
ject
man
ager
Prio
ritis
atio
n of
requ
ire-
men
ts is
per
form
ed b
y a
sing
le s
take
hold
er
Figure 2.1: Comparison of trade-off approaches
2.2. TRADE-OFF ANALYSIS METHODS 21
usually hidden in standard AHP practice. The approach by Zhu et al. thus de-pends on several architecture design options that are weighted for each qualityattribute. While ATAM involves a potentially large number of stakeholders inthe evaluation of the architecture, Zhu’s approach involves only one stakeholderfor assigning the priority weights.
In iterative software development, a software system is developed and deployedin increments, where each increment implements a subset of the system’s func-tionality. This opens for deploying the most important requirements early inthe development, and for gaining benefits of the system early [42]. This makesrelease planning and requirement prioritisation important activities, since theymake decisions on which functionality is most desired by the customers [12].
Trade-off methods for requirement prioritisation are presented in this sectionand release planning methods are presented in section 2.4. A general overviewof techniques for requirement prioritisation is given in [12] and [54].
In a recent paper, Yamamoto et al. [114] present the Attributed Goal-Oriented Analysis Method for Selecting Alternatives of Software Re-quirements. This approach uses goal-oriented requirement analysis [26] to de-tect dependencies among requirements, and a multi-criteria method Techniquefor Order Preference by Similarity to Ideal Solution (TOPSIS) for evaluation ofrequirements. Using a goal graph, the alternatives are identified and augmentedwith attributes and calculation rules. The calculations are used in the evalua-tion to find an ideal solution of the criteria under consideration. The result ofthe analysis is used as an input to the decisions made by stakeholders.
In the Trade-off analysis for requirements selection (TARS) approachby Ruhe et al. [86], AHP is used to balance the stakeholders’ preferences. Re-quirements are organised in requirement classes, and the prioritisation is donein two steps: first, the requirement classes are prioritised, and then, the re-quirements in each class are prioritised. This reduces the number of pair-wisecomparisons in AHP. The further steps of this method consist of identifyingcandidate requirements, and a trade-off analysis of requirements selection un-der resource and quality constraints. This is done in a heuristic approach withtwo operations: rebalancing and modifying. The first operation gradually re-laxes other criteria to remove constraint violations, and tries to stay withinlimits for all defined constraints. The latter operation reduces and extends therequirement class under investigation. The final decision can then be made froma small number of alternatives.
Liu [66] presents an approach to requirement prioritisation based on a trade-off analysis between the satisfaction degree of requirements and the marginalrate of substitution in decision science. Using this approach, software qualityrequirements are classified to be either hard or soft. Hard quality requirementsmust be satisfied and soft quality requirements can be satisfied to some degree.The soft quality requirements are prioritised relatively by analysing the trade-offs between the satisfaction degrees of conflicting quality requirements, and
22 CHAPTER 2. RELATED WORK
by the customers’ trade-off between attributes on which requirements imposeconstraints, based on the marginal rate of substitution in decisions science.
Another approach that uses AHP is the Cost-value requirement pri-oritization approach by Karlsson et al. [53]. This approach uses two AHPprioritisations: the customers and users apply AHP’s pair-wise comparison forthe relative value of each requirement, while an experienced developer does thesame for the relative cost of each requirement. Using a cost-value diagram thestakeholder discusses the candidate requirements and the final decision is madeby a project manager.
Common for all four requirement prioritisation trade-off analysis methodspresented here, is that the conflicting attributes are given by the context: de-velopment effort vs. project constraints, such as time-to-market or available re-sources. All methods aim at finding a subset of requirements that fulfil projectconstraints such as time, quality and cost. The methods are, however, differentin how the conflicting aspects are analysed and how candidate requirementsare analysed. Two methods use AHP and depend on the pair-wise comparisonof requirements to a criterion stated by the method (stakeholder preference orcost-value). The other methods use stakeholder satisfaction or let their usersdefine multiple criteria for the analysis. Finally, the methods differ in how can-didate alternatives are assessed and how trade-offs are settled. All candidatealternatives are identified in the Goal-Oriented Analysis method before the opti-mal alternative is calculated, using multiple criteria. TARS uses the result fromthe AHP prioritisation for stakeholder preference to select candidate require-ments and uses a heuristic trade-off analysis to find a solution. The Cost-valuerequirement prioritisation method uses two criteria for the AHP prioritisationand uses a cost-value diagram for a final selection.
2.3 Development practices trade-off analysis
Development teams can choose among several development practices, and choos-ing among them involves trade-offs on aspects such as development performanceor the quality of the final product. While it is straightforward to identify con-flicting aspects between development practices, it is much harder to understandthe consequences that the choice of development practices may have [67]. Thedevelopment practices are selected by the developers and the project manager.In most cases this selection is done based on prior experience.
In a study on development practices in a large corporation, MacCormacket al. [67] collected data on eight development practices and investigated theimpact of these practices on productivity and quality. Selecting more flexibledevelopment practices can have a productivity penalty as a trade-off. However,this trade-off can be overcome by selecting a coherent system of developmentpractices. By using a multivariate model of defect rate and productivity, de-velopment practices that did not have a significant relation on productivity orquality when considered alone have a strong impact when considered togetherwith other practices. The study shows that some trade-offs can be overcome
2.4. SOFTWARE RELEASE PLANNING 23
by selecting a coherent system of development practices. Among the practi-cal implications of this study is that development practices should be chosendepending on the project and that the selection should be driven by primaryperformance objectives for a project. For the development practices that areinvestigated, this study illustrates how they have an influence on each otherand provides some guidance in finding the ”coherent system of developmentpractices”. Finding such a coherent system, however, depends on the projectobjectives and making a decision is left to the developers and project manager.
When comparing agile and disciplined development practices, Boehm et al.[16] identify five critical factors – size, criticality, dynamism, personnel andculture – that are ”involved in determining the relative suitability of agile orplan-driven methods in a particular project situation”. These factors are usedto perform a trade-off between agile and plan-driven development practices.The authors use the term Middle Ground for a mix of agile and plan-drivendevelopment methods. The conflicting aspects in this trade-off are agile vs.plan-driven development. Using the critical factors in a project assessment, thedevelopers and the project manager can choose the right balance between agileand plan-driven development for each factor.
2.4 Software release planning
Software release planning is the task in which it is decided which customergets which features and quality at which moment [19]. Release planning thusaddresses the decision to select features for a product release. The featuresgive the customers value. At the same time, a release has to satisfy technical,resource, budget and risk constraints [88]. Among the issues that have to betaken into consideration when making a release decision are available resources,milestones, conflicting stakeholder views, available market opportunity, risks,product strategies, and cost.
As incremental software development replaces monolithic software develop-ment, a series of releases is offered to the users and customers with additivefunctionality. Incremental development gives the customers an early sense ofvalue and the opportunity to provide early feedback. At the same time it givesthe developers an early return on their investment .
Release planning is an important activity. Without a good release planimportant critical functions can be delayed, resulting in dissatisfied customers.A good release plan, thus, can ensure that the most important requirementsare deployed first. Incremental software development is sensitive to changes oradditions to requirements, and as new releases are planned, the notion of whatthe important requirements are may have changed.
2.4.1 Release planning methods
Several release planning methods are described in the literature. In order tocompare the methods, Saliu and Ruhe [92] present ten key aspects of software
24 CHAPTER 2. RELATED WORK
release planning. These aspects are used to evaluate release planning methods,and to illustrate the differences between them. Important dimensions are theobjectives, the prioritisation mechanism, the stakeholder involvement, the re-source and system constraints, and the scope. The objective of a research plan-ning method is typically a mix of aspects such as value, urgency and risk. Thestakeholder involvement addresses issues such as what stakeholders to involve,if the stakeholders are equally important, and when and how they are involved.The scope of a release planning method refers to the number of releases thatare considered in a release planning decision. Requirements are prioritised byexpressing their priority relative to the other requirements. This can be done byusing scales of different granularity or by voting schemes where the stakehold-ers can prioritise the requirements. A release planning method has to considerseveral constraints, such as interdependencies between requirements, budget oran impact analysis of the consequences of implementing the requirements.
Two common approaches to release planning are (1) applying human intu-ition, communication and negotiation between conflicting constraints and ob-jectives, and (2) applying an optimisation approach looking for the optimalresult [88]. The human oriented approach is well suited to address stakeholderconflicts and to balance their interests and preferences manually. However, anexponential growth of the number of possible release plans makes it impossibleto process the release plans manually. The human oriented approach worksfine with Agile Development, where release planning relies on physical meet-ings between stakeholders for requirement prioritisation and conflict resolution.The optimisation approach assumes that the release planning problem can beformalised and that it can be solved as an optimisation problem. The NextRelease Problem method (NRP) [5] assigns weights to customers according tohow important they are for the software, and aims at finding the subset of cus-tomers whose features must be satisfied within the budget. The OptimisingValue and Cost in Requirements Analysis (OVAC) method [50] selects featuresthat give maximum value for minimum cost. This approach is better than thehuman oriented approach when coping with a large problem size, but does notgive stakeholders the opportunity to participate in release planning decisions.Also, these methods only plan for one release. A third approach is a hybridapproach, using elements from both the human oriented and the optimisationapproach, thus using both an optimisation resulting in several potential resultsand a negotiation between stakeholders.
Most release planning methods considered in this research do release plan-ning only for the next release. One exception is the Evolve*-method [87].Evolve* is based on value, urgency, stakeholders’ weights and satisfaction, whilethe Incremental Funding Method (IFM) [27] is based on return of investment.Other objectives are customer satisfaction (used in Cost-Value Approach forPrioritising Requirements (COVAP) [53]), and value of requirements (used inOptimising Value and Cost in Requirements Analysis (OVAC) [50]). Many ofthe methods restrict stakeholder involvement to the project manager and the de-velopers. Exceptions are COVAP (which involves also the customers and users),and IFM and Evolve* (both of which involve all stakeholders). The requirements
2.4. SOFTWARE RELEASE PLANNING 25
prioritisation is based on the optimisation of Planning Software Evolution withRisk Management (PSERM) [42], the Next Release Problem (NRP) [5] andOVAC. In IFM a heuristic approach is used for prioritisation of requirements,while COVAP uses the Analytic Hierarchy Process [90]. A mix of prioritisationtechniques is used by Evolve*, which uses both stakeholder prioritisation andoptimisation. With respect to constraints, all release planning methods addressresource constraints in some way, either by cost, risk or effort. Technologi-cal constraints are not supported by most release planning methods, except forEvolve* (which addresses both coupling constraints and precedence constraints),and IFM (which addresses precedence constraints). The same holds for techno-logical constraints that are only supported by PSERM. A detailed comparisonof these methods are presented in [92].
2.4.2 Empirical studies on release planning
Empirical studies on release planning can be divided into two categories: Sur-veys and in-depth studies on industrial release planning practices, and releaseplanning methods evaluated in experiments.
Surveys and in-depth studies A study on requirement prioritisation hasbeen conducted by Lehtola et al. [63], studying requirement prioritisation prac-tices in two Finnish software development companies. This study resulted infive lessons learned, among others that ”requirement prioritisation is an am-biguous concept” and that ”prioritisation practices are informal and dependenton individuals”. The first lesson shows that it is not always clear if requirementprioritisation should focus on importance to customers or on time-to-market.The same holds for the categories that are used for the prioritisation (e.g. low,medium, high). Discussion among the stakeholders is needed to resolve thisambiguity. The second lesson finds that prioritisation is made on the basis ofthe stakeholders’ tacit knowledge or feelings. Due to time constraints there isno time to find all information that the stakeholders might need as a basis fortheir prioritisation decisions.
A study by Carlshamre et al. [17] investigates how companies manage theinterdependencies between requirements as part of their release planning prac-tices. The study investigated interdependencies within five sets of requirements,each including 20 requirements that are of high priority for five products fromfive companies. The results show that 20% of the requirements generate approx-imately 75% of the interdependencies. This finding should influence the releaseplanning practices by identifying these 20% early on in the release planningprocess.
In an attempt to validate the key aspects of release planning that are pro-posed by Saliu and Ruhe in [92], Lindgren et al. [64] investigate the releaseplanning practices of seven industrial companies. As part of their study theyidentify a new key aspect of release planning not covered by the work of Saliuand Ruhe – short- and long-term planning. The study showed that there islittle attention shown to long-term planning of software aspects. In practice,
26 CHAPTER 2. RELATED WORK
not taking long-term issues into consideration results in later technical problemsand poor stakeholder satisfaction.
Experiments There are not many papers reporting on experiments on releaseplanning in the literature. Both studies that are considered here share a focusfor the user’s trust in the release plans that are generated by the tested releaseplanning and requirement prioritisation techniques.
Two papers by Du et al. [31, 32] report on the results of experiments thatcompare ad-hoc release planning and tool supported release planning. The aimof the experiments is to compare the confidence in release plans obtained byeither generating them in an ad-hoc and manual fashion, or by generating themby using a release planning tool: ReleasePlanner, a tool that implements theEvolve* release planning method. Confidence in release plans is an importantaspect of release plans that needs to be addressed in empirical testing of releaseplanning methods.
Karlsson et al. [55] present the results from two experiments on requirementprioritisation techniques. The experiments’ purpose is to gain an increasedunderstanding of the techniques’ time-consumption, ease of use, and accuracy.Accuracy is considered an indicator of the trustworthiness of the considered tech-niques. Three requirement prioritisation techniques are used: Planning Game(PG) (used in Extreme Programming [11]), Pair-Wise Comparisons (PWC) (asfound in the Analytic Hierarchy Process (AHP) [91]) and Tool-Supported Pair-Wise Comparison (TPWC) [53]. The results show that TPWC is superior toboth PWC and PG with respect to time-consumption, and that PG is regardedas easier to use than PWC. It could not be determined which of the techniqueswas easier to use, and no difference in accuracy could be confirmed. Theseresults indicate that tool support may be beneficial for time-consumption.
2.5 Decision making and knowledge sharing
Decision making and knowledge sharing are important issues in this thesis.When searching for related work in the literature the following questions guidedour search: (1) How can knowledge sharing be facilitated?, and (2) How is itpossible to measure the effect of a decision on factors like knowledge sharingand understanding?
2.5.1 Knowledge sharing
Knowledge sharing is the behaviour of disseminating and assimilating one’s ac-quired knowledge with other members within one’s organisation. Sharing indi-vidual knowledge increases its organisational value [118]. In the learning modelof Nonaka and Takeuchi [75, 76], knowledge is categorised as either tacit or ex-plicit. This model differentiates four ways of transforming knowledge betweenboth forms: socialisation, externalisation, internalisation and combination.
2.5. DECISION MAKING AND KNOWLEDGE SHARING 27
Software development is a knowledge intensive activity that depends onknowledge being shared by the stakeholders developing it. It is highly un-likely that a single stakeholder possesses all knowledge required to perform allactivities necessary to develop a software system [21]. Understanding the waya development process captures and manages knowledge helps to understandthe different approaches taken to knowledge sharing and knowledge transforma-tion. A comparison of knowledge sharing between agile development methodsand traditional document-based development methods is done by Chau et al.[21, 20]. Agile methods depend on direct and informal communication betweenstakeholders, and have a stronger emphasis on tacit knowledge than explicitknowledge.
Research has been undertaken to understand knowledge sharing and to iden-tify factors that influence knowledge sharing. In a study on knowledge sharingby Zheng and Bao [118], the relationship between affective commitment, per-ceived task interdependence, and job involvement is explored within five Chineseaccounting firms. Among factors that have been identified to have an influenceon knowledge sharing (e.g. organisational structure, organisational cultures andtechnology support), commitment has been chosen as the factor to explore inthis study. Commitment is the relative strength of an individual’s identifica-tion with, and involvement in, a particular object. The object of an employee’scommitment is an entity such as an organisation, supervisor, team, profession,and so on. The results of this study show that both the employee’s organisa-tional and team affective commitment were positively correlated to the extentof their knowledge sharing. The authors conclude that these results suggestthat the employee’s relative level of knowledge sharing can be influenced bymanipulating the work environment.
Knowledge sharing has an associated social dilemma: It is more efficientfor individuals to withhold information from knowledge sharing, but at thesame time it is more efficient for a team or group when all individuals sharetheir knowledge. This dilemma is explored in a study by Cress and Hesse [23].By designing a series of experiments that simulate different approaches for thebenefits and costs of sharing knowledge, they found that the experiment subjectsassess the costs higher than the rewards they get through a bonus. It seemsthat the possibility to influence subjects’ contributions behaviour through astructural change is more limited than through social factors. However, whensubjects were sharing their knowledge they did so with the consideration of whatthe others needed.
In a study from Ye et al. [115] the motivation for sharing knowledge is fur-ther explored. They interviewed 50 university teachers such as professors andassistant professors, and asked about their willingness to share tacit knowledge,and factors that influence tacit knowledge sharing. In this study only smalldifferences for knowledge sharing motivation are found. The most importantfactors for motivation to share knowledge are individual and organisational de-velopment. Other studies such as Sui and Yang [104] or Abdullah and Selamat[1] focus on the use of technology to support knowledge sharing. Hooff andWeenen [107] found in their study that the use of computer-mediated commu-
28 CHAPTER 2. RELATED WORK
nication (CMC) is an antecedent of organisational commitment that in turninfluences the willingness of knowledge sharing.
2.5.2 Empirical studies on decision making
In order to find answers to the question of how to measure the effect of decisionmaking on factors like knowledge sharing and understanding, we searched forempirical studies on decision making.
Three experiments reported by Galinsky et al. [36] explore the effects ofperspective taking and empathy in negotiations. Perspective taking and empa-thy are two related but distinct social competencies. Perspective taking is thecognitive capacity to consider the world from another individual’s viewpoint,while empathy is the ability to connect emotionally with another individual. Itis assumed that both competencies will have a positive relation with negotia-tions and the ability to close a deal. In three experiments both competencieswere measured and manipulated in two negotiation tasks that involve conflict-ing positions and differing preferences and priorities. The experiments were runwith full-time M.B.A students that were enrolled in a negotiations course. Thestudents were divided into groups of two students that had to solve a negotiationtask. The main dependent measure was whether or not the groups were able tonegotiate a deal based on the parties’ interests. Measurements of perspectivetaking and empathy were taken one week later. The participants completed anon-line questionnaire, containing items on perspective taking and empathy. Anexample of such an item is ”I believe that there are two sides to every questionand try to look at them both”. In addition, personality traits were measured,to ensure that any observed effects were independent of other major personalityvariables.
The results of these experiments show that perspective taking increases theindividual’s ability to discover hidden agreements. Empathy did not prove asadvantageous and was at times a hindrance to closing a deal. The main resultis thus that perspective taking appears to be a particularly critical competencyin negotiations.
In an experiment by Tjosvold and Field [106] the effects of decision strategyand social context on decision acceptance, understanding, decision time, andaffective reaction were studied in the context of group decision making. The de-cision strategy was either consensus decision making or majority vote decisionmaking. Consensus decision making considers the views and ideas of all groupmembers, and does not dismiss views and ideas as unimportant minority opin-ion. In order to make a group decision, the issue at hand needs to be discussedso that an alternative that all group members will agree on can be chosen. Ma-jority decision making selects the alternative that more than half of the groupmembers prefer. The social context in this experiment was either a coopera-tive condition or a competitive condition that the decision had to be made in.The hypotheses for this experiment were (1) groups using consensus decisionmaking are more effective under cooperative conditions than under competitiveconditions, and (2) that groups using majority vote decision making under com-
2.6. SUMMARY 29
petitive conditions use less time and have more positive attitudes toward theirgroup, compared to consensus decision making.
The subjects in this experiment were undergraduate students that were di-vided randomly into groups of five. Each group was assigned to one of the fourconditions in this experiment: (1) consensus cooperation, (2) consensus com-petition, (3) majority vote cooperation and (4) majority vote competition. Inthe experiment the subjects first had to rank the importance of 15 items in atask given to them individually. After that, they had to make the same deci-sion in groups, according to the instructions given to each group based on theircondition. Finally, each subject indicated its individual decision and filled ina questionnaire on its attitude towards the group. The dependent variables ofthis experiment were (1) quality of the group decisions, (2) individual commit-ment to the decision, (3) understanding of the problem, (4) time needed, and(5) attitudes toward the group. Variables (1) – (3) were measured by comparingthe individual ranking, the group ranking and the correct ranking, and variable(5) was measured by using a Likert-scale attitude measurement.
The results of this experiment can be summarised as follows: (1) Subjectsin a cooperative condition had a more cooperative relationship with their groupthan subjects in a competitive condition. (2) There was no significant differencein decision quality between groups under different conditions. (3) Members ofgroups in the consensus condition revealed that they were more committed totheir own group’s decision than members of groups with a competitive condition.(4) Subjects in the cooperative condition showed more individual understandingof the problem than did those in the competitive condition. (5) Subjects in a co-operative condition completed their decision more quickly than did competitivesubjects.
Both experiments presented in this section measure several dependent vari-ables from a decision making context. These variables have been measured byusing a Licker-scale attitude measurement, and in the case of the experiment byTjosvold and Field, by comparing the written results of the solved tasks with acorrect answer. In addition, both experiments used students as subjects.
2.6 Summary
The reviewed works in this chapter cover the main areas of this research in webengineering, trade-off analysis, knowledge sharing and decision making. Theyare related to this research in the following ways:
• The development of web applications has some distinctive characteristics.The ”demand for rush to market”, which is described in [81], is one ofthese characteristics, and is central to this research. The studies of webapplication development practices reveal how developers are dealing withthe distinctive characteristics of web application development. This re-search studies trade-off practices in web application development, with astrong emphasis on time-to-market.
30 CHAPTER 2. RELATED WORK
• This research focuses on trade-off analysis in web application development,with a special focus on trade-offs where time-to-market is one of the com-peting aspects. The reviewed trade-off analysis methods illustrate thevariation in both the purpose and the approach taken by a trade-off anal-ysis method. The purpose of a trade-off method can thus be to identifyconflicting aspects or to resolve conflicting aspects. The approach takenby trade-off methods that aim at resolving the conflicting aspects is eitherbased on human intuition or on optimisation. The approach taken alsoinvolves stakeholder involvement, and can either involve the entire groupof stakeholders or a single stakeholder. The trade-off methods developedby this research cover both purposes. However, the approach taken bythese methods is based on human intuition and on group involvement ofstakeholders.
• The style of how a decision is made influences factors such as decisionquality, effectivity, understanding and knowledge sharing. In this researchthe chosen decision style is cooperation and consensus. Knowledge sharingis influenced by other factors as well, as is shown in the reviewed works onknowledge management and knowledge sharing. This research has madethe assumption that knowledge sharing is motivated through the expec-tation of mutual benefits – a win-win situation. Facilitation of knowledgesharing is also sought through a change in the communication patterninside a group.
Chapter 3
Research Methods
In this chapter, the overall research approach will be presented and discussed.The overall research approach has emerged over time. Starting with an initialidea, it has been influenced by the context in which it took place, the researchers’interests, and by the opportunities and challenges that occurred along the road.The research methods actually used will be presented and the way they havebeen used will be discussed.
3.1 Overall research approach
The overall research approach applied in this work consists of a combinationof qualitative and quantitative research methods. The approach emerged overtime. At the outset of this work there were some ideas and initial plans for thisresearch. Over time the approach changed, due to early results of this work andthe opportunities and challenges that came along with them, and by impulsesfrom the context in which this research took place.
This work is part of the Websys research project [100], and a natural startingpoint was the research goals defined by this project (see section 1.2), togetherwith the researchers’ ideas and interests. The main goal of this work was toimprove trade-off practices in industrial web application development by pro-viding guidelines and trade-off methods. This includes a better understandingand an improvement of the software processes and trade-off practices applied inindustrial development of web applications. To narrow down this rather broadstarting point, an initial set of research questions was formulated, a researchplan was drafted and a choice of research strategy and methods was made.Later both the research questions, plan, strategy and methods were revised.
It was the intention of this research to involve companies that develop webapplications and to focus on industrial development and trade-off practices. Thiswas to ensure that the research dealt with real world situations and that theresearch results would be relevant for industrial web application developers. Wecontacted several companies to learn how they develop web applications and to
31
32 CHAPTER 3. RESEARCH METHODS
find problems and challenges these companies are struggling with. We selectedsome of the real world problems for further work and constructed four methodsto improve the development practices. Two of these methods were validatedempirically by means of student experiments.
Before companies were selected for the research, the question of what typeof web application to study needed to be settled. Web applications are de-veloped in many areas and for many purposes. This research was interestedin companies that develop web applications with a strong emphasis on time-to-market, in order to observe trade-off practices between time-to-market andquality attributes. Hence, companies developing this type of web applicationwere the preferred choice for the research. An overview of the companies thatcontributed to the research is shown in figure 3.1. The companies were eitherapproached on the basis of personal contacts or because it was known that theydevelop web applications that fit into the research interest. In most cases thecontacts were with individuals, but in two cases contact was made with entireproject teams. The first contact with a company gave an insight into whatkind of web applications a company develops. The contact with the companiesresulted in the early identification of some real world problems that were inter-esting to this research, as it was an opportunity to gain a deeper understandingof web application development and to develop and evaluate methods that canimprove trade-off practices in web application development.
Company DomainCompany
SizeProject
sizeType of development
1 Travel 8 3 In-house development
2 Media 650 3 – 4 In-house development
3 Service 80 4 – 6 In-house development
4 Travel 20 3 – 5 In-house development
5 Software 10 1 – 3 Development for customers
6 Software 25 1 – 3 Development for customers
7 Finance 250 5 – 10 In-house development
8 Software 25 5 – 10 Development for customers
Figure 3.1: Overview of companies involved in the research
Initially, we considered using case study research as an alternative overallresearch approach. The main effect of using case study research in this researchwould have been that the research results would have been validated in a realworld setting with a company. A case study would have consisted of identifyingand solving a problem and the validation of the solution, in cooperation with acompany. Compared to the overall research approach applied by this research,the advantage of the case study approach is that it takes place within a realworld setting, and that an in-depth study of the problem setting is possible. Thedisadvantage is that the results of a case study are difficult to generalise and
3.2. RESEARCH PHASES 33
hard to interpret [116]. In addition, the study would depend on the continuousinvolvement of a company, such as having access to their development environ-ment for testing out trade-off methods. The companies we had contacted forthis research were not willing to commit themselves to this type of involvement.
Using an experimental validation of the research results allows the results tobe generalised. However, as we used students as subjects, the practical value ofexperiments with respect to the generalisation can be debated, as students lackthe experience of professional developers. A detailed discussion of this topic isgiven in section 3.3.4.
3.2 Research phases
The research work in this thesis is divided into four phases. The phases were notexecuted in a strict sequential order, neither was each phase executed only once.Instead, each phase had its own purpose, using its own set of research methods,and contributed to the other phases in an iterative manner. The questions forthis research emerged in a similar way. While a set of initial research questionswas stated, it was revised later as the results and insights from both the datacollection and analysis work were known. The phases are shown in figure 3.2and are shortly described below.
Phase 1:Data collection
Interview and Literature study
Phase 2:Analysis
Qualitative analysisGrounded theory
Phase 3:Problem Solving
Engineering
Phase 4:Validation
Experimentation
Figure 3.2: The four research phases with the applied research methods
3.2.1 Phase 1: Data collection
In this phase, data were collected from developers and other IT professionalsin companies on how they develop web applications and in particular how theyperform trade-offs that involve time-to-market and quality attributes. The in-terest of this research is on web applications that are developed with a strongemphasis on time-to-market and within a competitive business environment.
Two research methods were considered for collecting data from companies:survey research [34] and qualitative research interview [62]. Surveys are wellsuited to collect data from a large number of companies. However, the col-lected data would be restricted to the survey items identified before the surveystarted, with no chance to have follow up questions. The advantage of quali-tative research interviews is that they give a richer set of data, thanks to theopportunity to ask follow up questions and questions unthought of or excludedduring preparation. With a small number of companies and an interest in in-depth data collection the choice was made for qualitative research interviews.
34 CHAPTER 3. RESEARCH METHODS
Data collection was supplemented by a literature study and Postmortem Anal-ysis (PMA) sessions with two companies.
In this phase we also recorded problems that the companies were reportingconcerning their trade-off practices. These problems were used throughout theoverall research approach, as mentioned in section 3.1.
The work in this phase has been divided into two activities that are labelledActivity 1 and Activity 4 in figure 1.3. Activity 1 was the initial data collectioninvolving four companies. It consisted of four interviews and one PMA session.Based on the general understanding of web application development achievedby Activity 1, Activity 4 was a focused data collection on trade-off practicesand consisted of ten interviews and one PMA session. This activity involved allthe eight companies listed in figure 3.1. The collected data has been the basisfor all later work in this research. In addition, it contributed directly to paperP3 in this thesis on Web application development and quality [125].
3.2.2 Phase 2: Analysis
The objective of this phase is to build up an understanding of web applica-tion development and the related trade-off practices applied. For the trade-offpractices we sought an understanding of how the practices are emerging andchanging. The analysis used the collected data from the previous phase.
The research methods used for this phase were Grounded Theory [102] andRoot Cause Analysis (RCA) which is part of the Postmortem Analysis [99,111]. Grounded Theory was used to analyse the interview data to build up anunderstanding of web application development grounded in the responses of theinterviewees. Root Cause Analysis was used as part of the Postmortem Analysissession.
In this phase the problems recorded in the previous phase were analysedtogether with the other collected data. For the work in the next two phases,some problems were selected. The main criteria for this selection was the cor-respondence with the research interests. Another criteria was to ensure thatexperimentation could be used for validation. This excluded problems thatwere too closely related to a company specific situation, as it would have beendifficult to recreate such a situation in an experiment. The selected problemswere thus problems that were relevant to several companies.
As with the data collection phase the work of this phase was divided intotwo activities: An initial analysis was performed in activity 2 and the resultsof this analysis resulted in the focused data collection mentioned above. Asecond round of analysis was performed in activity 6, focusing more on trade-offpractices and their emergence. The real world problems were selected in bothactivities. In the first round of analysis, one real world problem was selected,leading to activity 3. Three more real world problems were selected in thesecond round of analysis, leading to activities 7, 8 and 11. The results from thisphase contributed to paper P3 in this thesis on Web application developmentand quality [125].
3.2. RESEARCH PHASES 35
3.2.3 Phase 3: Problem Solving
The purpose of this phase was to provide solutions to the selected real worldproblems, and to improve and support trade-off practices applied in web appli-cation development. The resulting methods and guidelines had to be easy andintuitive to use so that they could be used without long training.
The research method used in this phase was engineering. In addition, thisphase was influenced by a number of ideas and concepts, such as value-basedsoftware engineering [13, 109, 15], Nonaka’s theory of organisational knowledgecreation (with its four modes of knowledge conversion) [76], and a study ofknowledge sharing in knowledge management by Sui [104]. The work in thisphase was divided into four activities:
• Activity 3, resulting in a trade-off method for assessing consequences ofdesign decisions. Described in paper P1: The use of Trade-offs in theDevelopment of Web Applications [124].
• Activity 7, resulting in a release planning method for web application.Described in paper P4: A decision modelling approach for analysing re-quirements configuration trade-offs in time-constrained Web ApplicationDevelopment [123].
• Activity 8, resulting in a model to evaluated trade-offs. Described in paperP5: Trade-offs in Web application development – How to assess a trade-offopportunity [126].
• Activity 11, resulting in a process for change management. Described inpaper P8: Managing change in software development [120].
3.2.4 Phase 4: Validation
The purpose of this phase was to validate the trade-off methods that weredeveloped in the previous phase.
The research method used for this phase was experimentation. The collecteddata from phase 1 was used to create realistic scenarios for the experiments. Thesubjects for the experiments were students. The work in this phase was dividedinto three activities:
• Activity 5, one student experiment to evaluate the trade-off method forassessing consequences of design decisions. Described in paper P2: Trade-off Analysis in Web Development [127].
• Activity 9, one student experiment to evaluate the release planning methodfor web application. Described in paper P6: An experiment with a releaseplanning method for web application development [121].
• Activity 10, four experiments to evaluate the release planning methodfor web application development. Described in paper P7: Knowledgesharing through a simple release planning method for web applicationdevelopment[122].
36 CHAPTER 3. RESEARCH METHODS
3.2.5 Discussion of the overall research approach
Some issues related to the overall research approach need to be addressed indetail. Working with companies in a research project like ours has its ownchallenges and risks, as experienced by this research. Companies tend to have ashort-term orientation, while research has a more long-term orientation. Evenwhen a company acknowledges the relevance of a research topic, it can be putaside due to higher priority activities pertaining to the short-term goals of thecompany or when such goals change. Another issue is that research activitiescan interfere with company politics. Research aiming at improving developmentpractices will cause changes for a company. People and organisations are oftensceptical to change, and professionals in IT companies are no exceptions. Theuncertainty of how change will affect the company’s work process can result inthe research activities not being fully supported, delayed or blocked.
The knowledge collected by the data collection phase was based on individu-als’ experiences and consists of qualitative knowledge. The purpose of collectingthis data was to gain an in-depth understanding of web application develop-ment and how related trade-offs are performed under specified circumstances.By using an interpretative approach, the involved professionals could create andassociate their own subjective meaning of how they deal with web development[80]. The collected data reflects thus the interviewed subjects’ personal experi-ence, and can not be generalised to a larger population. The personal accountsof the subjects, however, revealed a lot of the complexity inherent in trade-offsituations. This complexity would have otherwise gone unnoticed. The personalaccounts could be used to inform other observations and cases about perform-ing trade-offs in web application development. This view of the collected datainfluenced both the revision of the research questions and how the selected realworld problems were handled. A real world problem is understood by severaldata accounts, provided by different persons. When providing solutions to sucha problem the different personal accounts are reflected in the solution.
In the validation phase, a more positivistic approach, including hypothesistesting and generalising the results from a sample, was used [80]. Thus, theapplied research strategy combines both quantitative and qualitative methods,and positivist and interpretative studies. When discussing the validity of thisresearch, this has to be taken into consideration (see [61, 39]).
3.3 Research methods used in this thesis
This section gives a short presentation of each research method together withsome details on how they have been used.
3.3.1 Qualitative research interview
The purpose of an interview is to obtain detailed descriptions on how personsexperience their real life world, including their feelings, thoughts and experiences[62, page 5] [25, page 15]. Compared to multiple-choice questionnaires, the
3.3. RESEARCH METHODS USED IN THIS THESIS 37
interview makes it possible for the subject to organise their own descriptions,emphasising what they themselves find important [60].
Interviews can take several forms. Robson distinguishes three types of in-terviews, based on their degree of structure or standardisation [84, page 270]:fully structured interview, semi-structured interview or unstructured interview.A fully structured interview has predetermined questions that are usually askedin a pre-set order. This is close to survey questionnaires, except that the in-terviewee can respond openly. A semi-structured interview has predeterminedquestions, but the order of questions can be modified by the interviewer, basedon the interviewee’s response. Also, new questions can be added to explore theinterviewee’s response in more detail. An unstructured interview has a generalarea of interest and concern. It can develop completely informally. This is thetype of interview Kvale describes as the qualitative research interview in [62].
In our case, interviews had both a semi-structured and an unstructured part.For the semi-structured part we developed an interview guide. It addressedissues such as the interviewee’s experience with web projects and his role in theproject team, the typical project size for web application projects, and what kindof trade-offs typically are performed. New issues that came up in interviews wereadded in between interviews. Beyond the interview guide, the interviews werefocused on web development and decision making in development activities,and developed informally. In two cases, groups were interviewed, resulting in amore structured and organised interview. In order to gather information fromall attendees, we had some around-the-table discussions. Some details of howthe interviews were conducted are discussed below:Preparation and interview setting – Interviews were prepared by makingan appointment and providing some information about the purpose of the in-terview to the interviewee. Appointments for interviews were made one ortwo weeks in advance. Selecting an interviewee was part of making the ap-pointment. In order to get different views of web application development, theresearch was open to interview both developers, project managers or managersof IT departments (and thus responsible for operating and maintaining the webapplications). Appointments were confirmed by telephone or email, confirmingboth a date, available time, and the name and role of the interviewee. Beforean interview the interview guide was checked and used to set the focus for aninterview. The work on the interview guide and how new issues were added hasbeen described above in this section.
Two important situations of an interview are the entry and exit situations[74]. The entry situation is the formal start of the interview, i.e. the point wherethe interaction between the interviewer(s) and interviewee(s) starts. In thisresearch all interviews started with a presentation of all participants and of theinterview’s purpose. At the end of each interview we thanked the participants fortaking the time for the interview and informed them of the next steps to be takenby us. The notes we took during the interview were sent to the interviewee(s)for feedback and approval.
None of the interviews were recorded. In the first interview the interviewerasked for permission to record the interview and the interviewee did not consent
38 CHAPTER 3. RESEARCH METHODS
to recording. The only textual documentation from the interview was the notestaken by the interviewers. As this happened with the first interview, none ofthe later interviews were recorded.Data collection – After the interviews we wrote a summary based on the notestaken by the interviewer(s). The summaries were sent to the interviewees, witha request for feedback to be incorporated into the summaries. However, not allinterviewees provided feedback.Discussion – Using qualitative research interviews was important to this re-search in two ways. First, we obtained an understanding of how companiesdevelop web applications with a strong emphasis on time-to-market and compe-tition and on how related trade-offs are performed. Second, this understandinghelped us to define and revise the research questions so that the objectives ofthe research could be met.
To understand how trade-offs and decisions are made in web applicationdevelopment it was necessary to understand how the involved persons experi-ence their contribution, role and interests in a trade-off situation. A lot of theknowledge used in these situations is tacit and decision makers must thereforetrust their feelings and beliefs. Qualitative research interviews thus became animportant tool in the research.
There are some problems and pitfalls in using qualitative interviews thatmay have an effect on either the interviewees or the obtained knowledge. Somepitfalls are identified in [74] and will shortly be discussed here:
• Level of entry – The level of entry into a company may restrict theaccess to other interviewees in the company. When senior employees havebeen interviewed, junior employees may hold back information. In thisresearch this was not a problem as we interviewed most companies onlyonce, and only with one interviewee. With one of the companies we hadcontact with interviewees that were at different levels. However, this wasagreed upon in an initial meeting and we did not experience any problemswith this situation.
• Elite bias – Another concern is to avoid a bias to the responses and theresulting understanding of web application development. The responsesgiven by the interviewees reflect their level in the company and their in-terests. Interviewing people with the same interests or at the same levelin the company may thus introduce a bias. In this research a mix of se-nior project managers, IT department managers, software developers andsales managers were interviewed. The obtained understanding of web ap-plication development thus reflects the experience from different views,interests and levels in companies.
3.3.2 Postmortem Analysis (PMA)
Postmortem Analysis is a method for knowledge management and a collectivelearning activity that helps the project participant to reflect over what happened
3.3. RESEARCH METHODS USED IN THIS THESIS 39
in a project – either at the end of a project or at the end of a project phase – inorder to improve future practice [29]. It is not really a research method, but ithas been used in this research to collect and analyse data. PMA is traditionallyadvocated for situations such as completion of large projects, and learning fromsuccess and failure, and is used commonly for software process improvement.The basic idea behind a PMA session is to capture the knowledge and experiencefrom a project or an activity after it has been finished [112]. PMA sessions areconsidered well suited for information sharing and team communication in smallprojects and small companies [14].
In this research we conducted two PMA sessions with companies we hadmade contact with. In both cases, the companies wanted to share their projectexperience within the company, which is why we came up with the idea ofrunning a PMA session. A PMA session is organised as a half-day workshop.The upper number of participants is 10 – 15 persons [14, 99]. This makes PMAsessions suitable for small projects, as the entire project team can participate. APMA session consisted of three phases: preparation, data collection and analysis[14, 29].
Preparation: – A PMA session is managed by a facilitator. Both PMAsessions in this research were facilitated by the research team. Using a facilita-tor from outside the company has the advantage that the participants regardthe person as neutral and objective. A PMA session can either be general orfocused. A general PMA collects all relevant experiences; a focused PMA re-stricts the experiences to a specific activity or topic. Both PMA sessions inthis research were focused. To provide a common ground for all participantsthe project manager was asked to present a short timeline of the project and ashort status of the project. We also asked the project manager for additionalproject documentation such as a project plan and a requirements document.
Data Collection: – The data collection is a group-discussion or experience-gathering session. In this research we constructed Affinity diagrams (also calledKJ-diagrams) [94]. Other techniques that can be used are semi-structured inter-views or facilitated group discussions. We asked each participant to answer twoquestions: (1) What am I not satisfied with in the project?, and (2) What amI satisfied with in the project? Participants wrote their answers individually onpost-it notes, one post-it note for each answer. We requested three answers foreach question as a minimum. After some time, the participants presented theiranswers and placed their post-it notes on a whiteboard. This was done sepa-rately for each question, so that there would be two KJ-diagrams. We startedwith the negative experiences, so that we could finish the data collection withthe positive experiences. The participants could then leave the session withsome positive experiences.
The next step was to group the post-it notes thematically. This was donefor both the negative and positive experiences. The participants joined in frontof the whiteboard, suggested groups for the experiences and moved the post-it notes accordingly. This took some time, until a consensus on the groupsemerged. The facilitator had to help in this process when the participants couldnot agree on groups. An example of a KJ-diagram is shown in figure 3.3.
40 CHAPTER 3. RESEARCH METHODS
Too many requests from customers to keep focus on one project
No process for trans-ferring an application from development to maintenance
No development process and tools
No clear responsibility for projects and decision making
Insufficient specification of tasks
Development process
Cooperation customer – developer
Recognition of personal work effort
Communication
Better exploitation of maintenance competance
Reuse
Figure 3.3: Excerpt from a KJ-diagram – Negative experiences
Analysis: – A Root Cause Analysis (RCA) of both positive and negativeexperiences is performed, using Ishikawa diagrams. This is done as a collab-orative process to find the root causes of the considered experience. Duringthe PMA sessions conducted for this research the participants had to select twoexperiences from the KJ-diagrams, as there was not enough time to analyse allof them. For each experience analysed, the facilitator drew an arrow on thewhiteboard that pointed to the discussed experience. The participants wereasked to come up with issues that they thought had caused the analysed expe-rience. For each issue, the facilitator drew an arrow attached to the arrow thatcorresponded to the issue analysed. An example is shown in figure 3.4.
Cooperation customer – developer
No process for project start
Lackingrequirements
Customer is not asavailable as needed
Figure 3.4: An Ishikawa diagram for an experience from figure 3.3
Discussion: – Using Postmortem Analysis is well suited to bring togethera project team and to harvest their experience from being on the project. Neg-ative experience may be blocked by the presence of an external facilitator orother participants such as a supervisor or project manager. There is little afacilitator can do to avoid this during a postmortem session. However, afterthe postmortem workshop follow-up interviews with workshop participants canbe conducted. This may reveal new data and contributes to the overall under-
3.3. RESEARCH METHODS USED IN THIS THESIS 41
standing of the research topic. No such problems were experienced during thePMA sessions conducted by this research.
3.3.3 Grounded Theory
Grounded Theory is a qualitative research method for generating a generalexplanation (a theory) of a process, action, or interaction, grounded on the viewsof a large number of participants [102]. A theory should, according to GroundedTheory, be ”grounded” in data from the field [24]. The methodology was firstintroduced by Barney Glaser and Anselm Strauss in 1967 [40]. Reacting againstextreme positivism, they argued that scientific truth results from both the actof observation and the emerging consensus within a community of observers asthey make sense of what they have observed [103]. The method is built upontwo key concepts: (1) constant comparison, in which data are collected andanalysed simultaneously, in a ”zigzag” fashion, and (2) theoretical sampling, inwhich decisions about which data should be collected are determined by theemerging theory.
The analysis in Grounded Theory begins with open coding, coding the col-lected data for major categories of information. From this coding, axial codingemerges as the selection of one open coding category as the focus area of theanalysis. Axial coding then goes back to the collected data and creates cate-gories around this core phenomenon. There are several types of categories thatrelate to the core phenomenon: (1) causal conditions, (2) strategies, (3) interven-ing conditions, and (4) consequences. The final step in the analysis is selectivecoding, in which propositions that interrelate the categories are proposed [24].
Grounded theory was used to analyse the collected data in order to buildan understanding of web application development and related trade-offs. Themotivation for using Grounded Theory was to build up an understanding ofweb application development that is based directly on the data collected in theinterviews that were conducted, and to understand how the interviewed personsexperience their role and give meaning to it. Grounded theory is a suitablemethod for that purpose.
Grounded Theory was used partly, as it had to fit in with the researchapproach. Of the two key concepts, only the concept of constant comparisonwas applied. The sampling of interview objects was guided by the researchinterests as described in section 3.2.1 on the data collection phase. The conceptof constant comparison was applied by performing open coding and axial coding.The step of selective coding was not performed since it was not the purposeof the analysis phase to create a new theory of web application development.Using axial coding seemed to be sufficient to build up an understanding of thephenomenon studied. Also, the number of interviews conducted for selectivecoding was quite small compared to what has been advocated – between 20and 30 interviews [24]. Grounded Theory helped us to organise the collecteddata and to form an understanding of web application on the basis of the data.Using Grounded Theory also led to an increased interest in the social aspectsof web application development and the related trade-offs. In the case of this
42 CHAPTER 3. RESEARCH METHODS
research, analysing the collected data as described made us more aware of thesocial aspects.
3.3.4 Controlled experiment
A controlled experiment is a test or series of tests where purposeful changes aremade to the input variables (independent variables) of a process or system, sothat we may observe and identify the reasons for changes that may be observedin the output response (dependent variables) [72]. The dependent variables arethe ones that are studied to see the effect of changes made to the independentvariables. The independent variables are the variables that are manipulated orcontrolled in an experiment. The independent variables that are manipulatedare also called factors, and a particular value of a factor is called a treatment. Ageneral model of an experiment is shown in figure 3.5. Controlled experimentsare appropriate to investigate aspects such as confirming theories, or exploringrelationships. Compared to other research methods, controlled experimentshave the advantage that the study can be planned and designed to ensure highvalidity.
Experiment{Treatment
With fixed levelDependent variable
Context
Subjects
Independent variables
Figure 3.5: General model of an experiment
Controlled experiments aim at identifying a cause-effect relationship [112].Therefore, all factors that could have an influence on the object of study need tobe controlled by the experimenter. Controlled experiments are thus often donein a laboratory environment, or in vitro. A laboratory setting is naturally quitedifferent from a realistic setting, or in vivo, such as a real project, where factorsthat are excluded or controlled in a laboratory setting can have an influence onthe object of study. A major criticism against controlled experiments is thustheir lack of realism [97]. Lack of realism is a threat to the external validity ofa controlled experiment, since the findings of a controlled experiment may nothold beyond the experimental setting. When designing a controlled experiment,the experimenter is thus facing the rigour-versus-relevance battle [41].
3.3. RESEARCH METHODS USED IN THIS THESIS 43
Controlled experiments were used to evaluate two trade-off methods in thisresearch. A total of six experiments were run (see figure 3.6 for an overview).Controlled experiments need to be planned carefully and systematically and weused the experiment process in [113]. This process consists of five steps thatwere carried out for all experiments. A brief summary is given below.
# Experiment Activity Paper Object of study
1 E 0 A 5 P 2 Qualitative trade-off tool2 E 1 A 9 P 6 Release planning method3 E 2.1 A 10 P 7 Release planning method4 E 2.2 A 10 P 7 Release planning method5 E 3.1 A 10 P 7 Release planning method6 E 3.2 A10 P 7 Release planning method
Figure 3.6: The experiments in this research at a glance
1. Definition Describing the purpose and objective of an experiment, andtaking place before the experiment is planned and executed. All our experimentshave been defined using the template provided in [113].
Object of study: Two trade-off methods for web application development.Purpose: To evaluate two trade-off methods for web application
development.Quality focus: To balance the stakeholders’ interest in a trade-off.Perspective: The participating subjects engaged in role play, taking
on several roles.Context: Multi-test experiments within the objects study, with
the subjects being 3rd and 4th year students.
2. Planning Describing how the controlled experiments will be conducted.Planning consists of seven steps resulting in an experiment design.
• Context selection – Defines the environment that a controlled experimenttakes place in. To be able to generalise the results from a controlledexperiment to a larger population, the experimenter must ensure ”a closetie between the experimental situation and the ’real’, industrial situation”that the results are generalised to [96]. A controlled experiment can eitherbe conducted in vitro or in vivo. Another context classification is usedby Wohlin et al. [113], using four dimensions: (1) Off-line vs. on-line, (2)Student vs. professional, (3) Toy vs. real problems, and (4) Specific vs.general.
44 CHAPTER 3. RESEARCH METHODS
Our controlled experiments were conducted in vitro, using students assubjects. The scenarios in which the subjects had to solve their task weretaken from the interviews with the companies and based on situations thatthe interviewees reported as problematic (see section 3.1). This made theexperiments more realistic than using invented scenarios. However, this isstill not as realistic as running the experiments with an industrial settingand with IT professionals as subjects.
• Hypothesis formulation – The experiment definition is formalised into twohypotheses: The null hypothesis, H0, and the alternative hypothesis, H1.The null hypothesis states that there exists no cause-effect relationship inthe experimental setting, and that the observed differences are coinciden-tal, while the alternative hypothesis states that a cause-effect relationshipexists. The aim of the experimenter is to reject the null hypothesis H0 infavour of the alternative hypothesis, stating that a cause-effect relationshipexists in the experiment setting.
The controlled experiments of this research evaluated two trade-off meth-ods for web application development. Both methods claimed to have aneffect on how stakeholders make decisions, and that they share knowl-edge in this process. The null hypothesis stated for the experiment wastherefore that the methods under evaluation do not have an effect on howstakeholders reach a decision, and that they do not share any knowledge inthe process, whereas the alternative hypothesis stated that this is indeedthe case.
• Variable selection – Both the independent and the dependent variables forthe experiment are selected. The independent variable can be controlledand changed by the experimenter. These changes will effect the dependentvariable(s); the design of an experiment should ensure that these effectscan be measured.
The independent variable of our experiments was the applied trade-offmethod; the two treatments were the developed trade-off method or anad-hoc approach. The dependent variables reflect the expected effectson how decisions are made, such as communication quality, knowledgesharing and understanding. They were measured using a post-experimentquestionnaire.
• Selection of subjects – The selection of subjects is closely connected to thegeneralisation of the experiments’ results. To generalise the results to agiven population, the sample of participating subjects must be represen-tative of that population. This is achieved by random sampling of theexperiment population. As with the context selection, the representative-ness of subjects is also an issue to consider before generalising the resultsof a controlled experiment to a given population.
The experiments in this research used students as subjects. Students wereinvited to sign up for participation in the experiment through their student
3.3. RESEARCH METHODS USED IN THIS THESIS 45
association, which the researchers contacted. Each association received asmall honorarium for each student. The issue of using students as subjectsis discussed later in this section.
• Experiment design – Describing how the tests of the treatments are or-ganised and run. To do so, general design principles for controlled exper-iment have to be considered, and a design type for the experiment has tobe selected. This again decides what measurement scales and statisticalanalysis can be applied.
All our experiments were of type one factor, with two alternative treat-ments. Designing such an experiment implies that the experimenter iscomparing two treatments. Both treatments – both the new method andthe reference method – are applied, and the effect on the dependent vari-ables is measured, in our case by means of a post-experiment question-naire. The experiments were designed completely randomised, i.e. thesubjects were assigned to one of two treatment groups at random. Eachof the treatment groups was divided into several experiment groups, con-sisting of 3 or 4 subjects, with an equal number of experiment groups forboth treatments, i.e. a balanced design.
• Instrumentation – There are three types of instruments for a controlledexperiment: objects, guidelines and measurements, and these include spec-ifications and documents about a task, instructions on how to perform atask and data collection tools such as a questionnaire.
All experiments in this research used role play to simulate a group decision.The most important objects were therefore the scenario descriptions thatset the context for the role play, a description of the involved roles, andrequirement specifications and other documentation about the system thatwas to be used in the role play. Two types of guidelines were providedto the subjects. First, general instructions on how to participate in theexperiment were given to all subjects. Second, a tutorial on how to usethe trade-off method was handed out only to the experiment groups usingit. The groups using the ad-hoc approach did not receive any furtherinstructions. Data collection was done using two questionnaires. First, apre-experiment questionnaire was filled in prior to the experiment. Second,a post-experiment questionnaire was to be filled in by each subject afterthe experiment and was used to measure the effect of the two alternativemethods.
• Validity evaluation – The important question is whether the results arevalid for the population of the experiment sample. Four types of validitythreats have to be considered [113]: (1) conclusion validity, (2) internalvalidity, (3) construct validity, and (4) external validity.
The validity of the experiments in this research are discussed in section5.6. Details of this discussion are also to be found in the papers reportingon the results of the experiments [127, 121, 122].
46 CHAPTER 3. RESEARCH METHODS
3. Operation – After the planning phase, the experiment is executed andthe data needed for the analyses are collected. This involves three steps: prepa-ration, execution and data validation.
• Preparation – The preparation has two main tasks: selecting subjects andpreparing materials to be used in the experiment. Subjects were selectedbased on their interest in participating in the experiment and the filledin pre-experiment questionnaire. In return for their participation, a smallfee was paid to the student organisation the subjects belonged to. It wasagreed with the student organisation that the fee was paid only when thepost-experiment questionnaire was completely filled in.
• Execution – Among the issues that affect the execution of an experiment,two need further consideration: the experimental environment and thedata collection.
In our experiments, the subjects were gathered in two auditoriums, onefor each treatment group. The seating in the auditoriums was taken careof by the experimenters to ensure that each group worked as indepen-dently as possible and was not disturbed or influenced by the other groups.The experiment groups were under continuous supervision by the exper-imenters. All subjects filled in the post-experiment questionnaire. Theexperimenters checked each questionnaire for completeness immediately.In addition, the experimenters observed how the groups solved their tasksand how well they followed the instructions given.
• Data validation – In this research, the experimenters checked each post-experiment questionnaire for completeness before each subject left theexperiment. After the last of six experiments, the experimenters invited asmall subset of subjects from the experiments for a qualitative interviewsession about the experiment. Among the concerns were the issues ofambiguities and uncertainties in the questionnaire, and how the subjectsexperienced being part of a role play. No objections were raised againstthe questionnaires. The reported experiences revealed that the role playworked as expected.
4. Analysis and interpretation – In the analysis phase of the experiment,the collected data will be used to draw a conclusion. Important steps of theanalysis are descriptive statistics, data set reduction and hypothesis testing.
• Descriptive statistics – Shows the distribution of the data set and identifiesfalse or abnormal data points. The statistics used in this research werethe arithmetic mean for central tendency and variance for dispersion. Boxplots were generated for a graphical presentation of each data set.
• Data set reduction – Identifies and removes data in the analysis that maylead to wrong conclusions. In this research we used box plots to identifyoutliers. However, no data set reduction was necessary.
3.3. RESEARCH METHODS USED IN THIS THESIS 47
• Hypothesis testing – Finally, the collected data set is used for formulationin the planning phase of the experiment. The objective of hypothesis test-ing is to see if a null hypothesis, H0, can be rejected [110]. Numerousstatistical tests can be used for hypothesis testing. They can be clas-sified into parametric and non-parametric tests. Parametric tests makeassumptions on a specific distribution in the sample to be analysed, e.g.assuming a normal distribution. Parametric tests also make an assump-tion on the type of measurement scale, e.g. assuming at least an intervalscale. Non-parametric tests do not make such assumptions.
In this research, all experiments were of the same design, in that theycompared a single factor with two different treatments on a randomiseddesign. We applied both the t-test and the non-parametric Mann-Whitneytest. The level of significance p was set to 0.1.
Discussion – This deals particularly with the use of students as subjects andthe statistical analysis.
In this research we used students as subjects in the six controlled experi-ments we conducted. The reason for using students was, as is in most cases,their availability and the associated cost. Gathering a sufficient number of ITprofessionals is not a trivial task due to their engagement in other assignmentsand projects, and is expensive. The companies that were involved in this re-search were not interested in a long-time involvement. Recruiting professionalsubjects from other companies for the controlled experiments was not consideredfor the reasons mentioned above.
Using students as subjects in controlled experiments threatens the externalvalidity, and it is debatable if the results can be generalised to a population of ITprofessionals [89, 83, 96, 46, 43]. Students lack the knowledge and experience ofIT professionals, and it is assumed that students will react or respond differentlyto tasks in experiments than professionals would [96, 83, 46].
However, claiming in general that students do not have the same level ofknowledge and experience seems to be an over-simplification [96]. Students arenot inexperienced. They have experience from previous employment, part-timework and summer jobs. Assuming that professionals are experienced and possessmore knowledge than students is also not generally true. A professional can bewithout experience when starting his first job after graduation, or can have 20or 30 years experience. In addition, professionals are not experienced in everyaspect of software development but only in their specialisations. Considering ifresults from a controlled experiment can be generalised to a bigger populationcan therefore not be made on the basis of whether the subjects are professionalsor students. In order to generalise the result of a controlled experiment to adesired population, the subjects must be representative of that population [113].Therefore, defining the population of interest [59] for a controlled experimentshould involve specifying the subjects’ experience [97].
The motivation of the subjects is another issue that may affect both theinternal validity and the external validity of an experiment [47]. The result of
48 CHAPTER 3. RESEARCH METHODS
an experiment will differ depending on the subject’s motivation. IT professionalsmay have a higher motivation when an experiment is using their real artefacts asobjects of study. This is, however, not the case in our controlled experiments.All artefacts used in the experiments are artificial and the subjects have noprior knowledge of them. In the classification scheme introduced by Host etal. [47] the classification of incentives for our controlled experiments is artificialprojects. We see, therefore, no difference in the motivation between studentsand IT professionals.
Another issue is the maturity of students. Simply discarding students asinexperienced subjects does not take into account student maturity. In a studyby Remus [83] a significant difference between undergraduate students and pro-fessional managers in an experiment was found, while no significant differencewas found between graduate students and professional managers. The maturityof students should therefore also be addressed in this context.
Our controlled experiments dealt with trade-off methods and group decisionmaking in development teams. The interviews with the involved companiesshowed the involved professionals were inexperienced at performing trade-offsand making group decisions, and that the difference in personal experience withthe experiment subjects was thus not substantial. The students used in theexperiments were graduate students.
The use of the statistical test in the analysis phase of the experiments needsto be discussed in detail. The data that we collected from the experimentsare attitude measurements. The experiment subjects expressed their attitudeto statements in the post-experiment questionnaire using a five-point Likertscale. In the analysis we used the Students t-test for hypothesis testing. Thisis not considered a straight forward analysis by everyone and deserves both anexplanation and some justification. The issue under debate is what measurementscale the data collected with a Likert scale has. There are different opinions onthis. Some claim that data collected with a Likert scale is of ordinal scale[105, 49] and that the use of the t-test, therefore, is not an applicable statisticaltest, as it assumes usage on the mean values of two samples, i.e. the data mustat least be of interval scale. Some, however, claim that the data collected witha Likert scale is of interval scale, and that the use of the t-test is applicable[30]. We addressed this issue as follows: First, during the analysis phase, theWilcoxon test (which is a test that is applicable on ordinal data) was applied aswell. The results were the same, except that the parametric t-test was strongerand yielded lower p-values. This had consequences for the significance level,as the t-test showed that more items from the questionnaire were different forboth treatment samples, with a significance level of 0.1. Second, given thatthe results from the t-test and the Wilcoxon test were so close, the results werereported based on the t-test value. Besides the fact that the tests yielded resultsthat were close, it was a pragmatic approach to use parametric tests on thesedata, as there is a lot of evidence that this is a good practice, even if the datais of a non-applicable measurement scale [108].
Chapter 4
Results
This chapter summarises the main results of our research and focuses on howthe results are related to each other and how they fit into the overall researchdesign. The details about the results are found in papers P1–P8 in part II ofthis thesis.
4.1 Web applications and development practices
In this section we present the results from the data collection phase and theanalysis phase. The work for these results has been carried out in the fouractivities that took place in the data collection phase and analysis phase (seefigure 1.3). The final subsection 4.1.7 motivates the work with the trade-offtools that are presented in section 4.2 to 4.5.
4.1.1 Web application characteristics
Web applications are found in many contexts and developed in many ways.The characteristics of a web application are among the factors that shape thedevelopment practices applied to develop it [68]. The companies that were in-terviewed for this research develop web applications in different contexts. Basedon the interviews with the companies, we found characteristics of the developedweb applications that have an impact on the applied development practices.These factors (see figure 4.1) can vary from application to application, and thedescription below shows the ”extreme” situations we found.
• End-users – The number of end-users varies from small to a potentiallylarge number, from known to unknown. For in-house web applications(e.g. a room booking system) the developers know the number of usersupfront, as well as what functionality they expect. Web applications thatoffer their services to ”all” users that have access to the Internet, haveno knowledge about how many users are using the application or whatfunctionality they expect.
49
50 CHAPTER 4. RESULTS
End-users
DependencyTime-to-Market
OwnershipRequirement elicitation
Finite/Known
Large/UnknownInternet
CriticalNewapplication
In-house
Bespoken
Marketingorientation
Customer
satisfaction
Rush
-to-
mar
ketM
anag
ed
rele
ases
Figure 4.1: Web application development classification
• Purpose – Web applications are developed for different purposes. Thepurpose can be a convenient way of deployment or operating a businesscritical application. In the latter case, the business is relying on the webapplication availability (no business when the application is unavailable),reliability (most users want a predictable service), newness (how it attractsnew users) and attractiveness (the experience it gives to its users).
• Requirements – Requirements are elicited for different purposes, relatedto the marketing activities of a web application. New requirements canbe elicited for marketing purposes only, with no value for the user. Theserequirements aim at attracting new users. New requirements may alsobe elicited to satisfy the users, with functionality that is valuable to theusers.
• Time-to-market – Time-to-market is dealt with differently than in tra-ditional software development. A typical approach is rush to market,releasing new functionality as early as possible. This is done to attractusers by offering new or updated functionality. Time-to-market becomesmore important than the quality of the application.
• Ownership – Web applications can be operated in several ways. Theycan be operated as an in-house application, where developers will maintaintheir own company’s application. But they can also be operated as aservice for a customer, with a contractual agreement between the customerand the service provider.
4.1. WEB APPLICATIONS AND DEVELOPMENT PRACTICES 51
4.1.2 Web applications lifetime phases
The developers’ focus when developing a web application changes over time (seefigure 4.2). When the first version of a web application is deployed, the focusis on attracting new users by introducing innovative functionality at a highspeed (curve 1). This focus changes to satisfy the users a web application hasattracted, by deploying web applications with a high quality (curve 2). Thischange of focus comes gradually and goes through several phases. The phasesrepresent a continuum of change and are not to be understood as self-containedphases. Three ”phases” for a typical web application are described below:
Importance
Application Lifetime
Curve 1: NewnessUser attractionShort TTM
Curve 2: High QualityUser satisfaction
Start-up phase Final phaseMiddle phase
Figure 4.2: Lifetime phases of a web application
• The start-up phase – The first version of a web application has beendeployed, and the most important objective is to attract users. New ideasfor services, fancy functions and short time-to-market are some of themeans to achieve this. The risks in this phase are late delivery, that notenough users are attracted and that the first-time users do not return.
• The middle phase – The web application is important for the owner’sbusiness. New functionality is added to fulfil users’ expectations. Therisks in this phase are failing to add the right functionality and failing tobuild up a loyal user base. Deployment is changing from rush-to-marketto a more planned deployment strategy.
• The final phase – The application has grown, is critical for the business,and has become mature with respect to quality. Not fulfilling the end-users’ expectations is considered the main risk of the application.
4.1.3 Web application development practices
When observing the development practices within the companies that we inter-viewed for this research, the focus was on the way trade-offs were performed in
52 CHAPTER 4. RESULTS
web application development. For most companies, the observed developmentpractices contribute to what the companies perceive as the successful develop-ment of their web applications. Only when the companies themselves experiencethese practices as problematic should they be changed.
Requirement elicitation and specification Requirements are elicited andspecified informally. Elicitation is done by the customer or by the developers.Customers may produce a document that describes the required functionalityof a web application as part of the contract. Developers identify requirementsin discussions with other stakeholders or in the user support activities they areengaged in. Requirements are documented in a simple way, such as in emails orsimple documents with no predefined document structure. No further analysisis performed. The developers have to rely on their experience and domainknowledge to understand the requirements and to implement a web application.
Testing Due to the lack of a detailed requirement specification, web applica-tions are tested informally as well. Functional requirements are tested in sucha way that the main paths through the code are executed and tested. Qual-ity attributes such as performance or reliability are tested without a properspecification, and the individual web application’s behaviour is compared to thedevelopers’ expectation. The application is tested usually only in the developers’browser.
Project organisation A typical size of a web application development teamfor the companies interviewed for this research is two to five developers. Thedevelopers are responsible for a wide range of activities or roles, from codingand graphical design to user support. Each role has its specific responsibility,and all contribute to the overall success of the application. In many cases thedevelopment team is also responsible for the operation and maintenance of theweb application. The main activity of the developers is coding. Developers arealso responsible for ensuring the security and the performance of the application.In most companies, developers have to maintain the applications they developedand to perform user support. Contact with the users is done either by chat, emailor phone. While helping the users, they also perform maintenance work initiatedby the users’ feedback. Part of the user feedback is bug reports, to be fixed bythe developers. Team communication is mostly done orally, in group meetings,or electronically, via email and IRC communication. There is little writtencommunication except for the emails, and the format of the communication issimple, without the use of templates.
4.1.4 Trade-off situations
In most cases, the developers we interviewed were not aware of the trade-offsthey performed, and performed them intuitively and informally. The trade-offsthat we identified involve time-to-market and development practices.
4.1. WEB APPLICATIONS AND DEVELOPMENT PRACTICES 53
Requirements specification and testing A typical trade-off found in webapplication involves time-to-market and the applied requirement and testingpractices. To develop new functionality rapidly, informal requirement and test-ing practices are applied. Thus, requirements are neither analysed, detailed norfully tested. New functionality can be deployed earlier than would have beenthe case with more formal requirement and testing practices. The risk with thistrade-off is that the new functionality does not meet the users’ expectations orthat it may be of bad quality. This risk is usually dealt with by frequent updatesto fix problems, or by performing a rollback to the previous version.
Release planning A second trade-off situation that involves time-to-marketis to deploy a web application in several releases. The first release can bedeployed at an earlier time, compared to deploying a web application in a singlerelease only. Such an early release can attract users, and can give an earlyreturn on investment to the stakeholders. The challenge of this trade-off is tofind a release plan that maximises the return on the investment and minimisesthe time to the next release.
Estimation and quality attributes Effort estimation becomes difficult withthe informal requirement and testing practices applied. To keep time-to-marketshort when web applications are developed, quality attributes are used as bal-ancing items. Functional requirements are easier to test and are implemented,whereas quality attributes are much harder to test and not fully implementedwhen the development project is pressed for time. In case a not fully imple-mented quality attribute becomes a problem, this can be fixed in a later update.
4.1.5 Trade-off strategy
Most companies were unaware that they performed trade-offs and thus had noexplicit trade-off strategy. Despite the lack of a clear trade-off strategy, mostcompanies had a clear idea of how to handle problems pertaining to their webapplications, and as many of the potential problems stem from the trade-offsthey have performed, this can be seen as a kind of trade-off strategy. There weretwo things that most companies were prepared to do: react quickly to problemsand watch the user reaction. The first part of this trade-off strategy is to reactquickly to a problem, e.g. by fixing a bug and redeploying, or by performing aroll back to the previous version of a web application. The second part of thistrade-off strategy is to watch the user reaction. As long as the current versionof a web application is accepted by the users, no immediate change is necessary.This makes user satisfaction the litmus test for any of the trade-off situations.
4.1.6 Tacit knowledge
The stakeholders that are involved in trade-offs have to base their decisionsmainly on tacit knowledge they have about the development project. The avail-able knowledge is thus fragmented and each stakeholder possesses his share of
54 CHAPTER 4. RESULTS
expectation about the development activities (see part A in figure 4.3). Eachstakeholder’s tacit knowledge is related to his needs and interests, and a stake-holder thus has little insight into other stakeholders’ interests and needs.
S1
S3
S4
S2
S1
S3
S4
S2
A B
S1–S4: stakeholders in a development project
Figure 4.3: Fragmented knowledge (part A) vs. shared knowledge (part B)
4.1.7 Improving trade-off practices
The companies interviewed for this research were not aware that they performedtrade-offs and were thus giving away the freedom that comes with making adeliberate decision. However, we found that issues the companies experienced asproblematic were related to the trade-off practices applied. To improve trade-offpractices we have identified three objectives that will help companies to performtrade-offs. These objectives have been addressed by the trade-off tools that aredescribed in the remaining sections of this chapter.
1. Awareness – Stakeholders need to be aware of the trade-offs they perform.Being aware of trade-offs implies that the stakeholders identify and evaluatecandidate options for a trade-off, are aware of the consequences of the candidateoptions, and make a decision based on an assessment of the available options.This gives the stakeholders the freedom to chose the option that best supportstheir interests. This topic is further addressed in this research by a tool forassessing trade-offs (section 4.3) and by a process for managing change in webapplication development (section 4.5). Both tools aim at making stakeholdersaware of the trade-offs they have to perform.
2. Qualitative assessment – To assess the candidate option of a trade-off, the stakeholders have to use their tacit knowledge, which consists of theiropinions, beliefs and expert judgements. This knowledge can be expressed qual-itatively, and can be used for analysis and prioritisation of candidate options
4.2. QUALITATIVE TRADE-OFF TOOL 55
in a trade-off. This is addressed in the qualitative trade-off tool described insection 4.2 and the release planning method for web application developmentdescribed in section 4.4.
3. Knowledge sharing – The fragmented knowledge needs to be sharedand combined to create a common understanding of a situation and to makea balanced decision that takes different stakeholders’ interests into account.The result of obtaining a common understanding through knowledge sharing isshown in part B of figure 4.3. In order to make the stakeholders share theirknowledge we have applied a consensus-based approach to decision making.This has been tested in a consensus-based release planning method for webapplication development that is described in section 4.4.
4.2 Qualitative trade-off tool
The first trade-off tool that we worked on in this research is a tool for assessingand comparing candidate options in a trade-off qualitatively [124]. The tool isdescribed in paper P1 (chapter 7).
Tools, methods
Met
hod
1
Met
hod
2
Rob
ust P
atte
rn
Load
Tes
t too
l 1
Brow
ser T
est
Tota
l Met
hod
1
Tota
l Met
hod
2
Requirements1 Dynamic effects 9 3 0 -3 -3 3 -32 High performance -9 0 0 9 0 0 93 Short TTM -3 0 0 -3 -3 -9 -64 Run on all browsers -9 0 0 0 9 0 95 Tolerate bad data everywhere 0 0 9 0 0 9 9
The qualitative trade-off tool is inspired by Quality Function Deployment(QFD) [3] and uses a similar matrix. Tools, techniques, or requirements can becompared with a set of important requirements. For each comparison, the rela-tionship is assessed and represented by a number. There are five relationships:(1) 9 – very strong positive relationship, (2) 3 – strong positive relationship, (3)0 – neutral relationship, (4) -3 – strong negative relationship, and (5) -9 – verystrong negative relationship. An example is shown in figure 4.4. In the exampletwo options for a component with dynamic effects for the user interface in a web
56 CHAPTER 4. RESULTS
application are compared. For each option (i.e. Method 1 or Method 2 in thefigure) its relationship to important requirements are assessed. The comparisonshows that Method 1 is best for the dynamic effects, but has negative conse-quences for the other requirements. Method 2 is not as good for the dynamiceffects as Method 1, but it is much better for the other requirements.
This example illustrates how this tool can be used to discuss a trade-off andto create an awareness of the consequences of this trade-off. The example is notjust a simple decision on what component to select. It is a trade-off betweenstrong user attraction (i.e. state-of-the-art dynamic effects) and a reliable webapplication. The assessment of consequences is done qualitatively and usesthe stakeholders’ tacit knowledge. The decision has still to be made by thestakeholders. However, the tool supports the decision making by showing theconsequences of the involved options with respect to important requirements ofthe development project.
This tool has been used in experiment E0 (see figure 3.6) [127]. The detailsand the results of the experiment are published in paper P2 (chapter 8). Thegeneral design of the controlled experiments in this thesis are also presented insection 3.3.4. The alternative hypothesis in the experiment was that the use ofthe trade-off tool will improve the quality of the communication between thestakeholders in a trade-off meeting. The results showed that the groups usingthe qualitative trade-off tool found it easier to make a decision and agreed moreon the decision they made than the groups not using the tool.
4.3 Trade-off assessment
Next to using qualitative assessment in trade-offs was the issue of being aware oftrade-off situations. The trade-off assessment method [126] has been developedto create awareness of trade-offs by comparing the consequences of candidateoptions to decisions. The method is described in paper P5 (chapter 11).
Being aware of a trade-off involves knowing the opportunities and risks ofthe trade-off, and finding a reasonable balance between them. To be aware ofpotential trade-offs when making decisions, the focus needs to be on these threefactors:
• The opportunities – what can be achieved with introducing new changes,and what are the chances to have success? This implies the identificationof opportunities and enablers.
• The risks – what can go wrong when changes are introduced, where canthe chance to have success be lost? This implies the identification of risksand of the barriers to prevent them.
• The trade-offs – how can the opportunities and risks be balanced in sucha way that the best combination between them is chosen. What is thebest combination depends on the stakeholder, and an agreement betweenthe stakeholders is necessary.
4.3. TRADE-OFF ASSESSMENT 57
To identify opportunities and risks we can perform a SWOT analysis [45].Assessing a trade-off involves the analysis of both the web application’s currentsituation and the candidate alternatives in a trade-off. By analysing the results,including the stakeholders’ long-time goals for the web application, the conse-quences of the trade-off alternatives are identified and a decision can be made.Five steps are involved in the analysis (see figure 4.5):
(4) Trade-off analysis
(3) Application assessment
(1) Application goals(not mandatory)
(2) Trade-off alternatives 1 – n
( 5 ) D e c i s i o n
Figure 4.5: Five steps in qualitative trade-off assessment
• (1) Describe the application’s objectives – Define the objectives andgoals of the web application project. What are the opportunities and risksinvolved in the development project.
• (2a) Identify candidate options for a trade-off option – The actualsituation is studied and potential options to a decision are identified anddescribed.
• (2b) Assess candidate options – A SWOT analysis is performed foreach candidate option. The candidate options should not be too close toeach other. If they are, it can be difficult to sort out the assessments andconsequences.
• (3) Assess the application situation – Perform a SWOT analysis forthe actual situation of the web application, including its standings withits competitors and how it does on user satisfaction.
• (4) Perform the trade-off analysis – The SWOT analysis of eachcandidate option (step 2) is combined with the SWOT analyses for theapplication (step 3). The elements of the SWOT analyses are combined byassessing if they support or block each other. This is done by identifyinga relationship between each combination of the SWOT elements, that caneither be strong, neutral or weak. The relationships are shown as ”+”,””, and ”-” respectively in the diagram.
58 CHAPTER 4. RESULTS
+ There is a positive relationship between two issues in the table.Strengths and opportunities support and complement each other, orthey are stronger than weaknesses and threats.
” ” There is a neutral relationship between two factors in the SWOTanalysis. None of the factors have an impact on the other factors.
– There is a negative relationship between two issues in the table.Strengths and opportunities conflict with each other (only one ispossible to implement), or weaknesses and threats are even strongertogether.
Based on this analysis, the candidate options are ranked, thus indicatingwhich trade-off alternative is perceived as the most appropriate in thisparticular situation.
• (5) Make decision – To make a final decision the stakeholders choose acandidate option from the ranked list of candidate options (step 4). As thestakeholders may have different rankings, a reconciliation between theirinterests may be necessary. This issue is addressed in section 4.4.
SWOT analysis for Application
A_S
1: C
an te
st n
ew
func
tiona
lity
A_W
1: L
ittle
or n
o te
stin
g
A_O
1: W
eb a
pplic
atio
n is
eas
ily c
hang
ed
A_T
1: U
sers
do
not r
etu
rn d
ue to
poo
r qua
lty
SWO
T an
alys
is fo
r ca
ndid
ate
optio
n 1
T1_S1: Flexible deployment routines – + –T1_S2: User participation + –T1_W1: Dependent on developers’ domain knowledge – +T1_W2: No verification and validation of requirements –T1_O1: Attracting new users +T1_O2: Getting early user feedback +T1_T1: Customers are not satisfied –
Figure 4.6: A combination of two SWOT analyses
An example is shown in figure 4.6, with the SWOT analysis for a web ap-plication and a candidate option for a decision. The web application that isanalysed here has been deployed for some time and is increasing its maturity.The quality of the application becomes more important. Each line of the SWOT
4.4. A RELEASE PLANNING METHOD 59
analysis for candidate option 1 is combined with each row of the SWOT analy-sis for the application. When combining ”T1 S1: Flexible deployment routines”and ”A W1: Little or no testing”, a negative relationship is identified in thissituation. No testing and flexible deployment routines is understood as a combi-nation that may result in deploying functionality with poor quality. Combining”T1 O1: Attracting new users” and ”A S1: Can test new functionality” resultsin a positive relationship, since the attraction of new users still is important forthe web application. In the example, the trade-off assessment method helps theinvolved stakeholders to be aware of the trade-off they are performing. The de-cision they are to make has to be balanced so that new attractive functionalityis deployed and the quality of the application is satisfying to the users.
The trade-off assessment tool is used to analyse a decision and to make thestakeholders aware of the trade-offs that are involved in the decision. Using aSWOT analysis in this method is not essential, and other methods to assess op-portunities and risks can be used, like Impact Estimation tables [38] or BalancedScorecard [51].
4.4 A release planning method
Knowledge sharing is an issue that has been addressed in the release planningmethod for web application [125]. The method is described in paper P4 (chapter10).
The release planning method consists of five steps to prioritise requirementsand candidate releases and to find a consensus on the final decision. The methodincludes all stakeholders of a web application development project, thus showingthat all stakeholders’ interests and views are important for the decision (see alsofigure 4.3). The requirements and candidate releases are prioritised by using avalue-based approach. The stakeholders give their opinion on a requirement’sor candidate release’s return on value [13]. Usually, there is a considerableinvestment at stake for every new release, and stakeholders want to realise somepart of the expected return on investment in the early phases of the developmentproject. To express their beliefs and opinions, the stakeholders use an attitudemeasurement. One widely used approach is the summated rating or Likert scale[84]. In the method described here, a similar approach is used, by using a simplescale from 1 to 5: 1 – no value, 2 – little value, 3 – some value, 4 – high value,and 5 – very high value.
• (2) Estimate and assess requirement – This step consists of threeparts: (a) The project manager provides an effort estimate, identifies de-pendencies among the requirements and identifies requirements that mustbe included in the next release. (b) All stakeholders use a five point Lik-ert scale to assess the return-on-value they expect from each requirement.In addition, they write short justifications for their assessment. (c) The
60 CHAPTER 4. RESULTS
(1) Requirement elicitation
(2) Requirement estimation and assessment
(3) Identifying configurations
(4) Configuration assessment
(5) Decision on a configuration
Consensus
Decision made
No
Yes
Start
Figure 4.7: The configuration decision process
project manager collects the assessments from all stakeholders, and calcu-lates the mean value of the assessments for requirement.
• (3) Identify candidate configurations – Identify candidate releasesthat can be implemented within the given time limit. A rule of thumb isthat three to seven candidate releases should be identified. The followingguidelines can be used:
– (a) The requirements are sorted. Two examples of sorting criteriaare: (1) the mean value of all stakeholder assessments for a require-ment, and (2) a frequency sorting, placing requirements with thehighest number of highest scores from the stakeholders first. Re-quirements with an urgency flag set must be included on top of thelist, independent of the sorting criteria.
– (b) Requirements are added to a candidate configuration in thesorted order (step 3a), until the total time estimate on the devel-opment equals the specified time-constraint, or until adding anotherrequirement will exceed this constraint. For each requirement added,all requirements this requirement depends on must be added.
– (c) When not all requirements with the same sorting rank can beadded to a single candidate configuration due to the time-constraint,
4.4. A RELEASE PLANNING METHOD 61
additional candidate configurations can be identified by adding theserequirements to new candidate configurations.
• (4) Assess candidate releases – Assess the candidate releases the sameway as the requirements have been assessed in step 2. Each stakeholderwrites a brief justification of his assessment.
• (5) Decide on final release – Decide which candidate configuration toimplement in the next release. The stakeholders must reach a consensus onthe decision. Thus, even when the assessment of candidate releases endswith a clear ”winner” – a candidate release that has a distinctly betterassessment than all other candidate configurations – all stakeholders haveto agree.
When no consensus emerges easily, the stakeholders will present their writ-ten justification for their release assessment, and share their views and in-terests with the other stakeholders. To find a consensus, a Delphi inspiredfeedback process [65] is used, repeating steps 3 and 4:
– (a) Each stakeholder assesses each release as in step 4 and gives hisresults in writing. The project manager sums up all the assessments(again in writing) and distributes them to all the stakeholders.
– (b) After having read the other stakeholders’ opinions, each stake-holder gives a new assessment, accompanied by a rational explainingwhy his assessment differs from that of one or more of the otherstakeholders. The new assessment is given to the project manager.This step is repeated until a consensus is reached.
An example of the release planning method is found in paper P4 in chapter10. In this research, the release planning method addressed in particular the is-sue of knowledge sharing. This involved both sharing the fragmented knowledgeof stakeholders with different interests, as shown in figure 4.3, and convertingtacit knowledge into explicit knowledge. The release planning method bringstogether all stakeholders with their interests and views, and engages them in acommon decision process. The decision strategy is consensus decision making,which in this method is reached by making each stakeholder take part in creat-ing a common understanding. When no consensus can be reached initially, thestakeholders have to reconsider their opinions and views, and work to create acommon understanding.
Five controlled experiments have been run to test the release planning method.The organisation of the experiments is shown in figure 4.8, and an overview ofthe experiment design is discussed in section 3.3.4. The details and results ofthe experiments are presented in paper P6 (chapter 12) and P7(chapter 13).Here, we give an overview of the results and the lessons learned from using themethod.
All controlled experiments were designed to test if the release planningmethod yields better results than ad-hoc release planning for five topics of in-terest: knowledge sharing, understanding, reaching a consensus, requirement
62 CHAPTER 4. RESULTS
Communication Pattern A
Communication Pattern B
Scenario 1Questionnaire 1
Scenario 2Questionnaire 2a
E 1
E 2.1 E 3.1
H0b
H0a H0a
Scenario 3Questionnaire 2b
E 3.2E 2.2H0b
H0a
Figure 4.8: The relation between the five experiments to evaluate the releaseplanning method
prioritisation and stakeholder satisfaction. The results of experiment E1 weresurprising in that the control groups using ad-hoc release planning performedbetter than the treatment group using the release planning method, for two top-ics: knowledge sharing and understanding. In case of knowledge sharing, thecontrol group performed significantly better. On the other topics, the treatmentgroups performed better, but not significantly. In an attempt to explain theseresults, two explanations were found:
• Learning effect – The first explanation is related to how a new softwaremethod is learned by the users. When a method is used for the first timethe focus of the users is on the method itself and not on the problem thatis to be solved. The users of a method focus on the instructions and onusing the method correctly. When a method is used several times the userswill become familiar with it and have their focus mainly on the problem.This effect has also been observed in experiment E0 [127].
• Communication style – A copy of the requirement list and the releaseform was handed out to all subjects. As a result, the subjects in thetreatment group were using the forms to make their assessments accordingto the release planning method. When they were done, the assessmentswere collected by the project manager with no discussion among the groupmembers (see figure 4.9, part A). Handing out the requirement list and therelease form was done for convenience only. The consequences describedcame as a surprise. In order to stimulate the treatment group to discusstheir opinions and beliefs, only the project manager should receive theforms and facilitate a group discussion, as shown in figure 4.9, part B.
On the basis of these explanations the decision was made to run four moreexperiments – E2.1, E2.2, E3.1 and E3.2 – in order to test these explanations.
Figure 4.9: The effect of introducing a new method
The learning effect was tested in experiment E1–E2.1, and E1–E2.2 , while thechanged communication style was tested in experiments E3.1 and E3.2. The re-sults of these experiments show that there was a learning effect. The treatmentgroups improved their knowledge sharing and understanding, and they did somore than the control groups, where only small differences between the exper-iments could be observed. However, none of the differences were significant atthe 0.1 level.
The changed communication style tested in experiments E3.1 and E3.2 wasintroduced by handing out the requirement assessments and release forms onlyto the project manager. This change made it easier to encourage the groupmembers to explain their opinions to each other, to discuss the issues at handand to share their knowledge. The results of these experiments show that thetreatment groups improved their knowledge sharing and understanding whilethe control groups showed little change. Again, none of the differences aresignificant at the 0.1 level. Still, the results are taken as a strong indicationthat a communication style that encourages a discussion about the stakeholders’beliefs and opinions enables knowledge sharing and understanding.
The changed communication style relies on personal communication andis thus also closer to the observed development practices. As mentioned insection 4.1.3, the communication in web development teams is often informal
64 CHAPTER 4. RESULTS
and through personal conversations. The changed communication style used inexperiments E3.1 and E3.2 therefore seems to fit in with these practices.
The release planning method facilitated a process that enabled and sup-ported knowledge sharing. A consensus decision making style was applied tomake the stakeholders create a common understanding.
4.5 Change process for web application devel-opment
The last three sections have focused on how trade-off practices can be improvedwith respect to creating awareness of trade-offs, qualitative assessment of candi-date options for a trade-off and knowledge sharing between stakeholders. Trade-offs have to be performed throughout the lifetime of web applications. How-ever, web applications change over time as described in section 4.1.2 on lifetimephases, and so do the related trade-offs. Creating awareness of trade-offs, usingqualitative assessments to express tacit knowledge about candidate options for atrade-off, and knowledge sharing are therefore activities that have to take placethroughout the lifetime of a web application in order to respond to the changingcontext of a web application. This research addressed this issue by combiningall three improvements to the trade-off practices in a change process for webapplication development [121]. This process has been described in paper P8 inchapter 14.
This process consists of four phases that can be used together with otherdevelopment methodologies (figure 4.10). The objectives for this process are toexploit the available knowledge, both tacit and explicit, and to enable learningfrom iteration to iteration, by evaluating how decisions were reached, and whatresults they yielded. The four phases are described briefly below:
• (1) Collect data – Create a common understanding of the goals andobjectives for the actual iteration and create an awareness of potentialtrade-offs in this phase. What are the opportunities and risks, and whatcandidate options are available to exploit the opportunities and mitigatethe risks? The data in this phase can be collected using several tools, suchas the trade-off assessment tool described earlier using SWOT analyses,affinity diagram [101] or interviews with relevant stakeholders.
• (2) Analyse data – Assess the consequences of the available candidateoptions, and how they relate to the web application’s objectives and goals.Exploit the stakeholders’ tacit knowledge and use qualitative assessmentswhen no explicit knowledge is available. Several methods can be used forthe analysis, such as the trade-off assessment tool, the qualitative trade-offtool, SWOT analysis, or Impact estimation tables [38].
• (3) Decision making – Decide on the candidate option that best servesthe application’s goal and best fulfils the stakeholders’ interests. Thisinvolves prioritising the candidate options and resolving conflicts that may
4.5. CHANGE PROCESS FOR WEB APPLICATION DEVELOPMENT 65
(2) Analyse data
(3) Make decision
(4) Implement and evaluate
decision
Lessons learned
(1) Collect data
Application objectives and vision
Knowledge base
Figure 4.10: Subprocess for managing changes in the development environment
exist between the stakeholders. The decision style is consensus decisionmaking, and the stakeholders use a similar decision process as with therelease planning method. Thus, each stakeholder’s interest is of equalimportance in the process of making a decision.
• (4) Implement and evaluate decision – Implement the decision andevaluate its outcome with respect to the stakeholders’ interests and theapplication’s objectives. Identify deviations from the expected outcomeand decide whether they can be tolerated or not. When the deviations cannot be tolerated, find root causes of the deviation and corrective actionsfor the deviation.
For every iteration, lessons learned and new knowledge about the develop-ment project that is believed to be of future use, are documented. Analysingthe knowledge behind a decision and the later outcome of such a decision helpsthe development team to identify where improvement is needed. There existmethods and tools to store and refine such knowledge and experience in a for-mal knowledge base. Two possible methods are Experience Factory (EF) fromNASA-SEL [8] and a Wiki (a collaborative authoring tool).
66 CHAPTER 4. RESULTS
Chapter 5
Evaluation
5.1 Research questions revisited
This research focused on the industrial development of web applications and therelated trade-offs. This section revisits the research questions stated in section1.3. The relation between the research questions and the research design wasshown in figure 1.1.
5.1.1 RQ 1: What characterises web application develop-ment?
Answering RQ 1 was the focus of the data collection phase and the analysisphases. This resulted in contribution C1, dealing with industrial web applicationdevelopment practices.
a) Eight Norwegian companies (see figure 3.1 for an overview) were interviewedto answer this research question. In addition we performed a literature studyon this topic. The development practices that were observed are described insection 4.1.3 and are summarised below:
• Small development groups
• Multi-disciplinary development
• Iterative web application life cycle (web gardening [73])
• Informal requirement specification
• Informal testing
• Rush to market
67
68 CHAPTER 5. EVALUATION
b) Web application characteristics that have an impact on web applicationdevelopment practices were identified in the collected data. They are describedin section 4.1.1.
c) Lifetime phases of web applications describe how a web application devel-opment project changes its focus over time. These were found in the analysis ofthe collected data and are described in section 4.1.2.
5.1.2 RQ 2: How are trade-offs performed in web applica-tion development, and how are trade-offs performedthat involve time-to-market as a factor?
RQ 2 was the focus of the data collection phase and the analysis phase. Thework with this research question led to contribution C1, dealing with industrialweb application development practices.
a) Eight Norwegian companies were interviewed to answer this research ques-tion, and two PMA sessions were conducted on the same topic. Trade-off prac-tices in web application development are described in section 4.1.4.
b) Typical trade-off situations involving time-to-market were found in theanalysis of the collected data, and are described (see section 4.1.4).
c) The collected data were analysed to find a trade-off strategy applied by thecompanies interviewed for this research.
5.1.3 RQ 3: How can a long-term trade-off strategy beenabled in web application development?
This research question was dealt with in the analysis phase, construction phaseand the validation phase. The following contributions are a result of the workwith this research question: C2, dealing with a trade-off strategy, C3, dealingwith the qualitative assessment of trade-off options, C4, dealing with knowl-edge sharing, and C5, dealing with knowledge management for web applicationdevelopment.
a) The collected data on the companies’ current trade-off practices and thereported experiences with problematic trade-off situations were analysed andsearched for potential improvements. A trade-off strategy that improves thecurrent trade-off practices was developed and is described in section 4.1.7. Themain elements of this strategy are:
• Creating awareness for trade-offs.
• Identifying and assessing options, using qualitative data.
5.2. WEBSYS GOALS REVISITED 69
• Sharing knowledge between stakeholders.
b) Three trade-off methods to support the trade-off strategy were developed(see section 4.2–4.4):
• Qualitative trade-off tool, to assess candidate options in trade-offs.
• Trade-off assessment tool, to create awareness for trade-offs.
• Release planning method, to facilitate knowledge sharing in a decisionmaking process.
c) A controlled experiment was run to test and evaluate the qualitative trade-off tool and the qualitative assessment of candidate options in trade-offs (seesection 4.2).
d) A process to implement the trade-off strategy was implemented by thechange process for web application development (see section 4.5).
5.1.4 RQ 4: How can knowledge sharing be enabled inweb application development?
Answering RQ 4 was the focus of the construction phase and the validationphase. This resulted in contribution C4, dealing with enabling and facilitatingknowledge sharing.
a) The collected data on web development with related trade-off practices wereanalysed and a need for knowledge sharing in web application development wasidentified.
b) A trade-off method for release planning was developed to facilitate knowl-edge sharing among stakeholders by using a consensus decision making ap-proach.
c) Five controlled experiments with the release planning method were run,comparing how knowledge is shared between stakeholders when using the releaseplanning method and ad-hoc release planning.
5.2 Websys Goals revisited
This research was part of the Websys research project (see section 1.2). TheWebsys research goals are revisited in this section.
70 CHAPTER 5. EVALUATION
5.2.1 Websys main goal
To develop guidelines for the industrial development of timely andreliable Web-based systems. Eight Norwegian companies were interviewedfor this research and none of the companies focused on reliability. The researchtherefore focused on how to perform trade-offs between time-to-market andquality attributes, such as usability and availability. This research developeda trade-off strategy to be used for web application development, together withtools and methods to support trade-offs, and a process to manage change inweb application development.
5.2.2 Subgoals G1–G3
G1: Better understanding of the software process and related tech-nologies on how to make trade-offs between time-to-market (TTM)and quality attributes. Eight Norwegian companies were interviewed forthis research and a literature study was performed to build up an understand-ing of web application development and related trade-offs.
G2: Contribute to better methods for Web-based systems develop-ment where quality attributes and TTM need to be considered to-gether. Four trade-off methods were developed by this research. Togetherthey aim at implementing an improved trade-off strategy.
G3: Dissemination and exchange of the gained knowledge, with astrong international component. This research resulted in ten papers thathave been published and presented at reviewed international workshops andconferences. In addition, an international workshop on qualitative trade-offswas arranged as part of the Websys project (September 2006).
5.3 Contributions
In this section, the contributions of the research are evaluated with respect tothe research questions, the research approach and the research methods used.A discussion of the research approach has been given in section 3.2.5. In thenext section, the contributions are evaluated with respect to the related works.An overview of the contributions and their relation to the research approach isshown in figure 5.1.
5.3.1 C1: Web development practices
Eight Norwegian companies were interviewed, and two of the companies alsodid a PMA. The foci of the interviews and the PMA sessions were how webapplications are developed by these companies and how the related trade-offsare performed. The collected data was analysed, using both Grounded Theory
5.3. CONTRIBUTIONS 71
Con
trib
utio
nR
esea
rch
ques
tion
Res
earc
h ph
ase
Res
earc
h m
etho
dC
omm
ents
C1
Web
Dev
elop
-m
ent p
ract
ices
R1,
R2
Dat
a co
llect
ion
phas
eAn
alys
is p
hase
Inte
rvie
w, P
ostm
orte
m
anal
ysis
, Lite
ratu
re s
tudy
Eigh
t Nor
weg
ian
com
pani
es w
ere
inte
r-vi
ewed
, and
two
PMA
wor
ksho
ps w
ere
cond
ucte
d. T
he fo
cus
was
on
the
com
pa-
nies
’ rea
l wor
ld p
robl
ems
and
thei
r tra
de-o
ff pr
actic
es.
C2
Trad
e-of
f st
rate
gyR
3An
alys
is p
hase
Con
stru
ctio
n ph
ase
Roo
t-cau
se a
naly
sis,
G
roun
ded
theo
ryTh
e tra
de-o
ff pr
actic
es a
nd re
al w
orld
pr
oble
ms
wer
e an
alys
ed a
nd re
sulte
d in
a
trade
-off
stra
tegy
. The
trad
e-of
f stra
tegy
w
as im
plem
ente
d by
the
trade
-off
tool
s pr
ovid
ed b
y th
e re
sear
ch. T
he tr
ade-
off
tool
s w
ere
eval
uate
d ag
ains
t som
e of
the
real
wor
d pr
oble
ms.
C3
Qua
litat
ive
asse
ssm
ent o
f tr
ade-
off o
ptio
ns
R3
Anal
ysis
pha
seC
onst
ruct
ion
phas
eVa
lidat
ion
phas
e
Roo
t-cau
se a
naly
sis
Gro
unde
d th
eory
Expe
rimen
tatio
n
Scal
es fo
r qua
litat
ive
asse
ssm
ent w
ere
de-
sign
ed a
nd te
sted
exp
erim
enta
lly. A
less
on
lear
ned
from
stu
dent
exp
erim
ents
was
to
focu
s on
dis
cuss
ing
and
prio
ritis
atio
n w
hen
usin
g qu
alita
tive
asse
ssm
ents
.
C4
Enab
ling
and
faci
litat
ing
know
-le
dge
shar
ing
R3,
R4
Anal
ysis
pha
seC
onst
ruct
ion
phas
eVa
lidat
ion
phas
e
Gro
unde
d th
eory
Expe
rimen
tatio
nBa
sed
on th
e de
velo
pers
’ exp
erie
nce
a tra
de-o
ff to
ol s
uppo
rting
kno
wle
dge
shar
ing
was
con
stru
cted
to fi
t in
with
the
deve
lop-
men
t pra
ctic
es. T
his
appr
oach
was
then
us
ed to
reac
h a
cons
ensu
s.C
5 K
now
ledg
e m
anag
emen
t for
w
eb a
pplic
atio
n de
velo
p men
t
R3,
R4
Anal
ysis
pha
seC
onst
ruct
ion
phas
eVa
lidat
ion
phas
e
Roo
t cau
se a
naly
sis
Gro
unde
d th
eory
Expe
rimen
tatio
n
Roo
ted
in th
e re
al w
orld
pro
blem
s re
porte
d in
this
rese
arch
, a p
roce
ss to
man
age
the
chan
ging
con
text
of w
eb a
pplic
atio
n w
as
prov
ided
, reu
sing
the
trade
-off
stra
tegy
and
de
sign
obj
ectiv
es fo
r the
trad
e-of
f met
hods
.
Figure 5.1: The contributions and their relation to the research approach andmethods
72 CHAPTER 5. EVALUATION
and Root-Cause Analysis (RCA). In addition, we conducted a literature studyon web development practices. The development practices were characterisedby informal requirement specification and communication, partial testing andmainly tacit knowledge. The related trade-off practices involved time-to-marketand how informal and unawarely performed development practices were.
The chosen research approach focused on an in-depth understanding of webapplication development and related trade-off practices. This was achieved byconducting 12 interviews with eight Norwegian companies. The findings of theinterviews could not be generalised to a larger population, which would havebeen the case with survey research. This research aimed at improving the trade-off practices applied in web application development. Therefore, an in-depthunderstanding of how trade-offs are performed in realistic situations was neededin order to identify improvement opportunities.
5.3.2 C2: Trade-off strategy for web application develop-ment
The data collected from the involved companies revealed the lack of clear trade-off strategies. With the involved stakeholders’ hindsight and the understandingof the trade-off practices built up during the analysis phase, a trade-off strategywas developed. The trade-off strategy was used as a starting point for devel-oping the trade-off methods in this research. The trade-off strategy consistsof steps to create awareness for trade-offs, exploiting tacit knowledge by qual-itative assessment of candidate options of trade-offs, and of knowledge sharingbetween the stakeholders.
The trade-off strategy was the result of the analysis of the collected data andwas thus based on our interpretation of the interviewees’ personal experiences.By selecting other companies we would have collected different data that mighthave yielded different analyses. However, the trade-off strategy concentrated onissues found with more than one company in our sample. As the purpose of thisresearch was to improve trade-off practices, we needed to understand realistictrade-off situations.
5.3.3 C3: Qualitative assessment of trade-off options
Three trade-off methods were developed by this research to support the trade-offstrategy and to help the stakeholders to assess candidate options for a trade-offusing qualitative assessments:
• Qualitative trade-off tool – Compares candidate options to a trade-off withimportant requirements of a web application to qualitatively evaluate ifthey conflict or not.
• Trade-off assessment tool – Evaluates the current status of a web applica-tion and the consequences of candidate options in a decision, to create anawareness of a potential trade-off.
Together with contribution C4 and C5, this contribution implements thetrade-off strategy. The use of qualitative assessments in trade-offs was tested insix controlled student experiments. A discussion of using students as subjectsis given in section 3.3.4. The students had relevant experience from working insmall teams and the results of the experiments can therefore not be discardedas irrelevant. By using controlled experiments to validate the improvements tothe trade-off practices it was not possible to test them in vivo, i.e. in a realisticenvironment.
5.3.4 C4: Enabling and facilitating knowledge sharing
A trade-off method for release planning in web application development wasdeveloped to enable and facilitate knowledge sharing among stakeholders. Thiswas done by using a
• consensus decision style, where all stakeholders’ interests and viewsare of equal importance, and where a common understanding is necessaryfor a decision.
• direct communication style, where the stakeholders address the devel-opment team directly.
The release planning method was tested in five student experiments. The resultsof the experiments led to the direct communication style. The comments onusing students as subjects in C3 apply also here.
5.3.5 C5: Knowledge management for web application de-velopment
A process to manage change in web application development was developed. Itimplements the trade-off strategy and applies the trade-off methods developedby this research. To manage the knowledge of web development projects, theprocess
• guides the collection of data for trade-offs,
• facilitates knowledge sharing among stakeholders,
• exploits tacit knowledge for performing trade-offs,
• evaluates the effects of performed trade-offs, and
• builds up a knowledge base.
Most elements of this contribution were tested in the six experiments thatwere run by this research. The same comments as for contributions C3 andC4 apply here. Building up the knowledge base has not been evaluated bythis research, but there exists some literature with experiences in building upknowledge bases that can be used here [82].
74 CHAPTER 5. EVALUATION
5.4 Contributions in relation to related work
The contributions of this research are evaluated with respect to related works,as reported in chapter 2.
5.4.1 C1: Web development practices
The development practices described in this research support and add to thefindings of the related works cited in section 2.1. This includes practices suchas informal specification of requirements and quality attributes, the changingculture of web application development projects and focus of web applicationsover time, and the iterative nature of web application development. The de-scription of trade-off practices in web application development is, however, notdescribed in the literature on web application development and is thus a novelcontribution.
5.4.2 C2: Trade-off strategy for web application develop-ment
The trade-off strategy developed by this research has three objectives: creatingawareness for trade-offs, using qualitative assessments of candidate options fortrade-off and enabling knowledge sharing. Qualitative assessment of candidateoptions for trade-offs and enabling knowledge sharing are addressed by contri-butions C3 and C4. All trade-off methods developed by this research create anawareness for trade-offs by
• identifying candidate options for trade-offs,
• supporting a discussion and comparison of their consequences, and
• assessing how they relate to the stakeholders’ interests and views.
This is a trade-off approach which is similar to the ATAM method. ATAMdetects trade-off points that can be resolved by the stakeholders. In the case ofthe trade-off methods, conflicts between the candidate options to trade-offs andthe stakeholders’ interests are revealed and can be resolved.
5.4.3 C3: Qualitative assessment of trade-off options
The trade-off methods developed by this research rely on the stakeholders’ tacitknowledge which is expressed qualitatively in the trade-off methods. The trade-off analysis methods discussed in section 2.2 all rely on explicit knowledge.However, while most of them express this knowledge in quantitative forms,some methods, such as the ATAM method, also use qualitative knowledge.
5.5. RELEVANCE OF CONTRIBUTIONS 75
5.4.4 C4: Enabling and facilitating knowledge sharing
Knowledge sharing has been enabled and facilitated by applying a consensusdecision making approach in a release planning method for web applicationdevelopment. Using this decision style within a cooperative social context asa development team has shown to increase the individual understanding morethan in a competitive context.
The release planning method applies a hybrid approach of human intuitionand optimisation. It uses return-on-value as a prioritisation criterion, and sortscandidate releases according to the most beneficial return-on-value rating. Thefinal decision is, however, based on the stakeholders’ communication and negoti-ation. It plans only for the next release as most of the release planning methodsconsidered.
5.4.5 C5: Knowledge management for web application de-velopment
The knowledge management process for web application development is a single-looped learning cycle [4] for web application development. By using the trade-offmethods developed during this research, it exploits the tacit knowledge of thestakeholders in a web application development project, and makes it possible tocompare the expected outcome of trade-offs with the obtained outcomes and totake actions according to the difference.
5.5 Relevance of contributions
The contributions of this thesis have been divided into three groups: Develop-ment practices, Trade-off methods, and Knowledge sharing (see figure 1.2). Wewill discuss the relevance of the contributions to practical use for each groupbriefly.
This work has contributed to the understanding of web application devel-opment practices by describing and analysing the related trade-off practicesand the changing environment in which web applications are developed. In bothcases, this understanding has been necessary to understand the problems withtrade-off situations that the developers reported during the interviews. Basedon this understanding, this research developed trade-off methods that help de-velopers to improve their trade-off practices in web application development.
The contributions to the trade-off methods aim at improving trade-offpractices in web application development. The trade-off strategy developed bythis research addresses what the developers interviewed for this research foundchallenging when performing trade-offs. Therefore, the trade-off methods sup-port awareness of trade-offs and compare the consequences of candidate optionsfor a trade-off by using qualitative knowledge. This is supported by one exper-iment, where a qualitative trade-off method was tested and where the subjects
76 CHAPTER 5. EVALUATION
using this method found they had a better communication quality than thecontrol group using ad-hoc approaches.
The contributions to knowledge sharing help to create a common under-standing among stakeholders that originally have a fragmented knowledge (asshown in part A of figure 4.3). In projects that apply development and trade-offpractices (such as the one described in this thesis), knowledge sharing can besupported by using a consensus decision style when performing trade-offs. Theexperiments with the release planning method also demonstrated that a directcommunication style involving all stakeholders in small development groups hasa positive impact on the groups’ knowledge sharing.
5.6 Validity discussion
The validity of the research results is addressed by discussing the experimentresults in the validation phase. The six controlled experiments that have beenconducted in this research have similar designs, and the validity of the experi-ments are therefore discussed together. According to [113] there are four typesof validity threats – conclusion validity, internal validity, construct validity andexternal validity – and they are discussed shortly below.
Conclusion validity This validity is concerned with the relation betweenthe treatment and the outcome. Two threats to conclusion validity have beenidentified and dealt with: the issue of permissible statistical testing, and themeasurements of the dependent variables.
First, there is the issue of permissible statistical testing. A detailed dis-cussion of this issue has been given in section 3.3.4, and only a brief summaryis given here. There is some discussion on whether the data collected using afive-point Likert scale is of interval scale, and if it is permissible to use the t-testin the statistical analysis. In this research it was seen as good practice to useparametric tests such as the t-test on this kind of data. However, in additionto the t-test we used the Wilcoxon test. This is a non-parametric test, that canbe used on data of type ordinal scale. The results are the same, except thatthe t-test is stronger, and more questions yield significant differences with thet-test than with the Wilcoxon test.
The second threat to conclusion validity is the reliability of the questionnaireapplied to measure the attitude of the subjects. The dependent variables in theexperiments are measured indirectly by using a five-point Likert scale. Themain question here is if the attitude measurements really measure what theyare supposed to measure. This topic is related to the more general discussionof measurement validity [2]. The statements in the post-experiment question-naire that were presented to the subjects can be understood differently by thesubjects than by the researchers. To ensure that the attitude measurementsin the questionnaire really measure the dependent variables of the controlledexperiments, we have taken the following actions in the design of the releaseplanning experiments:
5.6. VALIDITY DISCUSSION 77
• Pilot experiment – The questionnaire was tested in a pilot experiment.The participants of the pilot experiment were fellow researchers. Theygave valuable feedback on how they understood the statements, and somestatements were rephrased to make them clearer.
• Multiple statements for each dependent variable – For each depen-dent variable several statements were formulated. This allowed testing forvariance among the statements and to filter out ”outliers”.
• Post experiment interview – After the last experiment we invited afocus group of subjects and interviewed them about the experiments, in-cluding questions about the questionnaire.
Based on these actions several statements were rephrased to increase theirclarity. After the last round of experiments several statements were not con-sidered in the analysis since they were identified as ”outliers”, i.e. they wereanswered differently than the other statements for the same dependent variable.This occurred due to the change from positive to negative loaded statements.The interview with the focus group revealed that there were no problems orambiguities in rating the statements. We are therefore confident that the ques-tionnaire measured what we intended to measure.
Internal validity The internal validity is concerned with the causal relation-ship between treatment and outcome. A threat to internal validity is thus anuncontrolled factor that contributes to the outcome. Two threats to internalvalidity have been identified: the use of ad-hoc methods in the control groupand the motivation of the subjects.
A threat to the internal validity is the use of an ad-hoc release planningpractice in the control group. By choosing an ad-hoc practice we have no controlover the method or practice being used to solve the release planning task in thecontrol group. This introduces an uncontrollable factor that may affect theoutcome. In the case of the experiments on release planning in this research, itwas unlikely that the student subjects were experienced in any release planningmethod. They would therefore try to solve the given task without the supportof a release planning method and find the best possible solutions they couldthink of. Compared to more experienced subjects, such as IT professionals,this could pose a threat to internal validity. Further, even if some studentswere experienced in using release planning methods, it is rather unlikely that allstudents would have this experience. If this were the case, the results would showthat the groups with the more experienced students would yield different results.These kinds of variations in the results have, however, not been observed. It isthus fair to assume that no threat to the internal validity was present.
A second threat to internal validity is the motivation of the subjects. De-pending on their motivation the subjects could perform differently in the taskthey have to solve. The motivation would thus be a factor that is not controlledor measured but that influences the outcome of the task. Issues that may havean influence on the subjects motivation are if the subjects are working on their
78 CHAPTER 5. EVALUATION
own artefacts [47], or an expectation of a benefit in the teacher-student rela-tionship. In the case of our controlled experiments we used students as subjectsthat had no prior knowledge of the experiment materials. None of the studentshad a teacher-student relationship with any of the researchers. For the releaseplanning experiments, half of the students were recruited from a different fac-ulty.
Construct validity This validity is concerned with the relation between the-ory and observation. The limited training of the subjects in the release planningmethod may pose a threat to the construct validity. This issue has already beenobserved in the analysis of the first experiment, as the treatment groups’ perfor-mance in knowledge sharing and understanding was not as expected and lowerthan the control groups’ performance. As seen in the experiments on the re-lease planning method, using the same method repeatably on the same subjectschanges this.
External validity This validity is concerned with the generalisation of theexperiment to a scope outside the study. One threat to external validity has beenidentified. The use of students as subjects poses a threat to the external validitywhen generalising the results to a population outside the experiment setting. Adetailed discussion of this issue is given in section 3.3.4. This threat is takenseriously in this research and the results are not generalised to a population otherthan students. However, the difference in behaviour between IT professionalsand students may not be as big as many believe. An IT professional’s experienceis limited and there is no guarantee that IT professionals are more experiencedin release planning than students. Without an investigation of all subjects’backgrounds – both IT professionals’ and students’ – nothing can be said aboutthe difference in experience.
Chapter 6
Conclusion and future work
In this final chapter we give a short summary of this thesis and give somedirections for future work.
6.1 Conclusion
This thesis has studied web application development practices and the relatedtrade-off practices. Four research questions RQ1–RQ4 were formulated andanswered by this work (see section 1.3).
RQ1 and RQ2 dealt with what characterises web application developmentand the related trade-off practices. To answer both questions, we intervieweddevelopers from eight Norwegian companies and performed two postmortemanalyses with the development teams of two projects run by these companies.We found that web applications are developed with small development teams,with short development cycles and many releases. Functional requirements andquality attributes are specified informally, and are in many cases anecdotal.Their implementation depends on the developers domain knowledge and un-derstanding. The developers also maintain the applications and receive directfeedback from end-users. Trade-offs are in many cases performed unawarely andinformally. Trade-offs that involve time-to-market are made with respect to thedevelopment practices such as requirements, testing, and release planning.
RQ3 addressed how trade-off practices can be improved. This resulted in astrategy for trade-offs, that consists of three parts: (1) creating an awarenessfor trade-offs, (2) qualitative assessment of candidate options in a trade-off, and(3) enabling knowledge sharing. Awareness for trade-offs is created by assessingdecision situations in order to detect opportunities and risks that need to bebalanced. Qualitative assessment of candidate options in trade-offs are used toexploit the stakeholders’ tacit knowledge, by expressing the stakeholders’ beliefs,opinions, and expert judgements. Three trade-off methods that use qualitativeassessment were developed through this research. Knowledge sharing is dealtwith by RQ 4. It is enabled by facilitating a direct communication style between
79
80 CHAPTER 6. CONCLUSION AND FUTURE WORK
the stakeholders and by applying a consensus decision making approach. Thetrade-off methods developed by this research are:
• Qualitative trade-off tool – Uses a QFD inspired matrix to evaluate can-didate options for a trade-off and to allow a comparison between them.
• Trade-off assessment – Uses SWOT analyses on candidate options for adecision to detect trade-off opportunities.
• Release planning for web application development – A release planningtool, that uses a qualitative assessment of the expected return-on-valuefor each candidate release and that facilitates a consensus decision makingapproach.
RQ 4 dealt with how knowledge sharing can be enabled and supported. Inthis research, two approaches were used to support knowledge sharing:
• Consensus decision making approach – To reach a consensus, the stake-holders have to understand the interests and goals of the other stakeholdersin order to find a solution that all can agree on.
• Communication style – Stakeholders have to meet directly and presenttheir views and interests to each other.
Both approaches were applied in the release planning method for web applicationdevelopment.
Six controlled experiments were run to test both the qualitative assessmentof candidate options to a trade-off and the consensus based decision approach.The results indicate that communication quality and knowledge sharing aresupported by the trade-off methods. However, not all results are statisticallysignificant and the results should therefore not be generalised to a larger pop-ulation. Also, the use of students as subjects prevents the results from beinggeneralised to the population of industrial web application developers.
6.2 Future work
The research in this PhD thesis has given answers to the research questions, buthas at the same time raised a number of new questions that should be addressedin future work. The questions are organised into the two categories shown insection 6.2.1 and 6.2.2.
6.2.1 Web application development trade-offs
The knowledge needed to understand, facilitate and improve trade-offs in webapplication development is by no means complete. More can be learned abouttrade-offs and about the factors that influence trade-offs. There are three par-ticular areas where trade-offs should be studied in more detail: other trade-offsituations than the one covered in this work, multi-attribute trade-offs, andtrade-offs involving time.
6.2. FUTURE WORK 81
Trade-off situations The work in this thesis has covered trade-offs involvedin release planning decisions and understanding how trade-offs are performed.However, trade-offs are found in other decision situations as well and under-standing them better helps to improve the way they are performed and how theycan be facilitated in a typical web application development context. Trade-offsinvolving testing and requirement specification are examples of trade-offs thatare interesting for web application development.
Multi-attribute trade-offs The trade-off tools presented in this paper havemostly been two-dimensional since two factors have been balanced (such astime-to-market vs. return-on-value in a release). The reality, however, is multi-dimensional, and in general more than two factors must be taken into considera-tion when trade-offs are performed. Given the development practices describedin this thesis, how can multi-attribute trade-offs be performed with such infor-mal development practices and tacit knowledge?
Longitudinal study of trade-offs Web applications that are developed witha strong emphasis on time-to-market perform many trade-offs ”on-the-fly”. De-cisions have to be taken under strict time constraints, with uncertain and infor-mal knowledge, and within a changing context. Studying trade-offs with specialfocus on the time dimension will add two perspectives to the knowledge abouthow trade-offs can be improved. These perspective are (1) studying the longterm consequences of trade-offs and how long term consequences can be usedas additional knowledge when performing a trade-off, and (2) the change oftrade-off practices over time. We assume that as a web application becomesmore mature there will be a gradual shift to more formal practices with certainand quantifiable knowledge, and that the most important factor for success willshift from time-to-market to fulfilling end-users’ expectations.
6.2.2 Knowledge sharing
Knowledge sharing turned out to be an important factor in decision making withsubjective and uncertain knowledge. Future research directions should involve abetter understanding of knowledge sharing in this type of development projects,how decision methods and guidelines can facilitate knowledge sharing, and howthe effect of these methods can be compared and measured.
Understanding knowledge sharing in web application developmentteams Understanding knowledge sharing in web application development ispart of a strategy to improve the decision making in web development. Theknowledge involved in web application development is tacit and fragmentedbetween the stakeholders, and knowledge sharing is needed to create a commonunderstanding among the stakeholders. Knowledge sharing is, however, not astraightforward activity. As shown in the work of Sui et al. [104], knowledgesharing depends on the motivation of individuals, and can be supported by a
82 CHAPTER 6. CONCLUSION AND FUTURE WORK
culture of knowledge sharing. However, given the differences in the type oforganisations – larger enterprises vs. small development teams – the resultsof Sui et al.’s work are not easily transferred to web application development.We need to know more about what motivates developers in web applicationdevelopment to share knowledge, and what knowledge needs to be shared inorder to perform balanced trade-offs.
Supporting knowledge sharing in small development teams Anotherissue is how knowledge sharing can be enabled and supported in small develop-ment teams. We have some experience with knowledge sharing in the experi-ments on the release planning method. Can knowledge sharing be improved inother ways than the approaches used in the release planning method? Relatedto this issue is the question of what knowledge needs to be shared. Facilitatingsharing too much or the wrong type of knowledge may hamper the decisionprocess [36]. These issues need to be balanced, as the purpose for doing this isto improve the available knowledge for, and the quality of, decisions.
Measuring knowledge sharing In this research knowledge sharing was mea-sured with a post-experiment questionnaire, using attitude measurement on is-sues such as knowledge sharing and understanding. Using this questionnairemade it possible to compare the effect that two release planning methods hadon knowledge sharing. Better and more fine tuned measurements of knowledgesharing are needed for future research on knowledge sharing in small and in-formal software development teams. What factors are dependent on knowledgesharing and can function as an indirect measurement of knowledge sharing?
Part II
Papers
83
Chapter 7
Paper 1
Sven Ziemer and Tor Stalhane:
The use of Trade-offs in the Development of Web Applications.In Proc. First International Workshop on Web Quality (WebQuality 04) at4th International Conference on Web Engineering (ICWE 2004), 27-28 July2004, Munich, pp. 269-281. Also printed in Maristelle Matera and Sara Comai(Eds.): ”Engineering Advanced Web Applications”, In Proc. Workshops inConnection with the 4th International Conference on Web Engineering (ICWE2004), Munich, Germany, 28-30 July, 2004. Published by Rinton Press, ISBN1-58949-046-0.
Here, the version published in ”Engineering Advanced Web Applications” ispublished.
85
86 CHAPTER 7. PAPER 1
1
The use of Trade-offs in the development of Web Applications
The development of Web Applications has some special characteristics, such as very short Time-to-Market, parallel development and requirements that are not prioritized. This makes it hard to address quality attributes in a systematic way, to prioritize requirements and to resolve conflicts between them. The use of trade-off methods such as QFD will help to prioritize the “right” requirements and to resolve conflicts between them.
1 Introduction
In recent years, the World Wide Web has become a popular platform for software application development. One of the reasons for this is the ease of software deployment. When the software has been installed or updated on a web server, it is available for all users within seconds. Today, we are using Web Applications in many domains, and are depending on the proper function of these applications. The same is true for many companies who are using their Web Application in an important part of their business.
It is therefore a growing interest in how Web Applications are developed, and in the development practices used. Especially, there is an interest in how quality attributes are addressed during the development of Web Applications [13, 14]. WebSys is a research project whose main goal it is to develop high-quality research competence and concrete guidelines for industrial development of timely and reliable Web-based systems.
The most important factors for the WebSys project are Time-to-Market, reliability and robustness. These factors can have a contradicting influence on each other. E.g., spending time to build a more reliable and robust system will increase the Time-to-Market. Therefore, these factors have to be balanced in a controlled way, making it possible to assess both the delivery time and how reliable and robust the system will be.
This paper presents how Trade-off can be used to resolve conflicts that stem from contradicting requirements, conflicting expectations from the stakeholders, and from the time-to-market requirements. We also show how the Quality Function Deployment (QFD) method is a suitable method for performing trade-offs.
The rest of this paper is organized as follows: Section 2 describes Web Engi-neering Practices. Section 3 introduces how trade-offs can be used in Web Develop-
87
2
ment, and section 4 shows how Quality Function Deployment can be used in this context. A discussion is given in section 5 and conclusions are given in section 6.
2 Web Engineering Practice
Web Applications are applications that offer contents and services that can be used by a Web browser. This opens for a wide range of Web Applications. There exists several categories of Web Applications [16, 5, 9, and 3], organizations can have different perceptions on Web Application Development [5], and they use Web Applications for different purposes. This makes it difficult to give a description of Web Application Development that matches every Web Development project.
Based on some surveys for Web Application Development |2, 10, 11 and 17], and on a number of interviews conducted in the WebSys project with Norwegian companies, we will describe some development practices which are special for Web Development. This does not mean that these practices are not found in other software development projects, but practices are found in a more extreme version in Web Application Development.
2.1 Rush to market
Web Development projects often have a short Time-to-Market (TTM) requirement. In some cases the development cycle for Web Applications is only some weeks. The short Time-to-Market is maybe the most important characteristic of web application development. All the development practices mentioned later are related to this one. In order to meet tight TTM requirements, development teams uses both evolutionary and parallel development. There may be other reasons for the use of ad hoc development processes (e.g. differences in development culture), but the force of the rush to market is a prominent reason.
An interesting question is, what makes this TTM requirement so important. In our opinion, this lies outside the World Wide Web as such. It is rather a factor stemming from economics and marketing. It is about opportunities that can not be missed, and it is about presence in a competitive marked environment. This is happening when a company is focusing on the short term benefit only, such as accepting to deploy software with poor quality just to keep a market opportunity or to avoid that a competitor gets an advantage. It is also about reducing the economical risks. A small investment is made to implement a new function. This function will be evaluated by the internet users. If the investment pays off – meaning that this functionality gets repeated visits from Internet user – a new, and may be larger, investment can be made to improve the quality of this function.
In order to meet the TTM requirement, development teams are using some of the following practices.
88 CHAPTER 7. PAPER 1
3
– Evolutionary development – The Web Application can be delivered in successive releases. For each release, new functionality will be added, and existing functionality can be improved. This is sometimes called Web Gardening [12]. New releases of the Web Application are deployed at a fixed time interval (e.g. every second week). As a consequence of this, to keep the deadline for each release is more important than the content of a new release. Functionality that was expected to be part of one release can easily be assigned to a later release. It is accepted that software is developed without addressing quality issues in the way they should and that this software is released. When problems with quality issues are discovered, the software can be improved in one of the future releases. In this way, the current quality of software becomes less important.
– Parallel development – To meet the tight TTM requirements, development teams have to reduce the time for each development cycle. This is done by parallel development. In one iteration a developer can work on several releases. He can work on the implementation for the next release and on the planning for a future release.
– Ad hoc development process – Web projects often uses an ad hoc development process. The development teams depend on the skills of the most experienced developers in the team. The development of Web Applications lacks the rigour and systematic approach, quality control and assurance.
– Requirements elicitation – The requirements for Web Applications are evolving all the time. Prototypes are used to communicate with the customer, and they are also used to validate and refine the requirements.
The development practices described above cause a number of problems and makes it difficult to use traditional methods for quality control and assurance. The competitive environment of the Web and the rush-to-market makes it difficult to keep a long-time focus. Hence, quality issues will be addressed later in the lifecycle than is the case for traditional software development. The way requirements are dealt with makes it hard to prioritize among the requirements, and using ad hoc processes results in lack of a systematic approach for Web Application engineering and a dependence on the skills and experience of individual developers. Using traditional methods for quality control takes “too much time” [17], they assume that a proper requirement specification has been produced, and that there will be both analysis and design activities. This is not the case, as web applications are developed without a proper specification of requirements, coding starts in some cases even before the requirements are stable, and there are no analysis and design activities.
What is needed are not only methods for quality control and assurance that will work with these development practices. There is also a need for a method that will
89
4
help developers to prioritize the “right” requirements and to balance between conflicting requirements. This method must fit into the Web Development practices described here and be feasible for small project teams, since many Web Applications are developed by small project teams.
3 Trade-offs and Web Engineering
In the WebSys project we are studying trade-off situations that can occur in a Web Development project. A trade-off situation consists of several elements. First, there are a number of requirements, which possibly are in conflict with each other. Second, we need an analysis of the advantages and disadvantages of the proposed requirements and the costs needed to meet them. Finally, a decision based on the identified advantages and disadvantages has to be made, balancing them to yield the best result possible.
One trade-off situation for Web Engineering projects is the conflict between Time-to-Market and high quality. Both requirements are important, and we have to make a trade-off between “sooner but worse” and “later but better”. A web application can loose users if new functionality is first introduced on a competitors Web application. On the other hand, the only mechanism that brings users back to Web sites is high quality [13].
By using trade-offs we want to give small software projects a simple way to have an efficient decision making process. This is an important criteria for us, since we want to use methods that are well known, and easy and intuitive to understand and use.
3.1 The nature of Trade-offs
The use of trade-offs will be helpful for Web Development projects with respect to the aforementioned problems. Trade-offs help to create awareness of the conflicts between contradicting requirements, success factors and goals for a project. A trade-off shows the conflicts and the possible solutions that can be used to resolve them. The consequences of the solutions are also shown. Thus, using a trade-off method helps the project to find the best available solution.
This also makes a trade-off a tool for negotiation between stakeholders. The solution to the problem of conflicting requirements and interests from different stakeholders is usually one that make the most stakeholders satisfied. Using a trade-off is helpful in this kind of negotiation because all parties can see the consequences of their choices, and use this to find the proper balance.
Trade-offs can help to find a balance between the short and long term benefits in a development project. In the end the preferred balance depends on the decision-
90 CHAPTER 7. PAPER 1
5
makers. The trade-off method should show were the conflict between short and long term benefits is, and how these two views can be balanced in a compromise.
Trade-offs help a project team to focus on a common goal, since the trade-off analysis described above is a group activity. The pros and cons of conflicting requirement and solutions are discussed in the trade-off situation. Thus, all the available information is shared among the project team members, improving the communication in the development team and between all the stakeholders.
All trade-offs imply changing one or more goals in the project. In order to do this, we need to make the goal measurable – either directly or indirectly – and relate it to our experience and state of the practice. There are several ways to make the goal measurable. In our opinion, the GQM process [18] is the most efficient way to achieve this. To relate the goal to experience and state of the practice we need three pieces of information:
– What is our standard, average performance related to the goal
– What is the best we have ever done?
– What is – to our knowledge – the best that is done worldwide?
The purpose of this exercise is to highlight the relationship between the goals and the resources and time necessary to achieve it. The message should be clear: if the trade-off process requires us to go beyond the best we have ever done, the probability of achieving this goal is small. By knowing our own limits, based on experience, we can assure that we keep our project goals realistic.
An important parameter to consider during trade-off is the degrees of freedom available at the current stage of the project. The opportunities for making trade-offs will decrease over time as we make more decisions. At the same time, our understanding of the product, the customer's need and so on increases over time. The paradox is that when we deliver the product, we have full understanding but can not change anything. There is a way out of this – we need to increase our understanding from the start so that we have some information to base the early trade-offs on. This can be general knowledge or company-specific knowledge.
3.2 Five questions about trade-offs
The discussion related to the five questions following below will give us a better understanding of how we want to use trade-offs in a development process.
– When to start a trade-off? Trade-offs can be started in two settings. Either the trade-off is scheduled or there are situational circumstances that triggers the need for a trade-off. Trade-offs can be planed for situations we know from our experience are critical for the development project. They can also be triggered
91
6
when the project team runs into problems and conflicts that can not be easily resolved. %This is the case when the consequences of some decision is unclear.
– What are the prerequisites for using trade-offs successful? A clear understanding of the conflict or problem is needed, in addition to a list of alternatives or solutions, and the consequences attached to them. This helps us to assess the conflict at hand, to assess the consequences of possible solutions and to choose among the solutions with respect to the desired outcome. It is important to look at solutions that can create a consensus.
– How to perform a trade-off? For scheduled trade-offs this is straight forward. A member of the project team is assigned as responsible for this trade-off and have to prepare for it (see question on prerequisites for trade-offs). In case of non-scheduled trade-offs this is more difficult. As soon as somebody involved in the project discovers a conflict that he can not resolve himself, he should prepare for a trade-off meeting and be responsible for that trade-off. The trade-off should be performed as a group activity, with the responsible team member acting as a moderator. We want the trade-off meeting to be as informal as possible, so that it can fit into the development practices.
– What are the results of a trade-off? A trade-off should result in a decision on what actions to take, and how to resolve the conflict at hand. An important thing about a trade-off is that the project team should have the same focus and at least a common understanding of the rational for the decision made or proposed. In some cases it may not be possible to find a decision, but rather a list of alternative decisions, with the final decision to be taken at some later time. The decision can be made by the same group when the project has gathered more knowledge about the unresolved conflict.
– How many persons can participate in a trade-off? There is a limit on the number of persons that can participate in trade-offs as they are described here. We believe that the upper limit is 10 persons, and that the optimal number of persons is four. The reason for keeping the number low is to avoid making trade-offs a formal event. By keeping trade-offs informal they can be used as a way to help speed up the decision process.
The objective for trade-offs in the WebSys project is to find a simple way for the projects to clarify their goal and problems, to support a constructive communication about them, to help to gather all available information about them and to reach a decision based on consensus.
4 The use of Quality Function Deployment (QFD)
The WebSys project has no intention of solving the trade-off problems once and for all. The problems and the spread of stakeholders competence are too diverse for
92 CHAPTER 7. PAPER 1
7
that. Instead, we have put together a tool box that developers and project managers can use when they need to perform a trade-off [20]. When selecting trade-off tools for our tool box, we decided that our tools must fulfil the following requirements:
– Be easy to understand for all parties involved. – Include both qualitative and quantitative information. – Be applicable in all activities in a web project. – Be applicable with all development methods and process models.
Quality Function Deployment – QFD for short – is one of the methods in our toolbox. We find it easy to learn and use, and is well suited for the trade-off situations described above, because of its flexibility. This method is described in [1], see also [7]. The following is a description and discussion of the Quality Function Deployment method, and a discussion of how it can be used in Web Development projects.
4.1 Quality Function Deployment
QFD is a method for answering important questions during the requirement analysis, architectural design, technological assessment and implementation planning. We will here only consider the requirements phase, where QFD looks at the three important questions: what, how and how much.
– What – what shall we implement? This is pertaining to the customer's requirements.
– How – what method, technique etc. shall we use to realize each requirement? – How much – how much resources shall we put into meeting the requirements?
This will depend on the priority of each requirement and on how good these requirements are met by our competitors.
Traditionally, the QFD matrix uses three symbols to indicate the relationships between the “whats” and the “hows” (strong correlation, weak correlation and conflicting relation). In order to use QFD in trade-off situations, we are using numerical values directly into the matrix for the following situations: – 9 points, strong positive relation – 3 points, weak positive relation – 0 points, no relation – Corresponding for negative relations.
Each tool, method and function – the “hows” – get a score that is the sum of the products of its relationship to each requirement. All the information can be shown in a matrix (see figure 1). This method has much in common with the Multiple Attribute Evaluation method [8] used in economy.
93
8
4.2 QFD for Web Development
There are several factors that make the QFD well suited for trade-offs:
– The data needed – requirement priorities and our current level for these requirements compared to our most important competitor(s). By requiring that these data must be available, we have taken a major step towards a sound basis for trade-offs.
– The importance and difficulty scores, which gives us a good idea of the work needed when using each tool or method and for implementing each function.
– How we relate to our main competitors both with current and new versions of our products.
– The description of how each tool, method and implemented function contributes to the delivered product. There are two important pieces of information here; we have to make sure that each line in the table have at least one marked column and the strength of the relationships the “hows”.
The main importance of the information shown in the QFD matrix is that it improves the decision process by showing the stakeholders what they can manipulate and the consequences of their manipulations.
Trade-off problems in web development can often be solved by using QFD. Also, the use of QFD can support the requirement elicitation of web projects. When using QFD the way we suggest, every stakeholder can rate the importance of each requirement, which will reveal potential conflicts among the stakeholders. The tool and methods available to fulfil the requirements can be rated according to how they influence the intended requirements and how they influence other requirements. The requirements can be compared to the requirements from a competitor. The whole process will help us to exclude unrealistic requirements, assign a low priority to nice-to-have requirements and to identify the most important requirements.
When conflicts are discovered, the development team can identify advantages and disadvantages, and attach weights to each requirement. This will improve the communication inside the team, and with the stakeholders. All of this can be done without the use of formalized methods, by using ranking and weights in the QFD matrix.
94 CHAPTER 7. PAPER 1
9
Figure 1. QFD Matrix for method 1 (top) and QFD Matrix for method 2.
In order to show how this will work, we look at a small scenario taken from a company we conducted an interview with: The marketing director of a company wants to include some dynamic effects into a Web Application to make it more attractive to the main user group. The maintenance group on the other hand wants to achieve a high performance and to test this in advance. Also, short TTM, and the requirement “run on all available web browsers” are important requirements for the next version of the Web Application. In this situation a couple of “conflicts” have to be resolved:
– There is a value conflict between a marketing issue (dynamic effects) and maintenance (high performance), which can be seen from the different ratings these stakeholders (marketing = stakeholder A and maintenance = stakeholder C in fig) associate with these requirements. By comparing the weights of these requirements with two competitors (Product 1 and Product 2), a compromise can
95
10
be reached, which makes this the best product when comparing it with the competitors.
– Two different methods are available to implement the dynamic effects and there is a possible conflict on which one to use. Method 1 will result in the best dynamic effect (indicated by a “9” in the first QFD matrix), whereas Method 2 will give a not so good, but still acceptable effect (indicated with a “3” in the second QFD matrix). When analyzing the consequences of both methods with respect to the other requirements, it becomes clear that Method 1 will have a negative influence on all other requirements. In this case the decision is to use Method 2.
– In order to save time, the development team considered removing the requirement “run on all available web browsers”. This will remove the need to comply with the browser checklist, and to perform the Browser Test. However, the QFD matrix also shows us that we at the present are trailing way behind our most important competitors – Product 1 and Product 2 – in this matter. The new version was intended to bring us closer to our most important competitor – Product 1. Since the competitors are considered to be better on the "run on all browsers" requirement, but about the same level for TTM, the team decides to keep the requirement.
5 Discussion and future work
The aim of this approach is to help reducing Time-to-Market of Web Applications and at the same time improving the quality by using the information that is available in a given situation. This is done by assigning subjective measures to the requirements and the means to fulfil these requirements. Our understanding of the Web Development practices are that it is not feasible to use objective measures in most cases, simply because there is not enough time.
The use of subjective evaluation rises the question of how reliable the collected data are [21], and how this evaluation is done. We will discuss this in some detail. The method described above is dependent on expert opinions and will be no better than the experiences available from the persons participating in the trade-off process. An open question is how to represent the collected information. We chose to use a Likert scale with 5 values (strong positive relationship, weak positive relation, no relation, weak negative relation and strong negative relation). People with a mathematical or statistical background often have uneasy feelings about doing math on values from a Likert scale. Whether this is permissible or not has been debated for more than 50 years and the debate is still raging between the persons pure at hearth and the persons with a more pragmatic view. A good summary and discussion of the positions of the parties involved can be found in [6], pg. 150 – 154, which also contains a large number of references. In this short paper, however, we will only
96 CHAPTER 7. PAPER 1
11
note two points –- 1) It has been working for QFD in an industrial setting for more than 40 years and 2) several active researchers have found it permissible to use statistical methods on values from a Likert scale, e.g. [4].
The heavy reliance on experiences poses a risk and an opportunity. The risk of using subjective assessments and experiences stems form the uncertainty connected to the use of these values. There are two ways to reduce this risk – collect more information or improve the experiences over time. The first solution implies that we need more time and resources, thus increasing the time to market for the product. Usually this is not a practical solution. Thus, we are left with improving our experiences over time. There are several ways to achieve this and we will discuss two of them:
– Collect real data post festum. This will give us the possibility to compare at least some of the subjective data that we used with the later, observed results.
– Do a post mortem analysis [15] after the project is finished and the consequences of our decisions can be observed. Important questions that can be answered in this way are for instance: o Are we satisfied with the decisions we made? o What should have been done different? o How will we do this the next time the need arises?
In both cases, we can learn from our successes and failures and improve our decisions over time. As a starting point we can organize our experiences in a checklist but as the amount of information increases, we will probably need some sort of database. If the company already has a general experience repository and an electronic process guide [19], this could be extended with a separate trade-off process and a set of experiences collected under this process.
So far, we have performed interviews with a number of companies to identify trade-off situations that these companies encounter during the development of Web Applications and to study how they are resolved. The next step in our work is to perform case studies with companies using QFD to perform trade-offs. In addition, we will perform student experiments in a software engineering course. The goal of the experiments will be to see if student groups using QFD to perform trade-offs can make more informed decision given the short amount of time available.
6 Conclusions
The development of web application is different from traditional software development. Among the most important differences is the strong Time-to-Marked requirement, which makes it difficult to address quality attributes in a proper and
97
12
systematic way. The development practices identified for Web development result in problems such as the lack of requirement prioritization, no long-time focus and no systematic development approach with respect to quality attributes.
In the WebSys project we have looked at the decision making process with regard to functional and non-functional requirements, and how conflicts between them can be resolved by using trade-off methods. The method we are proposing is Quality Function Deployment.
We find the use of trade-offs helpful for several reasons: trade-offs can create awareness about conflicting requirements, they can help to negotiate how to resolve conflicts, and they can help us to reach a consensus on the projects goal.
98 CHAPTER 7. PAPER 1
13
References
1. Y. Akao. Quality Function Deployment. Productivity Press, Cambridge, MA, 1990.
2. L. Baresi, S. Morasca and P. Paolini. An Empirical Study on the Design Effort of Web Applications. In Proceedings of the 3rd International conference on Web Information Systems Engineering (WISE 2002), 2002.
3. J. Conallen. Building Web Applications with UML, Second Edition. Addison-Wesley, 2003.
4. D. Davis. Business Research for Decision Making. Duxburry Press, 1996.
5. Y. Deshpande, S. Murugesan, A. Ginige, S. Hansen D. Schwabe, M. Gaedke and B. White. Web Engineering. Journal of Web Engineering, 1(1):3–17, October 2002.
6. T. Dybå. Enabling Software Process Improvement: An Investigation of the Importance of Organizational Issues. PhD thesis. NTNU, 2001. IDI Report 7/01.
7. R. B. Grady. Practical Software Metrics for Project Management and Process Improvement. Prentice Hall PTR, 1992.
8. C.-L. Hwang and K. Yoon. Multiple Attribute Decision Making – Methods and Applications. Springer Verlag, Berlin, Heidelberg, 1981.
9. G. Kappel, B. Pröll, S. Reich and W. Retschitzegger. Web Engineering. dpunkt.verlag, 2004.
10. A. McDonald and R. Welland. A Survey of Web Engineering in Practice. Technical Report TR-2001-79, Department of Computing Science, University of Glasgow, 2001.
11. A. McDonlad and R. Welland. Web Engineering in Practice. In Proceedings of the fourth WWW10 Workshop on WebEngineering, pages 21–30, 2001.
12. S. Murugesan, Y. Deshpande, S. Hansen and A. Ginige A New Discipline for Web-Based System Development. In Proceedings of the Workshop on Web Engineering, ICSE99, 1999.
13. J. Offutt. Quality Attributes of Web Software Applications. IEEE Software 19(2):25–32, 2002.
14. L. Olsina G. Lafuente and O. Pastor. Toward a Reusable Repository for Web Metrics. Journal of Web Engineering, 1(1):61–73, 2002.
99
14
15. A. Pirk, T. Dingsoyr and T. Stålhane. Post Mortem: Never leave a project without it. IEEE Software, 19(3):43–45, 2002.
17. A. Ramesh, J. Pries-Heje and R. Baskerville. Internet Software Engineering: A Different Class of Processes. Annals of Software Engineering, 14(1-4):169–195, 2002.
18. R. van Solingen and E. Berghout. The Goal/Question/Metric Method. McGraw-Hill, 1999.
19. T. Stålhane and L. Scott. Experience Repositories and the Postmortem. In Workshop „Learning Software Organisations" (LSO 2003), 2003.
20. T. Stålhane and S. Ziemer. A Trade-off toolbox. Technical Report, Department of Computer and Information Science, 2003.
21. A. Wohlin and A. A. Andrews. Assessing Project Success Using Subjective Evaluation Factors. Software Quality Journal. 9():43–70, 2001.
100 CHAPTER 7. PAPER 1
Chapter 8
Paper 2
Sven Ziemer, Tor Stalhane and Magnar Sveen:Trade-off Analysis in Web DevelopmentInternational Conference on Software Engineering, 3-WoSQ: Proceedings of thethird workshop on Software quality, pp. 70–75, ACM Press, 2005. In Xiaoqing(Frank) Liu, Yan Sun, Gautam Kane, Yuji Kyoya, and Kunio Noguchi (Eds.):Proc. Third Workshop on Software Quality, held at the 27th InternationalConference on Software Engineering (ICSE’05), St Louis, Missouri, May 17,2005. Published by ACM, ISBN 1-59593-122-8, pp. 70–75.
101
102 CHAPTER 8. PAPER 2
70
Trade-off Analysis in Web Development An experiment on the use of QFD
Many Web Applications are depending on keeping a loyal user group. This is partly archived by frequent updates, including new content, new presentation and new functionality. Time-to-market (TTM) is an important requirement for these updates. This pose a challenge for the quality of Web Applications, which have to be balanced properly. When prioritising new requirements, the benefits and the consequences have to be assessed and conflicts between requirements have to be resolved. This can be done by using trade-off analysis. A tool that can be used to facilitate a trade-off analysis is the Quality Function Deployment (QFD) method. This paper reports on an experiment that we have conducted to observe how QFD contributes to resolve conflicts between requirements, to make a prioritisation and to enhance the communication in a development team.
1. INTRODUCTION Two of the factors that determines the success ofWeb Applications is the number of returning end-users and the number of end-users that are using the services offered by a Web Application. To run a Web Application successfully depends therefore partly on how good the development team behind it adapts to the competitive environment this application is running in. A typical Web Application will be updated frequently to react to changes in the particular business, to improve the quality of the functionality it offers and to keep it an attractive place on the Web. A Web Application will also be compared with its competitor(s) and will therefore be updated to strengthen its position vis a vis its competitor( s). The time to develop new
functionality, to perform all changes and to test them is limited. New functionality is often tested and validated by the end-users. A development team must be flexible to react to these requests immediately [10]. To be flexible, development teams will apply development practises that focus on TTM and flexibility. Teams are small and work in parallel [10]. Conflicting requirements are detected late and have to be resolved quickly. There is only a restricted amount of information available to solve such conflicts and to find a proper balance between TTM and quality. A simple method to help projects to balance TTM and quality and to resolve conflicts between requirements would be beneficially. In the WebSys project we have modified the QFD method to reflect the opinion of individual stakeholders and to guide a project team to find a balance that most stakeholders can support [18]. In this paper we present an experiment to test how this modified version of QFD supports the decisionmaking of small projects. The rest of this paper is organized as follow: In section 2 we present the modified QFD method and how to use it in projects. In section 3 and 4 the design of the experiment and the results are described. A discussion of the experiment and future work follows in section 5. Conclusion are given in section 6.
2. TRADEOFFANALYSIS IN WEBDEVELOPMENT As mentioned above many Web Applications are developed in a competitive environment. In order not to loose potential users to competing Web Applications, new functionality is introduced to increase the attractiveness of the Web Application. The quality of new functionality is often not as important as its presence. If the quality is not good enough it can be improved later. This focus on TTM is sometimes called Rush-to-market. New functionality is sometimes introduced for marketing reasons only [10]. The introduction of new and cutting-edge technologies adds to the attractiveness of a Web Application. Also, the appearance of a Web Application will be changed (makeover) to give the impression of ”newness” [3]. The development activities have to adopt to this competitive environment. This is done by following a number of development practises, including [9]: – Evolutionary Development – Applications are delivered in
successive releases with short time intervals. This makes it possible to correct bugs and improve the qualitative when
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 3-WoSQ ‘05, May 17, 2005, St Louis, Missouri, USA. Copyright 2005 ACM 1-59593-122-8/05/0005…$5.00.
103
71
this becomes a problem. This is sometimes called Web Gardening [9].
– Parallel development – In order to to reduce the time for each development cycle, the application is developed in parallel development. This means that a developer can work on different releases in one iteration.
– Ad hoc development process – The development of Web Applications lacks the rigour and systematic approach, quality control and assurance that is found in traditional software development.
– Requirements – Requirements are evolving over time. Prototypes are used to communicate with the customer, and end-users are sometimes used to test, validate and refine the requirements.
These practises are helping the developers to deploy new versions of a Web Application at short time intervals. Updates every week or every second week are not uncommon [10]. But what is good for marketing purposes of a business is not without problems for the quality of the applications [9]. Quality is itselfs an important competitive advantage. If the quality of a Web Application is not good enough, users will go to other Web Sites. After all, the nearest competitor is only one mouseclick away. When a Rush-to-market attitude dominates the development activities the focus is mostly on the functionality and not so much on quality (non-functional requirements). TTM, newness, and quality should therefor be balanced carefully. This is not an easy task. Given the short TTM, it is hard to have full knowledge of all the consequences of a number of choices and decisions that a development team have to take. To find the right balance between TTM, newness and quality one has to consider the pros and cons from the available options. This is mainly based on qualitative information – such as experience and beliefs – and not so much on exact knowledge or quantifiable information. The information is supplied by the involved stakeholders who express their beliefs on the available options. A decision can be made by the team by looking for a combination of options that can satisfy as many stakeholders as possible. This goalsetting has to be realistic and consider all aspects of the software product. Based on the present information, the solution that is considered the best possible by the team is chosen. This process is inspired from utility maximisation found in decision theory [15]. By using Trade-off analysis we want to support the effective decision making of small software projects. Trade-offs will be helpful for projects in several ways, by improving [18]: – Awareness – Awareness about conflicting requirements and
the possibility to balance different options to get a successful product.
– Negotiation – A way to consider the available options and their consequences. This may result in changing one or more goals.
– Communication – Bringing together all stakeholders and sharing all available information to reach a decision.
2.1 Quality Function Deployment When choosing a tool for trade-off analysis we decided that our tool must fulfil the following requirements: – Be easy to understand for all parties involved. – Include both qualitative and quantitative information. – Be applicable to all activities in a web project. – Be applicable with all development methods and process
models. We chose the Quality Function Deployment method – QFD for short – because we find it easy to learn and use, and it is well suited for the trade-off analysis described above, because of its flexibility. It is described in [1] and also in [6]. QFD is a method for answering important questions during the requirement analysis and other phases of a software project. In the requirement phase, which is the only phase we are looking at here, QFD looks at three questions: – What – what shall we implement? – How – what method, technique, etc. shall we use to realise
each requirement? – How much – how much resources shall we put into meeting
the requirements? Traditionally, the QFD matrix uses three symbols to indicate the relationships between the ”whats” and the ”hows” (strong correlation, weak correlation and conflicting relation). In order to use QFD in trade-off analysis, we use numerical values in the matrix as follows: – 9 points, strong positive relation. – 3 points, weak positive relation. – 0 points, no relation. – Corresponding negative values for negative relations. We find that the following factors make this method well suited for trade-off analysis (An example of a trade-off analysis is given in [18]): – The requirement are prioritised by the stakeholders. In
addition, the level of todays product can be compared with the most important competitor(s).
– The method allows us to include importance and dif- ficulty scores, which give us a good idea of the work needed when using each tool or method and for implementing each function
– By relating the prioritising of the stakeholders to todays level and the level of the most important competitor( s) the trade-off analysis supports the making of a realistic decision.
3. THE EXPERIMENT As stated earlier, one of the main concerns in a trade-off situation is communication. We hope that QFD would make the communication more efficient, since it would help the stakeholders to see the consequences of their decisions and how their decisions influenced the other stakeholders success criteria. Although a trade-off can be considered as a purely technical
104 CHAPTER 8. PAPER 2
72
problem [2], [11], we have chosen to take a somewhat wider perspective and also consider the people side of this problem.
3.1 Hypothesis and instrumentation A good hypothesis formulation is important to any experiment since it will influence all further work such as organisation, instrumentation and analysis. Our hypothesis was that: H0: ”The use of QFD will not improve the quality of the communication in a trade-off meeting”. H1: ”The use of QFD will improve the quality of the communication in a trade-off meeting”. If we want to test the hypothesis, we need to find a way to measure communication quality – the instrumentation of the project. Since communication quality to a large extent is a subjective matter, we decided to solve the problem by using a post-experiment questionnaire. The questionnaire used a Likert scale from 1 to 5 for all questions. The questions pertaining to communication quality, and thus to our hypothesis, were: – Did you find it a difficult decision to make? – Did the stakeholders different requirements create any
conflicts in the group? – Did you find the communication in your group to be
constructive? – Did you all agree on the final decision? The answers to these questions will, in our opinion, be a good measure of communication quality. In addition to the scores of the individual questions we also looked at how much the persons in each group agreed on the answers to the questions. Had they, so to speak, been at the same meeting?
3.2 Experimental design The participants were volunteers from the students taking software engineering courses – second and third year students. They were randomly organized into two groups – the treatment group – 14 persons – and the control group – 17 persons. The treatment group received a 15 minutes introduction to the QFD method. The participants were organised into small groups – three or four persons each – that should perform the trade-offs. We had
five groups that used QFD and five groups that just sat round a table and discussed the problem. Each group consisted of persons playing roles and each role had its own agenda for the project that they were about to start. The roles and their agendas were as follows: – Marketing manager – wants a fancy system and a short time
to market. – Systems developer – wants an easily updated browser and a
user friendly system. – Programmer – wants a challenge and the opportunity to use
new technology and methods. For the four persons groups, we used two programmers instead of one. The technology involved was: – Pure HTML – static information and to a large degree
independent of the browser. – Flash – a new technology as far as our firm is concerned. It
gives highly dynamic systems but is hard to change once it has been put into operation.
– JavaScript – dynamic HTML that is user friendly but can cause problems with some browsers.
Each of these technologies had a QFD table that shows the impact on the stakeholders agendas. The influence of each technology and the importance of each project goal for each stakeholder were fixed. The QFD table for JavaScript is shown in figure 1. Each stakeholder inserted his own priorities – weights – into the table and were thus able to see the effects on ”his” total score if the project decided for instance to chose JavaScript.
3.3 Experiment operation The experiments were run in meeting rooms on campus. The participants who had received QFD training were randomly assigned to one of the treatment groups while the others were randomly assigned to one of the control groups. We first run the experiments with the control groups and then, on the next day, the experiments with the treatment groups. The rooms used for each trade-off meeting were arranged so as to prevent discussions between the groups. It also made it difficult for one group to hear what was going on in any other group. In addition, one of the authors – the experiment supervisor – was present at all times in order to prevent information sharing between groups. The participants were given their role cards and a problem description – the goal of the trade-off meeting. The role card gave a short description of their hopes – what they want to get out of the discussion – and fears – what they want to prevent. For instance, the role card for the marketing manager contained the following information: – Want: fancy effects, short time-to-market. – Fears: a boring website. When the group had reached a decision, they gave a signal to the experiment supervisor, who handled them their post-experiment questionnaire. The participants were told to fill in their questionnaire without talking to each others. The experiment supervisor controlled this process. The data from the
Figure 1: QFD matrix
105
73
questionnaire were arranged into tables for later analysis. The tables pertaining to the communication quality are shown in table 1 in the analysis section.
3.4 Validity One of the main concerns when making an experimental design is validity. The results from the validity discussions tell us what kinds of validity we can claim for the results from our experiment. We have used the validity groups used by Claes Wohlin (see [17]). The in-dept discussion on this topic is too long for a short paper. The readers should instead consult Magnar Sveens report, [13]. Here we will just identify what we consider the most important threats to validity for this experiment. – Internal validity. Our main concern is the instrumentation for
the experiment. If one or more of the questions in the post-experiment questionnaire has been misunderstood or interpreted differently by one or more persons, the results will not be valid.
– Conclusion validity is, among other things, concerned with sample size and is discussed under analysis.
– External validity. Our main concern here is the generalizability of the setting, especially the fact that none of the participants in the experiment were real stake holders.
– Construct validity. Our main concern here is the monomethod bias of the experiment. We have, however, taken any reasonable precaution to reduce this effect – for instance by analysing differences in both communication quality and group agreement.
3.5 Analysis All results from the experiment – the data from the postexperiment questionnaire – are ordinal values. According to statistical orthodoxy we should use non-parametric statistical methods to analyse the data. Practitioners have, however, left this stance a long time ago – see for instance [12] and [7]. We have thus chosen to use the t-test to check for differences between scores on the Likert scale and the Ftest to check differences in variances. Just to put things into perspective, we quote one the grand old men of statistics – John Tukey – who states that:
”An oversimplified and over-purified view of what measurements are like cannot be allowed to dictate how data is to be analyzed” [14].
The other problem we are facing is the question of sample size. We have used the formulas and ideas of Will Hopkins [8]. The important concept in sample size calculation is effect size – ES. If the control group has the mean µ1 and the treatment group has the mean µ2, then ES = (µ1 – µ2)/σ. If C is a function of the probabilities of type I and type II errors, then for many situations – including ours – we have that the sample size N can be found from N = C/ES2. For a probability of 5% for type I errors and a probability of 20% for a type II errors, we get C = 32 and thus N = 32/ES2. This assumes, however, that we have the same number of persons in both the treatment group and the control group. If the two numbers are different, we can claim statistical power equivalent to N = 4 * n1 * n2/(n1 + n2). In our case, however, the difference – n1 = 14 and n2 = 17 – is too small to matter. A total of 31 persons participated. Thus, we can observe an effect size of approximately 1.0. This is a large to moderate effect. The observations pertaining to the questions used to measure the communication quality are summed up in table 1. Using the t-test we find that all differences are significant at the 5 % level.
4. RESULTS The t-tests showed that the observed differences for the communication quality factors were all significant at the 5% level. The effect sizes are between 0.8 and 1.0 and are, however, on the limit of what we can expect to observe with just 31 persons. The results can be summed up as follows: Did you find it a difficult decision to make? Those using QFD experienced fewer difficulties. Did the stakeholders different requirements create any conflicts in the group? Those using QFD experienced fewer conflicts. Did you find the communication in your group to be constructive? Those using QFD found the communication to be less constructive.
106 CHAPTER 8. PAPER 2
74
Did you all agree on the final decision? Those using QFD found that they agreed more on the final decisions. If it had not been for the assessment of communication constructiveness, the group the conclusion would be straight forward. Another question pertaining to communication was: ”Did it become clear to you what the different stakeholders found important?” Here those who used QFD scored slightly lower than those who did not use this method. This difference is, however, not significant at the 5% level. As stated earlier, we also wanted to check the variation for each of the groups engaged in the trade-offs. A small variance for a group on a question from the exit questionnaire will be used as an indicator for better communication. To check this, we used a standard F-test with a 5 % significance level. Only one question showed a significant difference between trade-off meetings with and without using the QFD. ”Which stakeholder would you say got the most requirements fulfilled?” Here the groups using QFD all agreed on the outcome, while the groups not using this method had a quite mixed opinion of who got what. In all other cases, the variance for the answers was the same.
5. DISCUSSION AND FUTUREWORK The result of the experiment encourage us to continue the work on trade-off analysis in Web Development projects. There are, however, a number of issues that we need to discuss. The first issue is the one about the less constructive communication from the groups using QFD in the experiment. We did not expect this to happen as we believe that performing a trade-off analysis with QFD will improve the communication. We found some explanations and also learnt also a lesson. In our opinion, the explanation is that the use of a new and unknown method – in this case QFD – resulted in a simpler communication pattern. It seems that the focus change from communication with other people to communication with or through a tool. Those who used QFD spent most of their time doing calculations to see what happened to their own priorities while those who did not use any special method communicated with the other participants in the meeting. This is consistent with the observations of the
experiment supervisor. The two situations are as shown in figure 2. The situation to the right hand side opens up for more communication and thus for more con- flicts but also more understanding of the other stakeholders and their needs. Another explanation is that there were a time limit of 40 minutes for every group. Given the previous explanation and the time limit, there was not enough time to discuss the options. What we can learn from this is that everything that can be done to direct the focus on the trade-off and not the method should be done, for instance tool support and better training with QFD. Also, the guidelines for using such a method can include a rule that every stakeholder has to explain his weighting of the requirements, or that the group will discuss the weight for the influence from technology A on requirement M. The goal of the QFD method is
Did you find it a difficult decision to make?Number of answers in each categoryCategory Using QFD Not using QFD
1 (easy) 5 12 7 83 2 64 0 0
5 (hard) 0 0
Did the stakeholders different requirements create any conflicts in the group?
Number of answers in each categoryCategory Using QFD Not using QFD
1 (none) 8 22 2 63 3 44 1 5
5 (many) 0 0
Did you find the communication in your group to be constructive? Number of answers in each categoryCategory Using QFD Not using QFD
1 (no) 0 02 1 03 4 04 6 8
5 (yes) 3 9
Did you all agree on the final decision?Number of answers in each categoryCategory Using QFD Not using QFD
1 (did not) 0 02 0 03 0 24 2 5
5 (totally) 12 10
Table 1: The results from the post-experiment questionnaire
Figure 2: Difference in the communication pattern between QFD and non-QFD groups
107
75
to support an easy and fast decision making process. Any new discussion will slow down the process. These rules should therefore only be used when the focus is too much on the tool and not on the problem at hand. An interesting point is that the groups using QFD – even if they had a less constructive communication – found the decision easier to make. This troubles us because this can also imply that a group makes a decision without noting that there is an obvious better decision. All agreed on the early decision and are blinded by it. This again emphasises how important it is to strengthen a constructive communication inside the team. The use of subjective evaluation rises the question of how reliable the collected data are [16], and how this evaluation is done. This issue is related to the discussion about validity. We will discuss this in some detail. The method described above is dependent on expert opinions and will be no better than the experiences available from the persons participating in the trade-off process. An open question is how to represent the collected information. We chose to use a Likert scale with 5 values (strong positive relationship, weak positive relation, no relation, weak negative relation and strong negative relation). People with a mathematical or statistical background often have uneasy feelings about doing math on values from a Likert scale. Whether this is permissible or not has been debated for more than 50 years and the debate is still raging between the persons pure at hearth and the persons with a more pragmatic view. A good summary and discussion of the positions of the parties involved can be found in [5], pg. 150 – 154, which also contains a large number of references. In this short paper, however, we will only note two points – 1) It has
been working for QFD in an industrial setting for more than 40 years and 2) several active researchers have found it permissible to use statistical methods on values from a Likert scale, e.g. [4]. A related issue is the possible misuse of the weights. They are subjective measures, and by choosing higher weights a stakeholder might try to increase his influence. If this becomes a problem it will be an easy task to ask every stakeholder to explain his weighting. Any obvious misuse will then be detected. In the future we would like to replicate this experiment with some changes to improve the communication pattern (as discussed above), and we are planning to perform case studies with companies using QFD to perform trade-off analysis. We will also work on an approach for multidimensional trade-off analysis, based on QFD.
6. CONCLUSIONS In this paper we have reported on an experiment on using QFD for a trade-off analysis. QFD is easy to use and can be used on quantitative information. The results from the experiment shows that decision become easier to make and that there are less conflicts between the stakeholders. There is, however, the problem that groups using QFD had a less constructive communication. Thus this method has to be used as a supplement to a discussion and not as a replacement for it. We would advise small Web Development teams to use this method for trade-off analysis and to be observant about the discussed weaknesses.
7. REFERENCES [1] Y. Akao. Quality Function Deployment. Productivity Press, Cambridge, MA, 1990. [2] B. Boehm. Safe and simple software cost analysis. IEEE Software, 17:14–17, September/October 2000. [3] F. Brassington and S. Pettit. Principles of Marketing, Third edition. Prentice Hall, 2003. [4] D. Davis. Business Research for Decision Making. Duxburry Press, fourth edition edition, 1996. [5] T. Dybå. Enabling Software Process Improvement: An Investigation of the Importance of Organizational Issues. PhD thesis, NTNU, 2001. IDI Report 7/01. [6] R. B. Grady. Practial Software Metrics for Project Management and Process Improvement. Prentice Hall PTR, 1992. [7] D. Hand. Statistics and the theory of measurement. Journal of the Royal Statistics Society, Series A, 159, 1996. [8] W. G. Hopkins. A new view of statistic. Technical report, University of Queensland, 2001. [9] S. Murugesan, Y. Deshpande, S. Hansen, and A. Ginige. A new discipline for web-based system development. In Proceedings of the Workshop on Web Engineering, ICSE99, 1999. [10] B. Ramesh, J. Pries-Heje, and R. Baskerville. Internet software engineering: A different class of processes. Annals of Software Engineering, 14(1-4):169–195, 2002.
[11] G. Ruhe and D. Greer. Quantitative studies in software release planning under risk and resource constraints. In International Symposium on Empirical Software Engineering, 2003. [12] S. Stevens. Psychological Scaling: Theory and Applications; Eds: H. Gulliksen and S. Messick, chapter Ratio Scales, Partition Scales, and Confusion Scales. Wiley, 1951. [13] M. Sveen. An experiment with the use of tradeoffs and quality function deployment in web development (master thesis). Technical report, Departmend of Computer and Information Science, NTNU, 2005. [14] J. W. Tukey. The Collected Works of John W. Tukey, Voll III: Philosophy and Principles of Data Analysis: 1949-1964; Ed: L. V. Jones, chapter Data Analysis and Behavioral Science or Learning to Bear the Quantitative Mans burden by Shunning Badmandments. Wadsworth & Brooks/Cole Advanced Books & Software, 1986, 1961. [15] P. Weirich. Realistic Decision Theory. Oxford University Press, 2004. [16] C. Wohlin and A. A. Andrews. Assessing project success using subjective evaluation factors. Software Quality Journal, 9:43–70, 2001. [17] C. Wohlin, P. Runeson, M. H¨ost, M. Ohlsson, B. Regnell, and A. Wessln. Esperimentation in Software engineering – An Introduction. Kluwer Academic Publishers, 2000. [18] S. Ziemer and T. Stålhane. The use of trade-offs in the development of web applications. In Workshop Web Web Quality, 2004.
108 CHAPTER 8. PAPER 2
Chapter 9
Paper 3
Sven Ziemer and Tor Stalhane:Web Application Development and Quality – Observations from in-terviews with companies in NorwayIn Jose A. Moinhos Cordeiro, Vitor Pedrosa, Bruno Encarnacao, and JoaquimFilipe (Eds.): Proc. Second International Conference on Web Information Sys-tems and Technologies: Internet Technology / Web Interface and Applications(WEBIST 2006), Setubal, Portugal, April 11–13, 2006, Vol. 1. Published byINSTICC Press, ISBN 978-972-8865-46-7, pp. 495–498
109
110 CHAPTER 9. PAPER 3
WEB APPLICATION DEVELOPMENT AND QUALITY –OBSERVATIONS FROM INTERVIEWS WITH COMPANIES IN
NORWAY
Sven ZiemerDepartment of Computer and Information Science, NTNU
Keywords: Web Application Development, software quality, development practises, trade-off.
Abstract: In this paper we present our findings from a series of interviews with companies developing web applica-tions, investigating how quality issues are managed when developing web applications in a rush-to-market andcompetitive environment. Our findings suggest that requirement practises are communication intensive, thatcompanies perceive quality attributes related to a good user experience important, and that companies don’thave a clear trade-off situation.
1 INTRODUCTION
Web Applications have become part of our every daylife. Hence, the quality of web applications are of in-terest to all users. We take a special interest into howquality issues are managed in web application devel-opment. In this paper we present our findings from aseries of interviews with Norwegian companies1.
The paper is organised as follows: We start withthe background for our study in section 2. The find-ings from the interviews are presented in sections 3, 4and 5. In section 6 we discuss our findings and con-clusions are given in section 7.
2 BACKGROUND
We interviewed representatives from seven Norwe-gian companies developing web applications. The in-terviewees had different roles in their companies: ITmanager (3 companies), developers (3 companies),and development manager (2 companies). For onecompany, we had two interviewees. We used a semi-structured interview. The selection of companies toour sample were partly based on their willingness andavailability for the interview and on the type of webapplication they are developing. We were approach-ing companies developing internet based web applica-
1The work is port of the WebSys project, which is sup-ported by the National Research Foundation in Norway
tions, that are critical for the companies business, thatare used in the marketing strategy of the company andwhere TTM is important.
The companies selected for the interview differquite a lot, both with respect to size, organisation andwhat they are developing. Some stats about the com-panies are given in figure 1.
3 REQUIREMENTS
Eliciting and specifying requirements are importantsteps in the development cycle of every software sys-tem. How do companies find the right balance be-tween specifying enough details for the requirementsand the development time. In our view this is a trade-off between reducing risk and TTM.
We found that requirement practises where dependon the ownership of the web applications: whether thedevelopment where done in-house or bespoken.
We asked the companies in our sample the follow-ing questions: (1) How are requirements elicited? (2)How are proposals for new requirements and changerequest managed? (3) How are requirements speci-fied? (4) How are requirements validated?
3.1 Elicitation
Requirements where elicited in a number of ways.The most obvious way are customer given require-
111
Company DomainCompany
SizeProject
sizeDeployment frequency Type of development
1 Travel 8 3Daily on contenct; bug fi xes and improvement every 2nd week; user-interface remake every 18 months
In-house development
2 Media 650 3 – 4Bug fi xes weekly; new functionality whenever customer dependent.
In-house development
3 Service 80 4 – 62 releases every year; project time for each release is 4 – 6 month´s. Custom releases: project time is 1 – 6 months.
In-house development
4 Travel 20 3 – 5Minor uppgrades every 3rd week, new services every 3rd month.
In-house development
5Development for customer
10 1 – 3Project delivery for customer; typical project time is 1 – 2 month´s.
Development for customers
6Development for customer
25 1 – 3Project delivery for customer; typical project time is 1 – 2 month´s.
Development for customers
7 Finance 250 5 – 10Scheduled releases; typical project time is 4 – 6 month´s.
In-house development
Figure 1: Statistics on the interview sample
ments. Four companies had a formal approach ofwriting requirements, including both companies de-veloping bespoken software. These companies usedthe requirements specification as part of an agreementwith the customer. Two of the companies developingin-house software used domain experts to elicit re-quirements and two where following a developmentprocess. The other companies used oral communica-tion and email communication in an informal way.
Requirements where also elicited by collecting userfeedback. This can both be error messages and moregeneral improvement suggestions. We found that userfeedback was used mostly by companies developingin-house software, where developers have direct con-tact with the users. Companies developing bespokensoftware have to discuss the user feedback with thecustomer to decide what to include.
Two companies used to specify requirementsfor marketing purposes. These where collectedfrom comparisons with competing applications, fromstrategic planning to make the web application moreattractive to new user groups, and from user satisfac-tion surveys. Both companies are developing theirsoftware in-house.
To sum up, the companies developing in-houseseem to have more informal requirement elicitationpractises. They rely on oral communication and onthe domain knowledge of their developers. Compa-nies developing bespoken software have in generalmore formal development practises, mainly becauseof the contractual relationship with the customer.
3.2 Specification
The two companies developing bespoken softwareproduce a written requirements specification, which
is then part of the contract between the companies.Requirements are detailed according to the informa-tion given by the customer.
Practises found in in-house development are moreinformal. Most requirements are presented and de-tailed in oral communication. This can take the formof ”this would be a nice function to include”, or ”doyou remember what we did on another system oneyear ago? That would work fine for our new systemtoo, with some changes”.
As mentioned earlier, user feedback is used to elicitrequirements. Both error messages and improvementsuggestions are received in written form, by emailor chat. But another part of the user-feedback isthe hands-on experience resulting from user support.Typically, this information is not documented. Thedetails of the requirements and change requests are inthe understanding of the developers, as a sort of tacitknowledge.
When the requirements have to be clarified or har-monised, this will be done either in conversationsbetween two developers or in team workshops. Ifnecessary, requirements will be detailed with somesketches or with a simple user interface mock-up.Some of the companies are producing written notesfrom such meetings.
When working with written requirements, thesewere presented as either plain text, use cases or screenshots. One company used the PLanguage language(developed by Tom Gilb, see (Gilb, 1989) and (Gilb,2004)) to specify their requirements.
3.3 Requirement validation
When requirements are not specified in details, it ishard to validate them. Therefor, most companies
112 CHAPTER 9. PAPER 3
where relying on user feedback for requirement val-idation. Only two companies from our sample dohave a dedicated testing department. We encountereda number of strategies to validate the product:• The web site is validated by the users.New ver-
sion are developed to fulfil the expectation of thedevelopers. After publishing, the application willbe changed in accordance with the user feedback.
• A prototype is validated by the customer.Cus-tomers can evaluate a prototype of the web ap-plication. Dependent on the customers opinion,the web application is completed for deploymentand changed according to the users comments oraborted.
• The web site is validated by a special user group.The same as above, except that the web applicationis used by a special user group, such as a group ofexperienced users or a test panel.
• Comparison with competitors.The web applica-tion is compared to the main competitors. If anyshortcomings are identified, they will be fixed priorto publishing.
4 QUALITY ISSUES
We have gained some insight into how the compa-nies in our sample handled the quality aspects of theirproducts. This was done by asking them questionssuch as ”What are important quality factors?” and”What is important for your customers?” Since thecompanies that we interviewed span a wide range ofapplication areas, we want to understand the spread ofopinions and their implications for the developmentof web systems.
The most important quality factors mentioned wereavailability and reliability, performance and to givethe users a good user experience.
By and large, the priorities given by the companieswhen asked about important quality factors were alsoreflected when the companies were asked to name im-portant project success factors (see figure 2).
To sum up – the most important factors for a suc-cessful web system are availability, performance andthe ability to give the user a good experience. Thisis the user view. In order to achieve this, the devel-opers need to focus on how to build systems that arereliable, scalable and have high usability.
5 DEVELOPMENT PROCESS
In the development process, we want to look at howprojects are initiated, estimated and staffed, and howthese factors contribute to trade-off opportunities.
5.1 Project organisation
All the development teams are small – three to fivepersons. From this it follows that most of the devel-opment projects in our sample are small. Once weknow this, it is hardly a surprise that all the compa-nies used one form or another of incremental devel-opment. The mode of development spans the wholespectrum – from just incremental to using Evo (Gilb,2004). The latter is a complete and documented de-velopment method.
Only two of the companies say that most of itsprojects are initiated as a result of a request for ten-der. In most cases, projects are initiated either by acustomer request or by product ideas stemming fromcooperation between the company and a set of keycustomers.
When it comes to estimation, the most interestingresult is that two out of six of the companies do not doany real estimation at all. Instead, they try to get anidea of what the customer is willing to pay and thenthey go backwards from there to define an acceptableproject. Since quality factors are the requirements thatare hardest to define and hardest to test, they easily be-comes the factors that are adjusted in order to deliveraccording to calendar time and budget.
5.2 Trade-off opportunities
Most of the companies involved had no clear trade-offstrategy. One reason for this was that they did not feelthe need, since they used an incremental developmentprocess. This process gives them ample time to adjustrequirements and quality throughout the project andthe need for explicit trade-offs is small.
One company said that they did trade-offs betweenall their important quality factors – performance, scal-ability, and newness. One company did trade-offs be-tween price and complexity while one said that it usu-ally adjusted the quality factors in order to finish theproject within budget.
6 DISCUSSION
Our sample of companies is small and we should notgeneralise our findings. Still, we consider our findingsas an indication that a different set of developmentpractises will evolve when developing software in arush-to-market and competitive environment.
An observation we made is that all companies seemto be successful. They had a clear understanding ofwhat they perceived as their success factors and man-aged fairly well to live up to them. Not surprisinglywere all success factors in some way related to theuser view of the system.
113
# QualitySuccess factors
Trade-offsEstimation
and planningTeam and
organizationProject initiation
1Availability, good user experience
Usability – Expert judgment
Small teamsEnvironmental changes, customer changes
Usability – EvoSmall teams, some outsourcing to Vietnam
Customer changes, small team from company and key customers
4Performance, availability, good user experience, scalability
Development time
– – – –
5Performance, reliability, availability, good user experience
Performance, reliability
Complexity vs. price
What will the customer pay?
–Invitation to tender, pre-project
6 Reliability Usability, customer relevance
All quality related requirements
Expert judg-ment and WBS
– Invitation to tender
7 Data integrity, security Development time
– – –All development is out-sourced
Figure 2: Quality issues from the interviews
Informal and oral communication plays an impor-tant role when developing web applications in a rush-to-market environment. Most people we spoke withwhere satisfied with this practise. It also seems thatthere are few problems associated with this practise.And if used wisely it can be a considerable contribu-tion to the success of the company. One explanationfor this can be the ”piece-wise” development strategyof web applications. Developers get early feedbackfor their work, problems and conflicts in the require-ments are detected early, and when necessary the levelof detail for requirements can be increased.
The dependence on the developers experience anddomain knowledge can be a challenge to this devel-opment practise. A lot of information is never docu-mented. This makes a development team vulnerablewhen people leave the company.
Another challenge is to balance requirements spec-ification and decision making. Most communicationis informal and oral. How can different options be as-sessed when the information available is limited. Asimple tool to show both the positive and the nega-tive impact of the chosen technology would have beenhelpful. We have proposed such an tool in our previ-ous work (Ziemer and Stalhane, 2004).
What surprised us was that most companies did nothave a clear trade-off strategy. This does not meanthat they did not perform any, but they were not awareof it. It is part of a trade-off to make a decision on howto balance two or more conflicting factors. Not beingaware of this means that the decision making is hap-hazard and that there is no opportunity to systematicimprove the result of the trade-off.
7 CONCLUSION
In this paper we have presented our findings from aseries of interviews with companies developing webapplications in a competitive and rush-to-market en-vironment. Our findings suggest that development insuch a environment is communication intensive. Ourcontribution lies in the documentation of developmentpractises found in a small sample of companies devel-oping web applications. We related these practises tothe success criteria identified by the companies themselves and pointed out some future work directions.
In the future we will to look more at the decisionmaking processes applied in web application devel-opment. How can good trade-offs be performed withthese informal practises.
REFERENCES
Gilb, T. (1989). A planning language (a planguage). InAPL ’89: Conference proceedings on APL as a tool ofthought, pages 169–177. ACM Press.
Gilb, T. (2004).Competitive Engineering: A Handbook forSystems and Software Engineering Management Us-ing Planguage. Addison Wesley Longman PublishingCo., Inc.
Ziemer, S. and Stalhane, T. (2004). The use of trade-offsin the development of web applications. In Matera,M. and Comai, S., editors,Engineering Advanced WebApplications. Rinton Press.
114 CHAPTER 9. PAPER 3
Chapter 10
Paper 4
Sven Ziemer, Pedro R.F. Sampaio and Tor Stalhane:A decision modelling approach for analysing requirements configura-tion trade-offs in time-constrained Web Application DevelopmentIn Proc. Eighteenth International Conference on Software Engineering andKnowledge Engineering (SEKE 2006), San Fransisco, 5-7 July 2006. Publishedby Knowledge System Institute Graduate School, Skokie, IL, USA, ISBN 1-891706-18-7, p. 144–149.
115
116 CHAPTER 10. PAPER 4
A decision modelling approach for analysing requirements configurationtrade-offs in time-constrained Web Application Development
Sven Ziemer1, Pedro R. Falcone Sampaio2 and Tor Stalhane1
1 Department of Computer and Information Science,Norwegian University of Science and Technology, N-7491 Trondheim, Norway
2 School of Informatics, University of ManchesterPO Box 88, Manchester M60 1QD, UK
Release planning is a key phase in a time-constrainedweb development process, enabling stakeholders to findan appropriate trade-off between time-to-market and qual-ity/functional characteristics of the delivered artifact. De-spite the importance of conducting release planning activ-ities, many web development projects still employ ad-hocmethods for deciding on the next release to be developed.In this paper, we discuss an effective release planning andconfiguration method underpinned by a decision modellingframework aimed at supporting small development teams toanalyze, prioritize requirements, and to find a candidate re-lease configuration that can be developed within the time,quality and functionality constraints relating to the project.The method is value-based, taking into account the stake-holders’ belief in what is regarded as the most valuable re-lease configuration. The method is also consensus-driven,seeking to promote a consensual agreement with regard tothe next release to be implemented.
1 Introduction
When developing software systems it is important tomeet the stakeholders’ needs and expectations. Usually,there is a considerable investment at stake and the stake-holders are keen to maximise their return on investment.Many stakeholders also want to realize part of the totalvalue at stake in connection with the software system in theearly phases of the development project and not wait untilthe project is completed in order to benefit from the systemproperties. One way of meeting stakeholders expectationsearlier is to deploy the system as successive functionalitychunks, as it is done in incremental development. For ev-ery new chunk that is delivered, some part of the system’s
value will be returned to its stakeholders. In addition, earlyfeedback can be collected and necessary changes can be im-plemented. Planning for the next release thus becomes animportant and complex activity. The complexity stems fromthe task of finding the optimal release that can be developedwithin the time constraints and that represents the highestpossible value in return for the stakeholders.
Web applications that are developed in competitive en-vironments put a strong emphasis on timely development.Hence, it is important to find the “best possible” release –consisting of both new features and/or improved features– that satisfy customers and that also address competitiveforces that may arise in connection with the application con-text, e.g., a release that covers all functionalities available ata competitor’s web site.
Release planning is often done ad-hoc [8]. In interviewswith Norwegian small and medium enterprises developingweb applications we have found that development practisesare informal, ad-hoc and ”rush-to-market” [9]. This makesit difficult to develop a release plan strategy. One reasonfor this is the lack of information and time to identify andanalyse candidate releases, or even to have a detailed analy-sis of the requirements that should be contemplated at eachstage. As these companies are depending on the domainknowledge of their developers and other stakeholders, thediscussions between stakeholders are based on their expe-rience and opinions. In many cases, these opinions can beconflicting as each stakeholder has his own interest and re-sponsibility. In order to make important decisions, like thenext release(s), it is important to find the right balance be-tween several options advocated by the stakeholders.
The remainder of this paper discusses an effective re-lease planning and configuration method underpinned by adecision modeling framework aimed at helping small devel-opment teams to analyze and prioritize requirements, andto find a candidate release configuration that can be devel-
1
117
oped within the time, quality and functionality constraintsrelating to the project. The method is value-based, takinginto account the stakeholders’ belief in what is regarded asthe most valuable release configuration. The method is alsoconsensus-driven, seeking to promote a consensual agree-ment with regard to the next release to be implemented. Thepaper is organised as follows: A description of the problemof finding the next release is given with some detail in sec-tion 2 and an example is presented in section 3. Section 4looks at how stakeholders can express their beliefs about re-turn on value using qualitative information. This approachis then used in section 5 to analyse, negotiate and decideabout requirements and configurations. Related work is pre-sented in section 6 and conclusions are given in section 7.
2 The problem
Developing software in a competitive environment puts astrong emphasis on time-to-market. There are usually morerequirements to implement than time available for the nextrelease. Thus, only a subset of the requirements can be im-plemented. Deciding on the content of releases is a non-trivial task. For a web application it is important to offerfeatures that attract new users and that at the same time sat-isfies the existing ones. Still, deciding on the content of thenext release is a decision that often is taken on the fly, with-out time for an in-depth analysis. Based on our experience,working with small and medium enterprises in the Websysproject, we have observed that this decision is made eitherby the strongest stakeholder, who is likely to prioritise hisown interests, or by the development team, excluding otherstakeholders that might have a say in this decision. Twofundamental questions follow in connection to the issue ofdeciding on the content of the next release to be developed:
Who should be involved in this decision?There are sev-eral options about who to involve. The decision can betaken by a single stakeholder such as the project manageror the marketing director, by the project manager and thedevelopers or by all major stakeholders. Involving all ma-jor stakeholders is the preferred approach in an environmentwhere a considerable amount of information is handled in-formally, enabling a comprehensive assessment of all rel-evant variables to the decision and also sharing the riskslinked to the consequences of the decision.
What should this decision be based on?In the literaturewe find a multitude of approaches and variables used fordeciding on what will be contemplated in the next release,such as customer feedback, return of investment, stake-holder weights and strategic urgency. Another importantapproach, which is discussed in this paper, focuses on thestakeholders perceived return on value associated with a re-lease. This is a criterion every stakeholder can relate to, asit expresses his expected gain from a release.
ID Description and Interests
S1 Product director. Wants a stable website, that runs smoothly and that has a positive impact on increasing the sales.
S2 Marketing director. Wants a website with a high “newness” and innovation factor and a short time-to-market. Wants to announce special prices on the website every week.
S3 IT manager. Wants a solution that is easy to maintain and extend. Prefers stable versions of tools and technologies that are to be used.
S4 Provider of payment transactions. Wants visibility and reli-able services.
S5 Customers. Wants easy access to the online shop, safe payment transactions, reliable transport (shipment) and access to product specs.
Figure 1. The stakeholders to the online shop
The problem to solve then is how to find the mostfavourable release for all stakeholders. The stakeholderswill assess each candidate release and express their per-ceived return on value for it. This is based on each stake-holders subjective belief. The goal is to find a consensusand to balance the interests from the stakeholders. The fol-lowing design principles guide the approach discussed inthis paper: It must be simple so that it can be understoodintuitively by all stakeholders, it must take into account allperspectives and stakeholder information available (whichin most cases are the stakeholders opinions and beliefs), andit must be effective in converging to a consensual decisionwith regard to the next release. The approach proposed is adecision making framework based on the return on value ofcandidate releases in which a strong emphasis is placed inpromoting a consensual agreement among stakeholders.
3 Example – An E-commerce system
In this section we discuss an example of a small elec-tronic shop, based on one of the Norwegian enterprises in-volved in the Websys project. The small shop for electronicequipment launched an ”information only” website in 1999.It allows customers to browse a product catalogue and tofind information and prices on products. Lately, the com-pany faced competition from two online retailers and froma local shop that started an online shop. The sales reportfor the last year shows that the shop is loosing customersto the online shops. To respond to this challenge from thecompetitors the shop decides to start an online shop itself.The goal of starting a new sales channel is to take back cus-tomers from the competing online shops, and also to winover new customers.
In the process of starting the online shop one had to iden-
118 CHAPTER 10. PAPER 4
Req Description
R1 Improve Browse functionality, add product specs for every entry.
R2 Add product to cart.R3 Place order. Requires that a user logs in. Edit contents
of shopping cart, place orders.R4 Register user, establish user profi le. Profi le contains
name, username, password, and shipment and pay-ment details.
R5 Change user profi le.R6 Cancel order, if order still is pending.R7 Specify a wish list.R8 Browse past orders.R9 Track shipment.R10 Search for a product.
Req Description
R11 Browse through a list of special offers.R12 Personalise product list. Show the user products he
may be interested in, based on previous sales.R13 Maintain catalogue. Add, change or delete items from
the systems catalogue.R14 Browse orders. For every order, check if the products
are available.R15 Manage orders. Change orders,
split orders, etc.R16 General design makeover.Q1 PerformanceQ2 ReliabilityQ3 SecurityQ4 Usability
Figure 2. Requirements
tify the stakeholders that should have an influence over thecontents of the new website, and to make some organisa-tional changes. The development work was done by a thirdparty development company. A manager for the online shophad to be appointed, who was responsible for all contactwith the development company. A new position that is re-sponsible for all administration work for the online cus-tomers had to be created. An important decision was theselection of a service provider for internet payment transac-tions. The number of stakeholders involved in this project,increased substantially compared to the first system. A de-tailed description of the stakeholders is given in figure 1.
The stakeholders arranged a workshop both to identify astrategy for the online shop and to elicit the requirements.The requirements identified are shown in figure 2.
A development scenario– The marketing director is re-questing that the new version of the company website hasto be deployed within six weeks. In his view this is crucialin order to provide an online offer that exceeds all featuresprovided by the competitors. The competitors are also in theprocess of changing their online sites, therefore adhering tothe six week time-to-market plan is essential. The first esti-mate from the development team concludes that it will takeat least 12 weeks to implement all the new requirements.The project manager is asked to produce a list of what canbe developed within six weeks. He comes up with three can-didate configurations, that each can be implemented withinsix weeks. In order to have an online shop requirements R2– R6 are mandatory. The configurations are shown in figure5 and the details on how they where identified are given insection 5.
4 Value assessment
In the example presented in section 3, only a subset of therequirements specification can be implemented within thegiven time-constraints. We call such a subset aconfigura-tion. A configuration consists of a number of functional andnon-functional requirements that collectively have a mean-ingful impact on advancing the implementation of the over-all business goals (e.g., a configuration that implements anew shopping cart) and that can be developed within thegiven time constraints. Each candidate configuration repre-sents a potential software release. There is more than onecandidate configuration in our example and a decision hasto be made on which configuration to implement. As men-tioned earlier, in a situation where a considerable amountof information is not available in writing but only as tacitand informal knowledge, this decision should be taken byall the stakeholders together to include all relevant informa-tion. This information would typically express the stake-holders’ beliefs and opinion about value, cost, benefit, risk,assessment of competition, etc. We will in this paper focuson the return on value.
One way of expressing this tacit information on the fly isthe use of attitude measurement. One widely used approachis the summated rating or Likert scale [7]. We use a sim-ilar approach, where each stakeholder assess the perceivedreturn on value of a requirement using a single scale from 1to 5: 1– no value, 2 – little value, 3 – some value, 4 – highvalue, and 5 – very high value.
We want to clarify some assumptions before describingour approach. The assessment represents the perceived re-turn on value for each stakeholder and is subjective in its na-
119
ture. We assume, however, that the stakeholders understandtheir business and the development project. The projectswhere the approach is applicable are typically conductedby teams of two to five developers and have tight time-constraints (typically between two and 24 weeks). There-fore, an assessment has to be based on every stakeholdersjudgement. The expressed belief represents each stakehold-ers expert judgement, which is based on their best infor-mation, knowledge and intentions. We recognise that anystakeholder who wants to misinform the other stakeholdersto influence the decision making process in his favour couldeasily do so, but then, this would also be possible when bas-ing the decision on ”hard facts”.
5 Analysing, negotiating and deciding aboutconfigurations
In this section we present the overall release decisionprocess, starting with requirement elicitation and resultingin a consensus based decision on which configuration to im-plement as the next release. This process encompasses fivestages, which are described below and shown in figure 3.
Requirement elicitation – The requirements are elicitedusing a elicitation method or practise. In our example aworkshop was arranged. The result from this stage is alist of requirements, including both functional and non-functional requirements (figure 2).
Requirement estimation and assessment– The nextstage is to estimate and assess each requirement. This in-volves the project manager (for the first part) and all stake-holders (for the second part). The result of this activity isshown in figure 4.
For each requirement, the project manager is respon-sible for providing a time estimate, any dependencies onother requirements and an urgency flag, that, if set, indicatesthat a requirement has to be included into the next release.All stakeholders assess each requirement using the scale asshown in section 4. They also write a short justificationexplaining their choice for each requirement. This justifi-cation will be used during the Decision on a configurationstage in case no configuration emerges as a clear collectivewinner. Also, for each requirement the mean value of allstakeholder assessment is calculated.
For our example, the result of this stage is shown in fig-ure 4. Note that the mean value is not shown. The total timeto develop all requirements is 133 days. With two develop-ers, the maximum effort possible within the time constraintis 60 days. Requirements R2 – R6 have to be included inevery candidate configuration to get a meaningful release.
Identifying candidate configurations– Using the infor-mation from the previous stage, a number of candidate con-figurations are identified. There is no rule about how manycandidate configurations should be identified, but as a rule
Requirement elicitation
Requirement estimation andassessment
Identifying configurations
Configuration assessment
Decision on a configuration
Consensus
Decision made
No
Yes
Start
Figure 3. The configuration decision process
of thumb there should be 3 – 7 candidate configurations. Tosimplify the decision scope, there are some guidelines foridentifying candidate configurations:
1. Sort the requirements according to some sorting crite-rias. For every criteria applied there will be a separaterequirement list, which in turn can be used in the sub-sequent step, thus producing more than one candidateconfiguration. Common for all sorted requirement listsis that requirements with the urgency flag set have tocome first. Two sorting criterias are (1) the mean valueof all stakeholder assessments for a requirement, and(2) the number of high scores from the stakeholders.
2. Include into a configuration the requirements in thesorted order, until the estimated totals equals the speci-fied time-constraint or until the addition of still anotherrequirement will exceed this constraint.
3. For each requirement added, include also all require-ments this requirement is depending on.
4. In case that not all requirements with the same sortingcriteria can be included into a candidate configurationdue to the time-constraint, additional candidate con-figuration can be identified by including requirementsthat have the same ranking and that are no yet includedin any configuration for the given sorted requirementlist.
120 CHAPTER 10. PAPER 4
Req Urgen-cy
Esti-mat
Depen-dencies
Stakeholder
S1 S2 S3 S5
R1 4 d R10 4 3 3 5R2 Yes 3 d 1 2 2 4R3 Yes 10 d R2 2 3 2 4R4 Yes 8 d 2 2 3 4R5 Yes 3 d R4 1 3 3 3R6 Yes 7 d R3 3 2 2 3R7 10 d R2 3 4 2 4R8 5 d R3 2 3 3 3R9 3 d R3 3 2 4 4
R10 10 d 2 1 2 3R11 4 d 3 5 2 2R12 15 d R3, R4 2 4 2 2R13 2 d 3 2 4 1R14 2 d R3 4 2 3 1R15 4 d R3, R14 4 2 3 1R16 20 d 3 5 3 4Q1 7 d 4 3 4 3Q2 3 d 3 3 4 3Q3 8 d 3 3 5 5Q4 5 d 3 4 4 4
The three configurations identified in our example areshown in figure 5. The first two lines show the requirementsincluded and the effort estimate.
Configuration assessment– When the candidate con-figurations are identified, they have to be assessed in thesame way as the requirements where assessed. Again,stakeholders write a short justification explaining theirchoice for each configuration. Also, for each configurationthe mean value of the stakeholders assessment is calculated.The configurations can be ranked by sorting them on themean value. The configuration with the highest mean valueis the one with the highest return on value according to thestakeholders assessment.
The result of this stage is shown in the two last linesof figure 5. Configuration 3 has the highest median value,followed by configurations 2 and 1. There should, how-ever, not be any automatism in choosing the configurationwith the highest rank as the next release. The informationthat leads to this rank is meant to support a discussion be-
tween the involved stakeholders in order to reach a consen-sus based decision.
Decision on a configuration– The purpose of this stageis to find a consensus based decision on which configurationshould be implemented in the next release. The result of theconfiguration assessment can end with a clear cut config-uration that emerges as the ”winner” or with two or moreconfigurations ending up with the same ranking. In bothcases, a consensus on which configuration to implement asthe next release, is needed.
If there is a winner from the configuration assessment,it is straightforward to reach a consensus. There should bestrong arguments for not choosing the winning configura-tion. When there is no consensus on which configurationto choose, the stakeholders should first present their writtenjustification to gain more insight into the other stakeholdersassessment and arrive at a consensus.
When no consensus can be reached at this stage, the con-figuration assessment step has to be repeated. A Delphi in-spired process [5] will be used to guide this reconciliation.To arrive at a consensus concerning the consequences of thepossible actions is the main problem in a trade-off situation.The whole decision process will break down if no agree-ment on a configuration that have an acceptable return onvalue for all stakeholders, can be reached.
The Delphi feed-back process is used to arrive at a com-mon understanding. As shown in figure 3 this means thatthe stage Configuration assessment will be repeated. Theprocess broadly runs as follows:
1. Each stakeholder assesses return on value for each con-figuration, and gives his results in writing. The coordi-nator sums up all the assessments – again in writing –and distributes them to all the stakeholders.
2. After having read the other stakeholders opinions, eachstakeholder gives a new assessment. This assessmentis accompanied by a rationale explaining why his as-sessment differs from that of one or more of the otherstakeholders. This new assessment is given to the co-ordinator.
Step 2 is repeated until a consensus is reached.
6 Related Work
There are a number of methods supporting software re-leases decisions, such as [8], [4], [3], [6] and [2]. Thesemethods differ both with respect to the stakeholders in-volved (all major stakeholder, project manager and devel-opers, customers and users), the objectives (Return of In-vestment, value of requirements, customer satisfaction andbenefit of features to customers) and the applied mecha-nisms (Optimization, Stakholder prioritiszing, AHP). The
121
Confi g 1 2 3
Req´s: R1 – R6, R10, Q3, Q4
R2 – R6, R7, R9, Q2
R1 – R6, R10, R11, Q3
Estimat: 58 d 60 d 57 d
Value: S1: 3 S2: 2 S1: 3 S2: 3 S1: 3 S2: 4
S3: 3 S5: 4 S3: 4 S5: 4 S3: 4 S5: 4
Mean: 3 3.5 3.75
Figure 5. Assessment of configurations
approach presented in this paper involves all stakeholders,and is assessing candidate configurations rather than re-quirements, thereby reducing the complexity of the decisionprocess.
Value-based software engineering acknowledge the factthat software systems are not developed in a value-neutralenvironment. A general theory for Value-Based SoftwareEngineering is given in [1]. The approach presented in thispaper uses qualitative measures to express the perceived re-turn on value.
7 Conclusions and further work
This paper described a consensus-driven and value basedrelease planning approach that can help small developmentteams to analyze, prioritize requirements, and decide on thenext release in a scenario of time-constrained web applica-tion development. The approach is underpinned by a qual-itative decision modelling framework that employs scalessimilar to Likert scales and Delphi techniques to identifyrelease configurations and support stakeholders in the pro-cess of assessing, negotiating and choosing the next projectrelease.
The approach was developed based on our experiencein working with small and medium Norwegian enterprisesin the Websys project. In the context of Websys, compa-nies manifested their interest in a systematic (to avoid theblind ”rush to market”) and inclusive (to cater for the lesshierarchical nature of SMEs) approach for deciding on sys-tem releases, taking into account the multiple stakeholderperspectives of value attached to system requirements. Weare currently experimentally assessing the framework withthe consortium of SMEs involved in Websys and prelimi-nary results indicate that the approach has a positive impactin fostering consensus and increasing project commitmentbetween stakeholders. The simplicity of the approach alsohelps in avoiding typical ”analysis paralysis” situations thatarise in connection with complex decision models and/orhighly democratic decision processes.
References
[1] S. Biffl, A. Aurum, B. Boehm, H. Erdogmus, andP. Grunbacher, editors.Value-Based Software Engi-neering. Springer-Verlag Berlin Heidelberg, 2006.
[2] M. Denne and J. Cleland-Huang. The incremental fund-ing method: Data-driven software development.IEEESoftware, 43(14):39–47, 2004.
[3] H.-W. Jung. Optimizing value and cost in requirementanalysis.IEEE Software, 15(4):74–78, 1998.
[4] J. Karlsson and K. Ryan. A cost-value approach forprioritizing requirements.IEEE Software, pages 67–74,September/October 1997 1997.
[5] H. Linstone and M. Turoff.The Delphi Method - Tech-niques and Applications. Addison Wesley PublishingCompany, Inc., 1975.
[6] D. A. Penny. An estimation-based management frame-work for enhancive maintenacne i commercial softwareproducts. In18th International Conference on SoftwareMaintenance (ICSM 2002), Maintaining DistributedHeterogeneous Systems, 3-6 October 2002, Montreal,Quebec, Canada, 2002.
[7] C. Robson.Real World Research. Blackwell Publish-ers, 2002.
[8] G. Ruhe and M. O. Saliu. The art and science of soft-ware release planning.IEEE Software, pages 47–53,November/December 2005 2005.
[9] S. Ziemer and T. Stalhane. Web application develop-ment and quality - observations from interviews withcompanies in norway. InProceedings of Webist 2006,2006.
122 CHAPTER 10. PAPER 4
Chapter 11
Paper 5
Sven Ziemer and Tor Stalhane:Trade-off’s in Web application development – How to assess a trade-off opportunityIn Jose Cordeiro and Slimane Hammoudi (Eds.): Proc. Third InternationalConference on Web Information System and Technologies (WEBIST 2007),Barcelona, Spain, March 3–6, 2007. Published by INSTICC Press, ISBN 978-972-8865-78-8, 8p.
123
124 CHAPTER 11. PAPER 5
TRADE-OFF’S IN WEB APPLICATION DEVELOPMENTHow to assess a trade-off opportunity
Sven Ziemer, Tor StalhaneDepartment of Computer and Information Science, Norwegian University of Technology and Science,
Abstract: This paper discusses a trade-off strategy for small projects and presents a preliminary guidelines for assessingthe appropriateness of a trade-off. The motivation of this work is to make the development team aware of theperformed trade-offs and to see both the associated opportunities and risks, to assess the appropriateness of atrade-off and to gather information for the future development of the system. The development environmentof a web application is changing over time and it is important to know what the success criteria are at any time.
1 INTRODUCTION
Web Applications have become part of our every daylife in such a way that we expect them to work prop-erly and that we can rely on them, both with respectto their availability and our satisfaction. Hence, thequality of web applications is of interest to all users,that want reliable access to the services offered bythe web applications of their choice such as inter-net banking, online shopping, news services and en-tertainment. Improving the development process tobuild high quality applications that fulfil the end-usersexpectation plays a key role in keeping the users sat-isfied. At the same time, businesses of all sizes andkinds are relying on their web applications to pro-mote their services and to fulfill the needs and expec-tations of their customers. They too are interested inthe quality of the applications they are relying on, butthey are also focused on their timely delivery. Manyweb applications have to compete for their users andcustomers against other providers of a similar service.Being the first web site to offer new functionality, orbeing the site with the most complete functionality fora user group, is important in this competition. In othercases, web applications provide their services to sup-port events and have to meet tight deadlines, so thatthe event can be promoted in the best possible way.
Developing web applications in competitive envi-ronments adds to the complexity of balancing func-
tional and non-functional requirements by adding thetime to market requirement. Finding this balance in-volves performing one or several trade-offs, such asarchitectural trade-offs or security trade-offs. A dif-ferent class of trade-off’s involves development activ-ities and tasks that are applied to build an application.This trade-off is about the emphasize that is placed onthe steps involved in building applications, and withwhat rigour and level of detail they are performed.
In a series of interviews that we have conductedwith seven Norwegian companies, we studied boththe companies’ development practises in general andwhat type of trade-offs the companies performed andwhat trade-off strategies that were applied in particu-lar (Ziemer and Stalhane, 2006). We found that thecompanies tend to specify their requirements in infor-mal ways, depending on the experience of the devel-opers and their domain understanding. Performing atrade-off with such practices is not easy since it is hardto assess the consequences that a potential choice fora given requirement might have on other importantrequirements. There are no hard numbers that canbe used in prediction or estimation models that canbe compared against measured real values at a laterstage. The only knowledge available is the qualitativeknowledge stemming from expert judgements and thestakeholders opinions and beliefs.
Part of performing a trade-offs is to share a com-mon understanding of the stakeholders priorities and
125
interests, and an understanding of the consequencesthese interests and needs have for the the application.When web applications are developed with the afore-mentioned development practices, there is a need foran approach to perform trade-offs that work withthese practices and that exploits the available knowl-edge.
In this paper we will present an approach to as-sess the consequences of a trade-off in an environ-ment with informal and tacit knowledge. This is donewith the available qualitative information. The rest ofthis paper is organised as follows: some characteris-tics of web application development are presented insection 2. Next, a number of trade-off opportunitieswill be explored in section 3, and some preliminaryguidelines for performing trade-offs are presented insection 4. An small example is presented in section 5.Finally, a discussion and conclusions are is presentedin section 6.
2 WEB APPLICATIONDEVELOPMENT
Web application development is characterized –among many other factors – by the informality of itsdevelopment practices, the available knowledge andby an ever changing development process. These fac-tors have an influence on how trade-offs can be per-formed in web application development. In our sur-vey of seven web developing companies (Ziemer andStalhane, 2006) the companies did not have a trade-off strategy and were in many cases not aware thatthey performed trade-offs.
Not being aware of performing trade-offs meansthat the freedom of choice is given away. This doesnot imply that the trade-off was a bad one – discussingwhat a good or appropriate trade-off is, is a seper-ate discussion – but the freedom to make a differentchoice is given away. This could be a choice thatwill enable the future success of the application, ora choice that will lead the future development into acertain direction. Any decision in based on a numberof assumptions. To make a good decision the assump-tion have to be correct and they have to be the rightset of assumptions to find the best decision. What isneeded is a trade-off strategy that helps developers toassess the appropriateness of a trade-off – and the as-sociated opportunities and risks – in a given situationor context. A trade-off results in a decision that caninfluence project and product risks and the productstrategy. A trade-off strategy should make perform-ing a trade-off an intentionally choice. This meansthat the pro and cons of a possible trade-off have to be
assessed and that the stakeholders have some expec-tations with respect to the result of that trade-off. Anyengineered system has to live with the consequencesof the decisions made during its design and construc-tion. Whether a trade-off will yield the expected resultor not can only be seen later. The decisions made fora system will restrict the future development of thesystem in such a way that it will be troublesome tochange a system in a way that is not foreseen or that isin conflict with the underlying assumptions of the de-cisions made so far (see (Lago and van Vliet, 2005)).Therefore, it is important to perform these trade-offsexplicitly, so that the information a trade-off is basedon, is documented for future use.
2.1 Informal Knowledge
Developing software is a knowledge intensive en-deavour. To build a software system that satisfies itsstakeholders it is necessary to understand not onlywhat functions the system shall perform, but alsothe need or motivation for this functionality, the con-text in with this functionality will be used, etc. Thisknowledge is found in existing systems, processes orpractices as well in individual, institutions and mate-rial structures (Hanseth, 2004).
The activities in a software process aim at find-ing this knowledge, elaborating on it and document-ing it. This requires that the documented knowledgeis unambiguous, so that it will have the same mean-ing to every stakeholder. Learning and conveying thisknowledge to other members of a community is adifficult task. The knowledge-sharing model – alsocalled the ”tacit-explicit model” – proposed by (Non-aka and Takeuchi, 1995) portrays this problem as acontinuous cycle. This cycle consists of socialisation,externalisation, combination and internalisation.
Looking to the practices found in web applicationdevelopment (Ziemer and Stalhane, 2006), we seethat a lot of knowledge is informal. Similar findingscan be found in other studies, too. One study foundthat the focus in web development is on the functionalrequirements. Non-functional requirements or qual-ity attributes are not specified with any rigor, partlybecause neither the developers nor other stakehold-ers are aware of them or tend to specify them asfunctional requirements based on previously experi-ence (N. Yusop and Lowe, 2006). It seems that whenTTM matters, also functional requirements are han-dled quite informally, by using analogies or examples.
The result of these practices is that the knowledgeis not documented unambiguous and hence, that it cannot be shared easily (Shull et al., 2004). This worksonly with experienced developers who have a good
126 CHAPTER 11. PAPER 5
understanding of the domain of the web application.There are other consequences to these practices:
• The knowledge about an application is frag-mented. Every stakeholder possesses a piece ofthe knowledge about an application. Document-ing this knowledge means bringing it together andlooking at the system from several viewpoints.This is, however, not easy when the knowledgeis fragmented.
• The knowledge is not unambiguous. It is open tointerpretation from every stakeholder, who tendsto look at it with his own goals and needs inmind. When discussing future aspects of an ap-plication, a common understanding is not neces-sarily present and have to be provided.
• When assessing the consequences of potential de-cisions, it is not possible to perform a formal anal-ysis that investigates several aspects of a decisionwith some rigour.
This is not to say that it is wrong to develop ap-plications with these development practices. Web ap-plications are developed in this way and many webapplications are quite successful. Still, it is importantto be aware of that there is a lot of tacit knowledgeinvolved. This is especially true when it comes to per-forming trade-offs.
2.2 Several phases in the life of a websystem
Web systems seem to go through several phases dur-ing their lifetime. There are several developmentphases, but there seem also to be several phases withrespect to the objective and purpose of a web systemin a competitive marketplace. The transition from oneto an other phase is a continous one. We present threephases, that are not necessary distinct. They are rathersnapshots at three different moments:
• The start-up phase. The web-page’s main pur-pose is to catch attention. Quite often it is justa brochure page, showing some info such as arti-cles for sale. Such pages are also often used to seeif it is possible to catch the attention of the mar-ket place. Often they contain simple games etc. Auseful concept here is ”Attention is everything –quality is nothing”.
• The mid-life phase. We have survived the fightfor attention and have added more feature to oursystem. The system is now important for our busi-ness although we could probably survive withoutit.
• The final phase. The system has grown and is nowcritical for our business. Most of our marketingeffort and a considerable part of our transactionsare now conducted via our web-pages. A long pe-riod – e.g. more than a week – where the web-page could not be accessed could be fatal for ourcompany.
These three phases and where we are in this pic-ture are important since it will decide the risk wetake when we change something. During the start-up phase the risk is low. We do not need any formalprocesses, neither for changes nor for change man-agement. Trade-offs can be frequent and done just tosee how they turn out. The decision process calledthe garbage-can process (Cohen et al., 1972) will doin most cases.
Things start to get more serious in the mid-lifephase. Our web-page now generates a considerableamount of business. This implies that bad changescan have serious consequences. We thus need a bettercontrol over the changes, even though there always isthe possibility to roll back to the previous version – atlest as a stop-gap action. Even so, a trade-off shouldnow be done with more considerations to possible badeffects. In addition to a well-defined trade-off processwe should now also consider possible bad effects –their probability and consequence together with pos-sible barriers. The risk assessment can, however, stillbe done at a rather informal level.
The option to do a roll-back in case of a bad deci-sion is dependent on a quick market feedback. Mostof the time, the only market feedback we get is thatour customers take their business elsewhere and whenwe discover this, it may be too late to do anythingabout it. It may be possible to avoid this problem byperforming frequent user satisfaction surveys.
All the problems that we can run into in the mid-life phase will also be important in the final phase –only more so. The risk is greater, not because theuncertainty of the outcome has changed but becausethe potential loss has increased. We need a formaltrade-off process, coupled with a formal risk assess-ment process.
2.3 Development process evolution
The change from one life phase to another is not with-out consequences for the applied development pro-cess. In order to face new risks that are introducedby this change, the development process has to adapt,either by introducing new activities or by changingexisting activities to handle the risk. The same is alsotrue for the opportunities that are associated with therisk. To exploit these opportunities the process has
127
to evolve. The authors of (Ramesh et al., 2002) statethat this evolution of a development process will re-quire different project cultures at different times.
Many companies developing web applications ina competitive environment have found their own suc-cess criterion. This can either be a technical or a non-technical factor. They apply development practisesthat support this success criterion and often establishan ad-hoc development process. As web applicationschange over times this can also change the success cri-terion. To stay in front in this competition the successcriterion has to evolve, too. This makes an assessmentof the success criterion and the present lifetime phase– with its risks and opportunities – necessary.
This process evolution does also effect the trade-offs and when it is appropriate to perform a trade-offor not. What was a good trade-off in an early life-time phase can be a non-appropriate trade-off in alater phase. Also here, an assessment of the trade-offversus the present lifetime phase is advisable.
2.4 Assessment of risks andopportunities
When discussing risk and consequences, it is all tooeasy to forget that changes can also bring opportuni-ties. It is the opportunities that drive the changes inweb applications, not the risks. Thus, there must bea reasonable relationship between opportunities andrisks. This implies that we need to focus on three fac-tors:
• The opportunities – what are we trying to achievewith our changes? This implies that we need toidentify opportunities and enablers.
• The risks – what can go wrong? This implies thatwe need to identify risks and barriers.
• The trade-offs – how can we balance the opportu-nities so that we choose the best combination?
The first two factors can be performed by do-ing a SWOT – Strength, Weakness, Opportunity andThreats – analysis. The last one – trade-offs – can bedone as suggested in (Ziemer et al., 2005).
Risks in our case stem from two sources – howmuch do your company depend on a properly workingweb system and what will be the reaction from yourcurrent customers. These two sources are strongly in-terdependent and are both related to your customers’cost of moving to another supplier. Thus, knowingthe current phase in a web system’s life is not enoughwhen we want to understand the risk of the system’sowner. If the customer incurs large costs if he wants tomove to another supplier, the risk related to changes issmall. This is for instance the case if you are a bank or
the homepage of apopular tourist destination. If you,on the other hand, sell a standard commodity in com-petition with many other suppliers, your risk is large.You will receive no complaints and no warnings; youwill just loose your customers at an alarming speed.By the time you understand that your latest releasewas a bomb, it is probably way too late for roll backor error correction.
The other important factor is your attitude to riskor you general risk handling strategy. We observe tworeactions to identified risks:
• This action, although it offers great opportunities,has so much risk related to it that we cannot do it.
• This action offers great opportunities. It has somerisks related to it but since the activity is impor-tant we will start looking for ways to prevent ormitigate these risks.
Both of these risk handling strategies are possible,but only the second one is viable in the long run. Thefirst one will inevitably lead to stagnation. Althoughit might be reasonable to assume that small compa-nies would use the first alternative while small com-panies would use the second one, no such correlationhas been observed.
In order to handle risk in a sensible way we needto do two things – we must:
• Understand our situation in the market – what willdissatisfied customers do – and which phase arewe in concerning the web system’s life – howmuch do we depend on our web system?
• Clarify our attitude to risky undertakings – avoid-ance versus control.
There exist standard approaches that can be usedin order to assess risk / barriers and opportunities /enablers. Our main message here is that you need toconsider these factors if you want to perform a sensi-ble tradeoff analysis.
The result of the risk and opportunity analysiswill, in our case, be a decision related to the choice ofprocess. Low risk implies the need for little or no for-mal process and a short TTM while a high risk impliesa formal process and a correspondingly long TTM.
2.5 Trade-off strategy
Performing a trade-off – whether we are aware of itor not – will have a lasting effect on the web applica-tion and can restrict future possibilities. As web appli-cations get larger and integrate an increasing numberof other systems, they also become harder to change.Small web applications can be re-engineered within
128 CHAPTER 11. PAPER 5
short time. This is not the case for larger web applica-tions. Thus, making the right decisions and perform-ing good trade-offs is an important task.
It has already been mentioned that developing webapplications with emphasise on TTM involves a lotof tacit knowledge, that can be expressed as expertjudgements and beliefs. Having a trade-off strat-egy that guides the development team in performingtrade-offs from iteration to iteration is helpful since it
• enables and supports the accumulation of explicitknowledge,
• helps to evaluate the results of trade-off deci-sions by tying together the decision and its con-sequences, thus enabling learning,
• keeps future options open by avoiding decisionsthat will prevent them.
By making this process explicit, the developersand the other stakeholders are aware of their decisionsand they can thus trace the results back to their deci-sion. They can formulate some future objective orgoal for the applications and then steer the further de-velopment into this direction. When they have to facesituations where it is not possible to make a decisioninto the preferred direction they will be aware of this,and the goal for the applications can – if necessary –be restated.
It makes no sense to talk about right trade-off ver-sus wrong trade-offs. A trade-off represents a choice,and a trade-off was good or wise with respect to astated goal or objective for the application. This in-volves in most cases the reconciliation of conflictinginterest among the stakeholders. A second conflictthat needs reconciliation is the one between short timeand long time term objectives. Therefore, a trade-offstrategy should include a way to
• reach a decision, that can be supported by allstakeholders,
• weight the consequences of a decision with re-spect to some stated overall goals or objectives.
In this way, it will be possible to make a decisionthat is as informed as possible – given that a lot ofinformation is informal and tacit – and to evaluate theresult of the decisions, and, when needed, to make thenecessary corrections.
3 TRADE-OFF OPPORTUNITIES
The objectives of the Websys project is to studyat trade-off’s between time-to-market and softwarequality. 3 examples of development practice trade-offs’ are presented in this section.
3.1 Requirement specification
Requirement specification trade-off has been ob-served in companies (Ziemer and Stalhane, 2006)as a trade-off between the level of detail in require-ment specification and time-to-market. Requirementsare specified informally, through oral communicationonly and sometimes based on anecdotes. When thelevel of detail in requirement specification is
• increased, the development process turns moreformal, and a longer TTM is to be expected. Amore detailed requirement specification enablesboth a more detailed test phase of the developedsoftware system, and a systematical way to man-age a wide stakeholder involvement into the re-quirement process.
• decreased, the development process turns lessformal, and a shorter TTM is to be expected.Communication with other stakeholders is opento more misunderstandings as every part in suchan discussion has his own interpretation of therequirements. Also, user involvement becomesmore limited.
The underlying assumption of this trade-off is thatthe developer(s) have detailed domain knowledge andthat verification of the requirements is not necessary.
3.2 Release planning
Release planning can be used as a trade-off betweentime-to-market and the amount of functionality and itsassociated return of investment to the stakeholders. Asystem is deployed in a number of successive deploy-ments, where each deployments consists of a set offunctionality. When the number of deployments are
• increased, time-to-market for every deploymentis decreasing. The end users perceived value ofa new deployment may be limited. Every de-ployment has also an associated cost, as it maygenerate technical problems or a peak in the userfeedback. On the other hand, a larger number ofsmaller deployments enables an earlier return ofinvestment for the stakeholders and opens up fora more flexible reaction to changes in the market.
• decreased, time-to-market for every deploymentwill be longer. The end users perceived value of anew release may be higher, as more functionalityis deployed, but it comes at a later time.
The underlying assumption of this trade-off is thatthe stakeholders agree to find a way of expressingthe value of a release. Dependent on the information
129
available, the value can either be expressed as a be-lief (as in (Ziemer et al., 2006)), as some scale or as amonetary value (see (Biffl et al., 2006)).
3.3 Testing
When testing there is a trade-off between test cover-age and time-to-market. When interviewing compa-nies developing web applications, we found compa-nies where the applications were tested by developerswith their expectation as a test criterion, and deployedupon reaching this criterion. When the test coverageis
• increased, the confidence to the software systemis increasing, both with respect to the proper im-plementation of the functional requirements andquality attributes. This will also result in a longerTTM. The number of user reported bugs and prob-lems should be low, and this should have an posi-tive impact on the user satisfaction.
• decreased, the confidence in the application is de-creasing. The developers do not know the prob-lems that might arise when deploying an applica-tion with a low test coverage.
The underlying assumption is that the developerexpectation to the application is in touch with themain user base, and that users are reporting the bugsand problems that they encounter.
4 QUALITATIVE ASSESSMENTSOF TRADE-OFFS
With no detailed information on important qualityfactors and development activities available, we stillhave the opportunity to use the qualitative informa-tion that expresses the stakeholders expert judgementand beliefs. This information can be used to assess atrade-off situation, in order to find a balance that willsatisfy most stakeholders.
Web Development is highly iterative. For ev-ery iteration it should be possible to improve theknowledge about an application. Quality factors canbe measured and judgements and beliefs can be en-hanced with quantitative data. Using qualitative as-sessments when no other information is available canhelp directing the efforts of collection more quantita-tive data.
There are several ways to collect and express qual-itative information. In the example in this paper weare using the SWOT analysis (Hill and Westbrook,1997). Another possible method is Impact Analysis
Trade-off
Application assessment
Application goals
(not mandatory)
Trade-off alternatives
1 – n
D e c i s i o n
Figure 1: A simple trade-off model
tables (Gilb, 2005). The most important thing, how-ever, is to start an exchange of viewpoints, beliefs andjudgements between all stakeholders or – if the num-ber of stakeholders is too big – from of the most im-portant stakeholders.
When performing a trade-off, the number of stake-holders involved is important. In many cases, it ispreferable to use a small number of key stakeholders.In the context described in this paper, it is importantto involve all stakeholders in order to elicit as muchknowledge as possible. In order to find a proper bal-ance it is crucial to have access to the whole picture,so that the trade-off can be performed based on thisinformation.
Reconcilement of conflicts – such as conflictingrequirements or conflicting priorities – is also an im-portant issues. One approach is to weight the im-portance of the stakeholders and thereby making theopinion of some stakeholders more important then theopinion of other stakeholders. A second approach isto find a consensus between all stakeholders. Thisworks only when the number of stakeholders is small.In a setting with a lot of tacit knowledge distributedamong the stakeholders, this approach is well suitedto get a common understanding of the trade-off situ-ation. This way, all stakeholders have to listen to theopinions of the other stakeholders.
Performing a trade-off involves the followingsteps (see the simple trade-off model in figure 1):• Decide on a list of alternative decisions or
choices – Each potential alternative is describedshortly. Make sure that each alternative is distinct.This is important in order to be able to assess theconsequences. When using textual descriptions itwill be hard to describe alternatives that are close
130 CHAPTER 11. PAPER 5
to each other in a distinctive way.
• Assess each alternative – This steps includeschoosing a method for the assessment. In the con-text of web development it is important to use amethod that is easy to learn and intuitive to use.Thereafter, each alternative from the previous stephas to be assessed.
• Assess the lifetime-phase – The risks and oppor-tunities of the web applications lifetime phase isassessed using the same method as in the previ-ous step.
• Weight the risks and opportunities and find asuitable action – Compare the risks and opportu-nities of both assessments, and discuss how theymight interfere with each other. Decide on theaction with the most preferred results. This canbe done by identifying 3 possible relationships (asshown in figure 2):
– a positive relationship between the trade-offand the web applications lifetime phase. Thisis the case when a strength from the trade-offis helping avoiding the threat from the systemphase.
– a neutral relationship between the applicationslifetime phase and the trade-off.
– a negative relationship between the trade-offand the applications lifetime phase. This is thecase when a threat from the trade-off can in-crease the threat from the applications lifetimephase.
5 EXAMPLE
In this section we present an example how antrade-off can be performed. In this example we usea trade-off on the requirement specification practices.Three candidate alternatives for this practise are as-sess using the the SWOT analysis. This analysis isassessed against a SWOT analysis of the web appli-cations situation. An example of how a trade-off al-ternative is assessed against the lifetime situation isshown in figure 2. The two SWOT analysis’s for theremaining trade-off alternatives are shown in figure 3.For each trade-off alternative a similar trade-off anal-ysis – using a QFD-like matrix – has to be performed.
The stakeholders have to assess the trade-offbased on the information in three SWOT analysis’sand three trade-off analysis’s to decide what is the ap-propriate trade-off in this situation.
S W O T
Can
test
new
func
tiona
lity
Littl
e or
no
test
ing
Web
app
licat
ion
is ea
sely
chan
ged
User
s do
not
retu
rn d
ue to
po
or q
uality
rapi
dly
Flexible deployment rutines – + –User participation + –Dependent on developers domain knowledge – +No verification and validation of requirements –Attracting new users +Getting early user feedback +
T Customers are not satisfied –
Appl assessment
Trad
e-of
f alte
rnat
ive
1
S
W
O
Figure 2: Trade-off evaluation for Trade-off alternative 1
6 DISCUSSION ANDCONCLUSIONS
The motivation for the work presented in this paper isto create an awareness about the trade-offs that areperformed during development of web applicationsand to provide a way of assessing trade-off situationsin order to keep the freedom of choice that is associ-ated with trade-offs. In the field of web applicationdevelopment – as it has been described in this paper– this has to be done in a way that brings togetherall stakeholders, that collects all information that isavailable, and that let the stakeholders assess it in away that helps them to make a good decision.
The decision on which method to use when col-lecting qualitative information from stakeholders isimportant. The decision is not on the pro’s and con’sof the method alone, but also on which method isknown to the stakeholders and enable them in bring-ing together their tacit knowledge. In our view, bring-ing the stakeholders together and let them share theirknowledge, beliefs and opinions, is the most impor-tant part of our approach. The decision on whichmethod to use for this purpose comes only second tothis. We have chosen to use the SWOT analysis, be-cause in our experience it is widely known and easyto use for professionals in developing companies.
Other methods to collect and organise qualitative
131
Threats – slow reaction to changes in market (ex. to a new competitior)
SWOT 3, Trade-off analysis 3: Long TTM and detailed requirement specification
Opportunities – carefully changing a system and trying to hold on the customer base, mananing user satisfaction
Strenght – easy communication over requirments, Documented knowledge about user satisfaction
Weakness – harder to get mainstream user involvement (requires user groups), harder to change requirements (it takes time)
Threats – too long respons when market is changing rapid
Opportunities – can respond to market changes within some time with a better understood solution
Weakness – no reliable verification and validation of requirements
Strenght – better documentation of requirements and therefore more knowledge
SWOT 2, Trade-off analysis 2: Medium TTM and short requirement specification
Figure 3: SWOT analysis for trade-off alternatives 2 and 3
information can be used. Other methods that couldhave been used are Impact Estimation tables (Gilb,2005), Affinity diagrams (Straker, 1995) and QFD(Akao, 1990). The advantage of using Impact Estima-tion tables is that it can be suited to the aspects that areof interest for a project. However, as the informationthat is collected is qualitative, it is important that nottoo many aspects are involved. The SWOT analysisuses four aspects, which should be relevant to mostprojects.
In the future this approach should be validated em-pirically with respect to its performance – how muchqualitative information can be collected and analysedin a reasonable amount of time. Other interesting di-rections for future research is to study the effect ofthis approach on knowledge sharing and establishinga common understanding among stakeholders.
In this paper we have shown how trade-off on de-velopment practices are performed in web applicationdevelopment, and presented an approach to performtrade-offs on development practices with qualitativeinformation. The objectives behind this approach is tocreate an awareness for trade-off situations that other-wise will go unnoticed, thereby leaving out opportu-nities to reach the objectives of a web application.
REFERENCES
Akao, Y. (1990). Quality Function Deployment. Productiv-ity Press.
Biffl, S., Aurum, A., Boehm, B., Erdogmus, H., andGrunbacher, P., editors (2006). Value-Based SoftwareEngineering. Springer-Verlag Berlin Heidelberg.
Cohen, M. D., March, J. G., and Olsen, J. P. (1972). Agarbage can model of organizational choice. Adminis-trative Science Quarterly, 17(1):1–15.
Gilb, T. (2005). Competitive Engineering: A Handbook ForSystems Engineering, Requirements Engineering, and
Software Engineering Using Planguage. Butterworth-Heinemann Ltd.
Hanseth, O. (2004). Knowledge as infrastructure. InAvgerou, C., Ciborra, C., and Land, F., editors, TheSocial Study of Information and Communication Tech-nology, pages 103–118. Oxford University Press.
Hill, T. and Westbrook, R. (1997). Swot analysis: it’s timefor a product recall. Long Range Planning, 30(1):46–52.
Lago, P. and van Vliet, H. (2005). Explicit assumptionsenrich architectural models. In Proceedings of the27th international conference on Software engineer-ing, pages 206 – 214.
N. Yusop, D. Z. and Lowe, D. (2006). The impacts of non-functional requirements in web system projects. InProceeding of European and Mediterranean Confer-ence on Information Systems.
Nonaka, I. and Takeuchi, H. (1995). The Knowledge Creat-ing Company. Oxford University Press.
Ramesh, B., Pries-Heje, J., and Baskerville, R. (2002). In-ternet software engineering: A different class of pro-cesses. Annals of Software Engineering, 14:196–195.
Shull, F., Mendonca, M. G., Basili, V., Carver, J., Maldon-ado, J. C., Fabbri, S., Travassos, G. H., and Ferreira,M. C. (2004). Knowledge-sharing issues in experi-mental software engineering. Empirical Software En-gineering, 9(1-2):111–137.
Straker, D. (1995). A Toolbook for Quality Improvementand Problem Solving. Prentice Hall.
Ziemer, S., Sampaio, P., and Stalhane, T. (2006). A deci-sion modelling approach for analysing requirementsconfiguration trade-offs in time-constrained web ap-plication development. In Proceedings of SEKE 2006.
Ziemer, S. and Stalhane, T. (2006). Web application de-velopment and quality - observations from interviewswith companies in norway. In Proceedings of Webist2006.
Ziemer, S., Stalhane, T., and Sveen, M. (2005). Trade-offanalysis in web development. In 3-WoSQ: Proceed-ings of the third workshop on Software quality, pages70–75. ACM Press.
132 CHAPTER 11. PAPER 5
Chapter 12
Paper 6
Sven Ziemer and Ilaria Canova Calori:Knowledge sharing through a simple release planning method for webapplication developmentIn Daniel Cooke et al. (Eds.): Proc. Nineteenth International Conferenceon Software Engineering & Knowledge Engineering (SEKE 2007), 9–11 July2007, Boston, USA. Published by Knowledge System Institute Graduate School,Skokie, IL, USA. ISBN 1-891706-20-9, p. 686-691.
133
134 CHAPTER 12. PAPER 6
Knowledge sharing through a simple release planning methodfor web application development
Sven Ziemer and Ilaria Canova Calori
Norwegian University of Science and Technology, NO-7491 Trondheim, NorwayE-mail: {svenz|canovaca}@idi.ntnu.no
Abstract
Web application development is – under circumstancessuch as a strong emphasis on time-to-market – charac-terised by the use of informal and ad-hoc development prac-tices and a lot of tacit knowledge. A recently proposedrelease planning method, that has been developed for theuse in these environments, aims at bringing stakeholders to-gether to share their knowledge and to decide on a configu-ration for the next release of a web application that satisfiesall stakeholders. An initial experiment to evaluate the re-lease planning method indicates that the method does wellon supporting a consensus and stakeholder satisfaction, butit does not contribute to knowledge sharing between stake-holders in small projects. To study the effect that learningand experience from using the method several times has onknowledge sharing and understanding among members ofa small development team, and to study the effect that achanged communication pattern has on knowledge sharingand understanding.
1 Introduction
Web applications that are developed with a strong focuson short time-to-market use development practices that canbe characterised as informal, chaotic, rush-to-market and adhoc [7] [11]. As a result of these practises, the knowledgethat is available to make a decision regarding the develop-ment activities, is spread between all stakeholders, and istacit, informal, and in many cases qualitative and based onthe stakeholders opinion and beliefs. To make a decisionthat takes all available knowledge into consideration, it isnecessary to bring the stakeholders together and to makethem share their knowledge.
Release planning is an important decision for a soft-ware project. Release planning for web application devel-opment must balance important factors such as a short time-to-market, that is important to give an early return on theinvestments that have been made into the web application,
and innovative and attractive functionality that aims at at-tracting the attention of potential users. In a recent paper weproposed a release planning method that brings together allavailable knowledge from the stakeholders and make themreach a consensus based decision on the next release [10].
Release planning is an important decision for a softwareproject [2]. Moreover release planning for web-applicationdevelopment must balance important factors such as a shorttime-to-market, that is important to give an early return onthe investment, and innovative and attractive functionalitiesthat aims at attracting the attention of potential users. Manyworks refer to empirical studies on planning of software re-leases [4] [5] but there are not specific works about webapplication release planning. In a recent paper a releaseplanning method has been proposed to bring together allavailable knowledge from the stakeholders and make themreach a consensus based decision on the next release [10].
To study the effect of this method on small developmentteams, we conducted an experiment with students at ouruniversity [9]. Based on the analysis of the results fromthis experiment we decided to repeat the experiment twotimes in two different settings, in order to study the effectof experience and learning on knowledge sharing and un-derstanding and the the impact of changing the instructionson the communication pattern of a method with respect toknowledge sharing and understanding.
This paper describes the details of the experiments andreports the results and lessons learned from them. The pa-per is organised as follows: We give a more detailed back-ground for our motivation to repeat the initial experiment insection 2. The experiment planning is given in section 3,and the results are shown in section 4. A discussion of rel-evant topics is presented in section 5, and conclusions aregiven in section 6.
2 Background
To make good decisions, the decision makers must haveaccess to the knowledge that is available about the issue at
hand. When the knowledge leaves out certain aspects, thiswill be reflected in the decision and in its results. Whena lot of the knowledge that a decision has to be based onis tacit, we need a different technique to share knowledgeand create a common understanding than we need when theknowledge is explicit.
Web development projects involve often tacit knowl-edge, that is spread between all stakeholders. A recentlyproposed release planning method, aims at bringing to-gether stakeholders, make them share their knowledge andcreate a common understanding, and to make a decision onthe next release for a web application development project.To do so, the stakholders should perform a qualitative as-sessment on the expected return on value for every require-ment and candidate configuration. They should use a a sim-ple scale with five points, where ”1” stands for no valueand ”5” for very high value. In addition, each stakeholdershould give a short justification for his assessment. Basedon the assessment requirements and candidate configura-tions are prioritised, and a final decision on the next releaseis made [10].
We decided that the proposed release planning methodshould be evaluated and validated experimental. We there-for planned an experiment to observe the effects the pro-posed method had when used by small development teams.The factors we decided to study were: knowledge shar-ing, understanding, prioritisation, reaching a consensus, andstakeholder satisfaction. Due to limited resources the ex-periment was planned as an off-line, student experiment,with one factor and with two treatments. The students weredivided into a treatment group and a control group. Bothgroups had to participate in a role play and solve a releaseplanning task. The treatment group used the proposed re-lease planning method whereas the control group solved thetask in an ad hoc manner. After the experiment all studentshad to fill in a post-experiment questionnaire that we usedto analyse the effect of the release planning method on theaforementioned factors. The results showed that the treat-
ment group did perform better on prioritisation, reaching aconsensus and stakeholder prioritisation, whereas the con-trol group performed better on knowledge sharing and un-derstanding.
The results were not as expected and we could there-for not reject the null hypothesises stating that the releaseplanning method did not perform different for knowledgesharing and understanding. When analysing and discussingthe results we found two possible explanation for the unex-pected results:
• When introducing a new method, the focus of the de-velopers is on understanding the method and to use itcorrectly. The problem to be solved will consequentlyreceive less focus, and the communication about theproblem between the team members will be less inten-sive. With more experience in using the method, theunderstanding of the method is growing, and the de-velopers will move their focus to the problem. Thecommunication between the team members will alsofocus more on the problem again. The control groupshad not to learn a new method and could use all theirattention on the task.
• Another explanation is based on the communicationpattern that was introduced with the instructions onhow to use the release planning method and the hand-outs supporting the use of the method. Every sub-ject on the treatment groups received a form wherehe could write down his notes on the problem duringthe experiment. This was done for the team membersconvenience, but seemed to have the unintended side-effect of disabling the communication between groupmembers (see figure 1 A).
Our conclusions were that we would expect the treatmentgroups to perform better on knowledge sharing and under-standing when running the experiment again and observingthe effect (1) of the team members experience in using themethod, and (2) of changing the communication pattern thatis introduces to the treatment groups by the instruction theyreceive. The changed communication pattern is shown infigure 1 B, and aims to enable as much oral communicationbetween the group members as possible.
We decided to investigate these explanations further, andperformed four new experiments. The initial experiment isdenoted E1 (see figure 2). Two experiments – E 2.1 and E2.2 – are replications of the first experiment. Their resultswill – together with the results from the first experiment E1– let us study the effect that learning and experience hason knowledge sharing and understanding. For the two otherexperiments – E 3.1 and E 3.2 – we will change the commu-nication pattern in our instructions. Only the group leaderwill receive a form for taking notes, and the group members
136 CHAPTER 12. PAPER 6
Communication Pattern A
Communication Pattern B
Scenario 1Questionnaire 1
Scenario 3Questionnaire 2b
Scenario 2Questionnaire 2a
E 1
E 2.1 E 3.1
E 3.2E 2.2
H0b
H0a H0a
H0b
H0a
Figure 2. The organisation of experiments.
have to give their comments to the group leader orally. Theresult of this experiment can be compared to the results of E1, E 2.1 and E 2.2, to study the effect that a changed instru-mentation have on knowledge sharing and understanding.
3 The experiments
We planned the four experiment together. Figure 2shows how the experiments are related and the factors thatchanged during the experimentation: The communicationpattern, the scenario and the questionnaire.
Experiment definition The experiment definition is(using the template from [8]):
Object of study: The release planning method [10].Purpose: Study the effect of learning and a changedinstrumentation on knowledge sharing and understanding.Quality focus: The stakeholders increased shared knowl-edge and understanding.Perspective: The development teams point of view Con-text: Experiment with industrial economy and computerscience students in their 3rd year of study, forming smallgroups of 3 or 4 members. The study is conducted asMulti-test within object study.
Experiment planning The four experiments have beenplanned as replications of the initial experiment E1. Werefer to [10] for a detailed description of the experiment set-ting. However, even when the experiments are planned asreplications of E1, we had to make some small changes tobe able to reach the goal for these experiments:
• As indicated in section 2, we changed the communi-cation pattern that we introduce in our instructions onhow to use the release planning method and the hand-outs that each group receives. These changes apply toexperiments E3.1 and E3.2. We recruited also a dif-ferent student group for these two experiments. Thestudents were new to the experiment and the proposedrelease planning method. Experiments E 2.1 and E2.2are unchanged from E1, and where run with the samecommunication pattern, handouts, and student group.
• We created two new scenarios to be used in the roleplays that take place in each experiment. ExperimentE2.1 and 3.1 used the first scenario, and experimentE2.2 and 3.2 used the second scenario. Both scenariosare from the development of web applications. Thefirst version of the new applications have to be de-ployed within three weeks, and the task of the groupsis to find a release that satisfies all stakeholders.
• We used the same post-experiment questionnaire asin the first experiment. Based on the leasons-learnedfrom the fist experiment, we added four questions ina new group of questions on communication, to get abetter impression on the communication style in eachgroup. In addition we changed the order of the ques-tions in the questionnaire for experiments E2.2 andE3.2, and changed the wording of approximate 30 %of all questions to a negative wording. Otherwise, thequestionnaire remained unchained.
Hypothesis formulation The hypothesises of the experi-ments are as follows:
H0a: There is no effect on knowledge sharing and under-standing from learning when the users gain experience inthe release planning method.H1a: There is a effect on knowledge sharing and under-standing from learning when the users gain experience inthe release planning method.H0b: The changed communication pattern used in the re-lease planning method has no effect on knowledge sharingand understanding.H1b: The changed communication pattern used in the re-lease planning method has an effect on knowledge sharingand understanding.
4 Results
The experiments collect data by means of a post-experiment questionnaire, filled in be every participant orevery experiment he participated in. The questionnaireswere divided into six groups – knowledge sharing, un-derstanding, prioritisation, consensus reaching, stakeholder
137
Figure 3. Boxplot for question 6 for E1, E3.1.E2.2 and E3.2
satisfaction and communication – and contained a total of28 issues. For each issue, a statement is given together witha five-point Lickert scale. Each respondent used the Lickertscale to express his degree of agreement with the statement,by checking of one of the following: strongly disagree, dis-agree, neutral, agree and strongly agree.
The questionnaire were analysed group for group, andeach group were analysed question by question. The nullhypothesis was tested for each group of questions. To rejectit we required that the results for one sample were signif-icantly better for at least 50 % of the questions than theresults of the sample they compared to. We considered theresults to be significant for p-value less than 0.1.
4.1 Hypothesis A – Learning
What is the effect on knowledge sharing and under-standing when the subjects in the treatment groups gainsome experience in using the release planning method insmall teams, and start to learn from previous experience?Are there any differences between the treatment groups us-ing the proposed release planning method and the controlgroups solving the task in an ad hoc style. To answer thesequestions we analyse the differences between the resultsfrom experiments E1–E2.1, E1–E2.2 and E3.1–E3.2.
Knowledge sharing – Six questions
• E1 – E2.1 The results for the treatment groups in E2.1are better for 6 out of 6 questions compared with theresults in E1. For one question, the difference is signif-icant. For the control groups the differences are reallysmall and sometimes even negative – i.e. the results inE2.1 are worse than in E1.
• E1 – E2.2 The results for the treatment groups in E2.2are better in 4 out of 6 questions compared with the
results in E1. For two questions, the difference is sig-nificant. For the control groups the results are worse on5 out of 6 questions, and this difference is significantin two cases.
• E3.1 – E3.2 The same observations are valid for theresults in E3.1 and E3.2. The treatment groups showbetter results in 4 out of 6 questions, while the controlgroups show better results only in 1 out of 6.
The results are not strong enough to reject the null hy-pothesis with the criteria we have defined for this experi-ment. A boxplot for the results of question 1 for the treat-ment groups from these experiments are shown in figure 3.
The results show that the treatment groups improvedtheir performance on knowledge sharing as they gain expe-rience in using the proposed method and start to learn fromprevious experiences. The control groups do not improvetheir performance on knowledge sharing. While the controlgroup did perform better on knowledge sharing on E1, thishas now changed, and the treatment group performs betterin both E2.2 and E3.2 In our opinion, using the release plan-ning method in this situation helps the group to organise andshare their knowledge, and help them to learn from it.
We consider the results as a strong indication that theproposed release planning method is enabling and support-ing knowledge sharing, and conclude that a little experienceand learning is needed to take advantage of the release plan-ning method to improve knowledge sharing over the levelthat exists when working ad hoc style.
Understanding – four questions
• E1 – E2.1 The results for the treatment groups in E2.1are better in 1 out of 4 questions compared with the re-sults in E1, and in this case the difference is significant.For the control group the results in E2.1 are better in 2out of 4 questions, but the difference is not significant.
• E1 – E2.2 The results for the treatment groups in E2.2are better in 1 out of 4 questions compared with the re-sults in E1, and in this case the difference is significant.For the control group the results in E2.2 are worse in 4out of 4 questions, and the difference is significant in3 of them.
• E3.1 – E3.2 The results for the treatment groups inE3.2 are worse in 4 out of 4 questions compared withthe results in E3.1, and on 3 of them the difference issignificant. For the control group the results in E2.2are worse in 4 out of 4 questions, and the difference issignificant in 4 of them.
The null hypothesis can not be rejected based on the re-sults. The results show that understanding is not affected
138 CHAPTER 12. PAPER 6
by experience and learning. The results are the same forboth the treatment groups and the control groups. Our con-clusions is that understanding is influenced by other factorsthat experience and learning. The results in E2.2 and E3.2might be affected by the fact that 3 out of 4 questions wherenegatively rephrased (see also section 5) and it seems thatparticipants tend to be more neutral in this case.
4.2 Hypothesis B
What is the impact of changing the communication pat-tern for the treatment group on knowledge sharing, under-standing and communication? We compare the results fromexperiments E1–E3.1 and E2.2–E3.2. The subjects of ex-periment E2.2 are participating in their third experiment,whereas the subjects of experiment E3.2 are participatingin their second experiment. Hence, the subjects of experi-ment E2.2 are more experienced than the subjects of E3.2.However, comparing the results from E3.2 with E2.1 intro-duces other problems with respect to the questionnaire andthe scenario. This will be discussed in a section 5.
Knowledge sharing
• E1 – E3.1 The results for the treatment groups in E3.1are better in 5 out of 6 questions on knowledge sharingcompared with the results of E1. For two questions, thedifference is significant. For the control groups, thereis almost no difference in the results of E3.1 and E1,and none of the difference found have significance.
• E3.2 – E2.2 The results for the treatment groups inE3.2 is slightly better for 5 out of 6 questions. How-ever, the difference is not significant for any question.The control group has similar results with 4 out of 6questions with a slightly better result for E3.2 com-pared with E2.2. However, the treatment group hasbetter results for E3.2 than the control groups.
None of the results are so strong that we can reject thenull hypothesis. The results show that the treatment group isimproving their performance on knowledge sharing at bothoccasions. We conclude from the results that the improve-ments stemming from a changed communication pattern de-cline with more experience on how to use a new method.In other words, to support the communication when intro-ducing a new method in a context where communication iscritical, it is important to promote communication. Afterthe method has been internalised this is not necessary anymore.
Understanding
• E1 – E3.1 The results for the treatment groups in ex-periment E3.1 are better then for E1, with two of the
differences being significant. The control groups havebetter results in E3.1 then in E1 for 3 out of 4 question,with one significant difference.
• E3.2 – E2.2 The results for the treatment groups arebetter in 3 out of 4 questions (with one significant dif-ference). The control groups have the same result.
The results show that the null hypothesis can be rejectedfor the treatment groups on E1 – E3.1. Understanding canbe improved by instructing group members to communicatemore when a new method is introduced. When the groupmembers gain some experience in the methods, there seemto be other factors that influence understanding more thanlearning (see above) and a more explicit communicationscheme.
5 Discussion
There are some issues about the experiment that have tobe considered in more detail.
The first issue is the questionnaire that was used for ex-periments E2.2 and E3.2 (shown as questionnaire 2b in fig-ure 2). To avoid that the respondents reuse their responsefrom the previous experiment and try to be conform by giv-ing the same answer, we decided to regroup the questionsand change around 30 % of the questions into a negativeform. The original statement ”The release chosen by mygroup was my preferred release” was changed into ”The re-lease chosen by my group was not my preferred release”. Tocompare the results of these negative worded questions withthe results of the corresponding positive worded questions,we had to change the value of these questions, assuming thatthe response ”Disagree strongly” on a negative question isthe same result as ”Agree strongly” on a positive question.However, when we compare the results for questions with anegative worded variant in questionnaire 2b, we see that thedifferences between the questions are bigger than for otherquestions. It seems that it is more easy for respondents tosay that they agree than to say that they do not agree. Incases where the respondents feel that they do not agree, itseems that they prefer to choose ”Neutral” or ”Disagree”,but not ”Strongly disagree”. This effect – unknown to us atthe time we planned the experiment – has been described –among others – in [6].
This effect disturbs the comparison between experimentsE1–E2.2 and E2.1–E2.2. We assume that without this ef-fect, some of the differences would have been stronger, andmaybe even significant. We have decided not to removethese questions, as they still show how the difference be-tween the treatment and control groups changes over time.This difference has a positive tendency for both knowledgesharing and understanding for the treatment groups.
139
Another issue are the impact of the scenarios on the re-sults. The scenarios had some variations, such as the totaldevelopment effort that could be included into a configura-tion, the total number of requirements in a scenario, and the”size” of the requirements, i. e. the estimate of developmenteffort. These factors have an impact on how easy a group ofstakeholders think it is to find a final release. In an interviewwith 10 students that participated in experiments E3.1 andE3.2, they told us that each decision had its own challenges,and that they found difficult aspects in both scenarios.
In the comparison with experiment E2.2–E3.2 the stu-dents of E2.2 are more experienced as this is their thirdexperiment, while it is only the second experiment for thestudents of E3.2. The reason for comparing these two ex-periments is that it involves two experiments with the samescenario and questionnaire. In our opinion it is better tocompare two experiments with the same scenario and ques-tionnaire – and accept the difference in experience – than tocompare two results that were collected with different ques-tionnaires.
Our last issue is the validity threat posed by the use ofstudents in the experiments. Students are not as experiencedas professionals, and the experiment may therefore not be asrealistic as it would be if professionals were used. However,other studies using both students and professional indicatethat the relative difference in the results between studentsand professionals for two or more experiments are close [1][3]. Also, it would be very resource demanding to acquirea sample of professionals of the same size as used in ourexperiment. The experiments reported in this paper could,however, be expanded with one or more case studies usingthe proposed release planning method in a more realisticcontext, i.e. with professionals.
6 Conclusions and further work
In this paper we have presented the results of four ex-periments that we conducted to further investigate a releaseplanning method [10]. An initial experiment [9] gave ussome answers about the use of the method, but also somenew questions. In the experiments described in this paper,we addressed two issues: what is the effect of experienceand learning on factors such as knowledge sharing and un-derstanding, and how will a changed communication pat-tern impact these factors.
The results of the experiments show that knowledgesharing is positively influenced by both experience andchanged communication pattern. Understanding is im-proved little by changing the communication pattern and notfrom experience. The results indicates that the communica-tion pattern introduced in this experiment will accelerate theimprovement in the performance on knowledge sharing.
References
[1] E. Arisholm and D. Sjøberg. Evaluating the effect of adelegated versus centralized control style on the main-tainability of object-oriented software. IEEE Transac-tion on Software Engineering, 30(8):512–534, 2004.
[2] P. Carlshamre. Release planning in market-drivensoftware product development: Provoking an under-stadning. Requirement Engineering, 7(3):139–151,2002.
[3] J. Carver, L. Jaccheri, S. Morasca, and F. Shull. Is-sues in using students in empirical studies in softwareengineering education. In Proceedings of the NinthInternational Software Metrics Symposium (MET-RICS’03), 2003.
[4] G. Du, J. McElroy, and G. Ruhe. A family on em-pirical studies to compare informal and optimization-based planning of software releases. In ISESE ’06:Proceedings of the 2006 ACM/IEEE internationalsymposium on International symposium on empiricalsoftware engineering, pages 212–221, 2006.
[5] J. Momoh and G. Ruhe. Release planning processimprovement – an industrial case study. SoftwareProcess: Improvement and Practice, 11(3):295–307,2006.
[6] H. W. O’Neil. Response style influence in public opin-ion surveys. The Public Opinion Quarterly, 31(1):95–102, 1967.
[7] B. Ramesh, J. Pries-Heje, and R. Baskerville. Internetsoftware engineering: A different class of processes.Annals of Software Engineering, 14:196–195, 2002.
[8] C. Wohlin, P. Runeson, M. Host, M. C. Ohlsson,B. Regnell, and A. Wesslen. Experimentation in Soft-ware Engineering: An Introduction. Kluwer Aca-demic Publishers, 2000.
[9] S. Ziemer and I. C. Calori. An experiment with a re-lease planning method for web application develop-ment. In Submitted to Eurospi 2007, 2007.
[10] S. Ziemer, P. Sampaio, and T. Stalhane. A decisionmodelling approach for analysing requirements con-figuration trade-offs in time-constrained web appli-cation development. In Proceedings of SEKE 2006,2006.
[11] S. Ziemer and T. Stalhane. Web application develop-ment and quality - observations from interviews withcompanies in norway. In Proceedings of Webist 2006,2006.
140 CHAPTER 12. PAPER 6
Chapter 13
Paper 7
Sven Ziemer and Ilaria Canova Calori:An experiment with a release planning method for web applicationdevelopmentIn Pekka Abrahamsson, Nathan Baddo, Tiziana Margaria, and Richard Mess-narz (Eds.): Proc. 14th European Conference on Software Process Improve-ment (EuroSPI’2007), 26–28 September 2007, Potsdam, Germany. Published inSpringer LNCS 4764, ISBN 978-3-540-74765-9, ISSN 0302-9743, p. 106–117.
141
142 CHAPTER 13. PAPER 7
An experiment with a release planning methodfor web application development
Sven Ziemer and Ilaria Canova Calori
Department of Computer and Information Science,Norwegian University of Science and Technology,
Abstract. Web application development is under certain circumstancessuch as a strong emphasis on time-to-market characterised by the usageof informal and ad-hoc development practices and a lot of tacit knowl-edge. Here we present an experiment that has been carried out in orderto evaluate a recently proposed release planning method for web appli-cation development. This method aims at bringing stakeholders togetherto share knowledge and to decide on a configuration for the next releasethat satisfies all stakeholders. The method has been evaluated in termsof its effect on factors such as knowledge sharing, understanding, supportto reach a consensus and stakeholders satisfaction.
1 Introduction
Developing software systems is a knowledge intensive endeavour, and the qualityof a software application is limited by the quality of the knowledge that is avail-able to the development team. Improving the amount and quality of knowledgeis therefor a central activity in software process improvement. The first refers tohaving as much knowledge as possible from multiple stakeholder’s, and the laterrefers to having more precise knowledge. Depending oh the context of a softwaredevelopment project, the knowledge about the development effort will either beexplicit or implicit, and quantitative or qualitative. In the case of developmentprojects that apply informal and ad-hoc development practises, and that have astrong focus on time-to-market the knowledge is mostly implicit and qualitative.Improving both the amount and quality of knowledge here is as important asfor all software development projects. This can be achieved by using the avail-able tacit knowledge. Bringing the involved stakeholders together and sharingthe available knowledge to create a common understanding, is an improvementof knowledge, and a potential first step to systematic software process improve-ment activities. Sharing knowledge in such an environment involves sharing theopinions, beliefs and expert judgements of the stakeholders.
In a recent paper [1] we presented a method for release planning to be used inweb application development projects under such conditions. In order to validatethe proposed release planning method and to learn how it works in development
143
projects, this method was tested empirically. Among several options, it was de-cided to use a student experiment. This paper presents the experiment, and thelessons learnt.
The rest of the paper is organised as follows: The background from webapplication development for the release planning method is given in section 2and related work is shortly described in section 3. The next sections, experimentdefinition and goal (section 4), experiment planning and operation (section 5),and the questionnaire and experiment analysis (section 6) are presenting theexperiment. The results from the experiments are shown in section 7, and adiscussion and further work is presented in section 8. Finally, conclusions aregiven in section 9.
2 Background
The way web applications are developed depends on such factors as the applica-tions maturity, the number of returning users and the competition with similarweb applications. Some web applications have a strong focus on time-to-market,that in some cases can be described as a rush-to-market focus. The developmentof such web applications can be best described by the usage of informal devel-opment practices, the use of implicit and tacit knowledge, by the involvementof a group of stakeholders that have diverse and possibly conflicting interests,and the iterative nature of the development activities [2]. That is not to say thatthese characteristics can not be found elsewhere, but when the main focus of thedevelopment efforts of a web application is on rush-to-market, they constitute aspecial combination.
Requirement specification and release planning for web application develop-ment is in many cases done in an ad-hoc way, with no assessment of the potentialconsequences, of selecting one configuration over another. The focus is on theshort-time satisfaction of the stakeholders and not on having a long-time strat-egy. Even if this may be a common approach with acceptable results, it canresult in unbalanced decisions – satisfying only one stakeholder – that turn outto be bad for the web application. In order to be able to assess the consequencesof candidate configurations and to prioritise them, a release planning method,that is using the tacit knowledge of all the stakeholders as input, is needed.
The release planning method that is used in this paper is described in [1]. Inprojects with the aforementioned development practices, stakeholders knowledgeis limited to their own involvement and interests in the project (see part A offigure 1). The proposed release planning methods aims at bringing togetherthe knowledge from all stakeholders in a development project and at creatinga common understanding among stakeholders reflecting the knowledge of eachindividual stakeholder. This shared knowledge can be thought of as the union ofall stakeholders knowledge (see part B of figure 1).
The communication and knowledge sharing that is expected to be intro-duced into a development team by the proposed release planning method will
144 CHAPTER 13. PAPER 7
S1
S3
S4
S2
S1
S3
S4
S2
A B
Fig. 1. The diverse knowledge of stakeholders
thus contribute to increase both the amount and quality of knowledge for thestakeholders that are involved in the decision making on the next release.
3 Related work
There has been a lot of research into release planning. There is an increasingnumber of research planning methods published, such as [3], [4], [5] and [6]. In[7] seven existing release planning methods are evaluated with respect to tendimensions, such as stakeholder involvement, prioritization mechanism, resourceconstraints, and tool support. None of these methods is designed to use thestakeholders belief and opinion as only input.
There are a few empirical studies on release planning methods. In [8], two casestudies of a release planning process are presented. In [9] and [10] the authorspresent a family of empirical studies to compare ad-hoc based and systematicrelease planning. The focus of these experiment is on confidence, understandingand trust related to the research planning practise applied. In [11] the authorspresent an industrial case study to stress the advantages of employing an in-telligent decision support system compared with an ad hoc approach. With theproposed system the satisfaction of the stakeholder priorities is maximized, andthe time spent in pre-planning activities is reduced and channeled on the deliveryprocess to include more requirements in a release.
There are also some studies on knowledge sharing. In [12] it has been pointedout how important is for multidisciplinary team members working on complexproblems to share their knowledge in order to cope with different prospectiveson the problem and different individual knowledge and skills. The stakeholdersare therefore encouraged to make their beliefs and values explicit. Another study
145
has investigated the motivation factors for knowledge sharing to take place [13].The result of this study is that two factors – ”Development of organisation” and”Development of individuals” – are the most motivating factors for knowledgesharing to take place.
4 Experiment definition and goal
The objective of the proposed release planning method is to facilitate bettercommunication and knowledge sharing between the stakeholders of a small webapplication development project. The goal of the experiment is to validate ifthis indeed is the case. The experiment definition – using the definition templatebased on GQM and shown in [14] – is:
Object of study: The object studied in this experiment is a release planningmethod, presented in [1].
Purpose: To evaluate the release planning method with respect to:– understanding of the overall development project situation and of the
consequences of potential decisions– enabling and contributing to increased shared knowledge– prioritising the requirements and candidate configurations– reaching a consensus among the stakeholders– the stakeholders satisfaction after a decision is taken
Quality focus: The quality focus is the stakeholders satisfaction, the increasedshared knowledge, better understanding, prioritisation and an easier reachedconsensus.
Perspective: The development teams point of view.Context: The context is a student experiment with 63 students, forming groups
of 3 or 4 members. The study is conducted as Multi-test within object study.
5 Experiment planning and operation
To test the release planning method and to evaluate its effect on knowledgesharing, understanding, reaching a concensus, prioritisation of requirements andstakeholder satisfaction, we decided to create a scenario with a release planningproblem and to use groups of students – divided into treatment and controlgroups – to solve this problem by using either the proposed release planningmethod (treatment groups) or by using an ad-hoc practise (control group).
5.1 Planning
Context and subject selection Due to limited resources the experiment is runwith volunteer students. They will receive a modest economic compensation toparticipate in the experiment. After initial contact was made with a studentorganisation 63 students in their 3rd year of a industrial economy class signed
146 CHAPTER 13. PAPER 7
up for the experiment. They all filled in a pre-experiment questionnaire to assesstheir skills.
The experiment is off-line. A scenario with a release planning task is used andthe students take on roles that are described in the scenario. The task is to decideon the next release that can be implemented within a given time constraint andthat satisfies all stakeholders.
Hypothesis formulation Based on this expectation we formulated the formal hy-pothesis. The hypothesis will be tested for each factor defined in the purposesection of the experiment definition.
H0: The release planning method will not improve the overall development pro-cess.H1: The release planning method will improve the overall development process.
Variables and design The experiment is a one factor experiment with two treat-ments. The independent variable is the decision process used and can take ontwo values: using the proposed release planning method or using an ad-hoc baseddecision process.
The dependent variables are the degree or amount of (1) shared knowledge,(2) understanding the overall development process, (3) requirement prioritisa-tion, (4) reaching a consensus, and (5) stakeholder satisfaction. These variablesare measured with a post-experiment questionnaire.
Instrumentation The instrumentation of the experiment includes objects andguidelines that are handed out to the subjects. These are (1) a pre-experimentquestionnaire to assess the participants skills, (2) general instructions on theexperiment, such as a description of the scenario, a requirement list, and a de-scription of each role, (3) a release form, where the groups can write down therelease they decided on, (4) a post-experiment questionnaire that is handed outto the groups after the release form has been delivered to the experiment su-pervisors, and that contains questions used to measure the dependent variables,and (5) an instruction on how to use the proposed release planning method (onlyhanded out to the treatment groups).
Validity evaluation There is not enough space here to present a full discussion ofthe validity evaluation. This section is therefore narrowed down to a discussionconcerning the construct and internal validity.
Using no method as the second treatment may pose a threat to the validity. Itis not possible to control how the control groups are solving their task, whetherthey use a release planning method or not. But given that the subjects arestudents in their 3rd year, they will probably have no knowledge about releaseplanning methods, and will just try to solve the problem as best they can. Withmore experienced subjects – like professional developers and project managers– this could pose a threat to validity, but in the case of our student experimentwe believe that it is allowable to ignore this threat.
147
The members of the the treatment groups are receiving only a short introduc-tion to the release planning method. It is therefore the chance that they may notuse the method totally as instructed. This may pose a threat to the internal va-lidity, as not using the method correctly may have an impact on the results. Thetime that is available for training in a students experiment is limited. To copewith this problem we decided to observe how the students used the method, andin case we observe a misuse of the method we have to consider if this threatensthe validity. We were also available to answer questions from the subjects.
5.2 Operation
Sampling: 63 students had signed on to participate in the experiment. We di-vided them into 20 groups, 10 treatment groups and 10 control groups. Eachgroup had 3 students, with one subject taking on the role as project manager,another as marketing director and a third as programmer. Three groups had tohave 4 students, and the fourth subject had to take on the role of a programmer.
The participants filled in a pre-experiment questionnaire, where they an-swered questions about their skills, experience and preferred role in a groupsetting. Before the experiment started we decided on what role each partici-pant was to take on. We found this necessary since the subjects were studyingindustrial economy and only a small number of the students had skills and/orexperience in programming and marketing. This way we insured that all subjectshad some knowledge of the role they had to play in the experiment and wereable to look at the task from the corresponding viewpoint.
We proceeded by drawing for each group a project manager, a marketingdirector and a programmer. In the end, we had to assign three programmers tothree groups, and we did assign groups to them by drawing a group for them.After we had populated the 20 groups, we assigned them either to the treatmentgroup or the control group by chance.
Running the experiment: On the day of the experiment we used two large rooms,one for the treatment groups and one for the control groups. We gathered allstudents in one room to give them an introduction to the experiment (ca. 15minutes), and to hand over all handouts. After the control group had left forthe other room we gave the treatment groups a 10 minutes introduction to therelease planning method and also presented a small example. The experimenttook about two hours in total.
When the groups delivered their final release form and the questionnaires,the experiment supervisors controlled that all questions in the questionnaireshad been answered. If this was not the case, the subjects were asked to completethe questionnaire.
In the discussion of validity threats we found that using the method in a notintended way may disturb the results. However, observing the students duringthe experiment showed that they mostly were using the method as intended. Weobserved cases where groups had small deviations in the use of the method, but
148 CHAPTER 13. PAPER 7
we considered them not to be so serious that they posed a threat to the validityof the experiment.
6 The questionnaire and experiment analysis
The questionnaire consisted of 23 statements, divided into five groups, one groupfor each purpose of the experiment. For each statement the subjects had toexpress their attitude to the statement. To measure the subjects’ attitude weused a five point Lickert-scale [15].
The data collected using the Lickert-scale are data of type interval scale [16].This is not a straight forward decision, as there exists several opinions on themeasurement scale of data collected by using a Lickert-scale. Some will treat thedata collected from a Lickert-scale only as ordinal scale, and some will ask for aLickert-scale consisting of at least seven points before they will treat data fromit as interval scale. Our view is that we find it helpful to treat the data collectedby using a Lickert-scale as interval scale. This allows us to use the Student t-testin the analysis. Beyond the argument of measurement scale, we consider it to bepragmatic to use parametic tests on this data. There is a lot of evidence showingthat this is a good practice, even if the data is of a so-called non-applicablemeasurement scale [17].
The responses in the questionnaire were coded (1-dissagree strongly, . . . ,5-agree strongly). For every question we calculated the mean value and thevariance from all responses both for the treatment groups and for the controlgroups. Using the mean values, we applied the Student t-test. We have chosen asignificance level of 0.10. Given the discussion above, we also chose to performa non-parametric test, the Wilcoxon test. As can be seen in figure 2, the resultswere mostly the same.
We have chosen to reject the null hypothesis when the difference is statisti-cally significant for at least half of all questions for each group of questions.
7 Results
The results will be discussed separately for each group of questions. The resultsfor each item from the questionnaire are shown in figure 2.
Understanding: Items 7 – 10 in figure 2.The results are not what we expected. As can be seen from the results, the
control groups perform better then the treatment groups, and thus, the nullhypothesis can not be rejected.
The main reason for this result is, in our opinion, the lack of discussionswithin the treatment groups. They were more focused on using the method(with the provided artifacts) and on using the method correctly. This will bediscussed in more detail in section 8.
149
Shared knowledge: Item 1 – 6 in figure 2.The results for these items show that the control group performed better then
the treatment group. Three out of six questions have a significant difference. Thenull hypothesis can thus not be rejected. This is – as is the case with the previoustopic – not an expected result.
The main reason for this unexpected result is – as for the previous topic –the lack of communication between the group members in the treatment groups.Using the method for the first time, the focus was on using the method andon the requirement list. The control group did not have a method that tookaway there focus and discussed the importance of requirements and candidateconfigurations.
Fig. 2. Mean value, variance and p-value for each questionnaire item
We think that the communication between the group members of the treat-ment group will improve with more experience with using the new method, andby changing the communication pattern in the groups (see section 8).
150 CHAPTER 13. PAPER 7
Reaching a consensus Items 15 – 18 in figure 2.On all four items the treatment groups performed better, and on three of the
items the difference between treatment groups and control groups is significant.Hence, the null hypothesis can be rejected.
There are two possible explanations for this result. First, due the lack ofcommunication there are not so many conflicts, or the conflicts are not reallyunderstood. Second, once the requirements and candidate configurations hadbeen assessed, and the mean values had been calculated, finding a commondecision seems to be quite straightforward.
Requirement prioritisation: Items 11 – 14 in figure 2.The results are as expected, as the treatment groups perform better than the
control groups. However, only on one item is the difference between the groupssignificant. The null hypothesis can thus not be rejected.
Using the method made it easy for the treatment groups to prioritise therequirements and candidate configurations. They could simply sort the require-ments and configurations according to the assessment they had given to the re-quirements. However, due to the lack of communication and discussion it seemsthat the consequences of choosing one candidate configurations over the otherswere not discussed. The same is true for the written justification of each assess-ment. The release planning method has simplified the prioritisation process, buthas not improved the communication among the stakeholders.
Stakeholder satisfaction: Items 19 – 23 in figure 2.The treatment groups perform better on all five questions, and the difference
between the treatment groups and the control groups is below the 0.1 level onthree of the five questions. Therefore, the null hypothesis can be rejected.
The better stakeholder satisfaction in the treatment group most likely stemfrom the better performance on requirement prioritisation. When all stakeholderscan identify their top priorities it is ok to find a configuration that satisfies moststakeholders.
8 Discussion and further work
The results of the experiment have been surprising because of the treatmentgroups performance on understanding and knowledge sharing. Whereas we ex-pected the treatment group to perform better on these issues, in fact the controlgroup performed better. In our opinions there are two contributing factors thatcould have been controlled if we had been aware of them from the beginning. Thefirst contributing factor is the treatment groups focus on using a new method.The release planning method used in the experiment was unknown to the stu-dent volunteers, and they focused mainly on using the method correctly. Whereasthe method is meant to be a tool for supporting communication between stake-holders in this type of web application development, using the method correctlybecame the main activity and focus.
151
The other contributing factor has been the instrumentation of the experi-ment. For the convenience of the treatment group, the requirement list receivedby each student had columns for writing down the assessment and a small justifi-cation. Additional columns on the requirement list allowed the project managerto collect all groups members’ assessment. Each member of the treatment groupworked on his own copy of the requirement list, writing down his assessmentsand justifications. Bringing the assessments together on the project managerscopy became a mere mechanical activity. This is shown as situation A in figure3.
Fig. 3. Communication patterns when introducing a new method
The result can be improved by introducing two changes to the experimentlayout:
– Repeating the experiment with the treatment group using the release plan-ning method two or three times, using a different scenario each time. Thestudents would gain some experience using the method. This should result ina larger focus on the problem to solve and not so much on the method. Themethod would become a tool that supports the communication between sev-eral stakeholders and enables a better common understanding of the problemto solve.There is a chance that this may introduce a learning bias on the subjectsof the treatment groups. The scenarios will not be so different from time totime, and the students may respond in a way they think is expected fromthe experiment organisers.
– The instrumentation for the treatment group can be changed by only let-ting the group’s project manager write down the stakeholders assessmentand justification. This situation is depicted as situation B in figure 3. Theproject manager will work sequential on the requirements, and will ask the
152 CHAPTER 13. PAPER 7
stakeholders for their assessment and a short justification. This will increasethe communication within the group, at least between the project managerand the other group members.
We will conduct two new series of experiments, where the effect of the twosuggested changes will be implemented and studied. This will be done in thenear future, and will give us some indication on how important the learningeffect of using a new software development method will be with respect to thecommunication of its users, and what effect changing the communication patternwhen using a method will have on the results.
Another direction for future work is to study and understand how tacit knowl-edge is transformed into explicit knowledge in a rush-to-market type of develop-ment environment. A general model for Knowledge Sharing is for instance givenin [18] or [19]. What practices are enabling knowledge sharing and learning ina web application development project with its short deadlines and informality,and how can they be applied in a practical setting?
9 Conclusions
In this paper we have presented the details and results from a student exper-iments using a new release planning method. The objective of the experimentwas to study the effect of the release planning method on factors like knowledgesharing, understanding and stakeholder satisfaction. The experiment consistedof 20 groups of students that participated in a role play, where half of the groupsused the described release planning method and the other half had to solve theproblem at hand in an ad-hoc style. The results of the experiment were not as ex-pected for all of the factors. The members of the control group did communicatemore and achieved better results on two out of five factors. Possible explana-tions for the unexpected results have been identified and discussed, together withchanges that can be applied to the next experiment.
The lesson learned from this work is that it is necessary to have a even greaterfocus on how new software engineering methods should be used. Still, even if notall results are as expected, the experiment shows that the use of the proposedrelease planning method is suitable and that it achieves reasonable results onthree out of five factors.
10 Acknowledgements
We would like to thank our colleagues Tuulikki Gyllensvard, Tor Stalhane andJianyun Zhou for their help with the experiment.
References
1. Ziemer, S., Sampaio, P., Stalhane, T.: A decision modelling approach for analysingrequirements configuration trade-offs in time-constrained web application develop-ment. In: Proceedings of SEKE 2006. (2006)
153
2. Ziemer, S., Stalhane, T.: Web application development and quality - observationsfrom interviews with companies in norway. In: Proceedings of Webist 2006. (2006)
3. Karlsson, J., Ryan, K.: A cost-value approach for prioritizing requirements. IEEESoftware (September/October 1997 1997) 67–74
4. Jung, H.W.: Optimizing value and cost in requirement analysis. IEEE Software15(4) (1998) 74–78
5. Penny, D.A.: An estimation-based management framework for enhancive main-tenance in commercial software products. In: 18th International Conference onSoftware Maintenance (ICSM 2002), Maintaining Distributed Heterogeneous Sys-tems, 3-6 October 2002, Montreal, Quebec, Canada. (2002)
6. Greer, D., Ruhe, G.: Software release planning: an evolutionary and iterativeapproach. Information and Software Technolgy 46(4) (2004) 243–253
8. Karlsson, L., Regnell, B., Thelin, T.: Case studies in porcess improvement throughretrospective analysis of release planning decisions. International Journal of Soft-ware Engineering and Knowledge Engineering 16(6) (2006) 885–915
9. Du, G., McElroy, J., Ruhe, G.: Ad hoc versus systematic planning of software re-leases – a thre-staged experiment. In: Product-Focused Software Process Improve-ment, 7th International Conference, PROFES 2006, Amsterdam, The Netherlands,June 12-14, 2006, Proceedings. (2006)
10. Du, G., McElroy, J., Ruhe, G.: A family on empirical studies to compare informaland optimization-based planning of software releases. In: ISESE ’06: Proceedingsof the 2006 ACM/IEEE international symposium on International symposium onempirical software engineering. (2006) 212–221
11. Momoh, J., Ruhe, G.: Release planning process improvement – an industrial casestudy. Software Process: Improvement and Practice 11(3) (2006) 295–307
12. Beers, P.J., Boshuizen, H.P.A., Kirschner, P.A., Gijselaers, W.H.: Common ground,complex problems and decision making. Group Decision and Negotiation 15(6)(2006)
13. Ye, N., Zhi-Ping, F., Bo, F.: Motivation factors that make knowldge workersshare their tacit knowledge in universities: an empirical research. Services Systemsand Services Management, 2005. Proceedings of ICSSSM ’05. 2005 InternationalConference on (2005)
14. Wohlin, C., Runeson, P., Host, M., Ohlsson, M.C., Regnell, B., Wesslen, A.: Exper-imentation in Software Engineering: An Introduction. Kluwer Academic Publishers(2000)
15. Robson, C.: Real World Research. Blackwell Publishers (2002)16. Cooper, D.R., Schindler, P.S.: Business Research Methods. 8th edn. McGraw-Hill
(2003)17. Velleman, P.F., Wilkinson, L.: Nominal, ordinal, interval, and ratio typologies are
misleading. The American Statistican 47(1) (1993) 65–7218. Nonaka, I., Takeuchi, H.: The Knowledge Creating Company. Oxford University
Sven Ziemer:Managing change in software developmentIn Lech Madeyski, Miroslav Ochodek, Dawid Weiss and Jaroslav Zendulka(Eds.): Software Engineering in Progress. In conjunction with 2nd IFIP Centraland East European Conference on Software Engineering Techniques (CEE-SET2007), Work-in-progress session, 10-12 October 2007, Poznan, Poland. Pub-lished by Wydawnictwo Nakom, Poznan, Poland, ISBN 978-83-89529-44-2, pp.184–198.
155
156 CHAPTER 14. PAPER 8
Managing change in software development
Sven Ziemer
Department of Computer and Information Science,Norwegian University of Science and Technology,
Abstract. The software practices used to develop software systems emergeunder the influence of a number of factors, including such factors as cul-ture, size and criticality. These factors will change over time and thechange must therefore be managed. Software practices that have emergedunder the influence of factors that later has changed have to be adoptedto reflect the change of influencing factors and thereby the context of adevelopment project.Web applications are no exception and are operated in a changing envi-ronment. The continuous change takes place both with respect to theirfunctionality, the technology used to implement an application, the de-velopment practices applied to develop them, and the environment theyare deployed into. Managing the changing environment is a key factorto success. To be successful, change has to be coped with continuously.This paper proposes a subprocess to manage the changing environment ofsoftware applications and their development activities in a continuouslyway, using mostly qualitative information.
1 Introduction
Change is an integral part of life, and brings both insecurity, opportunities andrisks. We feel insecure in a time of change because we do not know what it willbring us and we may feel threatened by the risks that comes with the change.This can be used as motivation to spend time and effort to prevent change tohappen at all. But we can also work to exploit the opportunities that comes withchange and to build up barriers against the risks we are facing. In our globalisedworld this is more true than ever before.
Many companies and organisation are relying on the proper functioning oftheir software systems. This involves both offering the ”right” functionality andthe ”right” quality. As a consequence, there is an interest both in how softwareis developed, and that the software development team is aware of a changingenvironment, takes advantage of the opportunities and mitigate the risks thatcome along with change. Especially when there is a lot of competition betweencompanies (as between competing web applications), it becomes important toreact quickly and to be first out with new ideas and functionality.
Understanding software practice is an important aspect in reacting to achanging environment. The applied software practice in a development project
157
is influenced by a number of factors, with the changing environment of a soft-ware application being one of them. Adjusting software practice in a changingenvironment becomes therefor part of adjusting to it and taking advantage of if.
To exploit the opportunities of a changing environment and to avoid the risksof it, the change has to be monitored and managed. This should be an ongoingactivity as the environment is changing continuously. For many small develop-ment teams there is only limited information available to monitor the changingenvironment. In addition, the information is mostly qualitative. In this paper,a subprocess for managing small software development projects in a changingenvironment is proposed, that aims at supporting especially applications de-veloped with a strong emphasise on short time-to-market. This work emergedfrom studying web application development, and web application developmentwill therefore be used as an example. The paper is organised as follows: Thebackground for this work is presented in section 2, and related work is shown insection 3. Section 4 gives an overview of changes in the development practicesthat have been observed in web application development. A subprocess to man-age the changing environment in software application development is shown insection 5. Finally, some conclusions are given in section 6.
2 Background
The research in this paper has its origin in observing and studying web appli-cation development projects. The goal of this research was to understand andpossible improve the way in which trade-off’s involving quality attributes andtime-to-market are performed in web application development. The implicationsof this work is, however, not limited to web application development but ratherto software development projects that share a common set of characteristics andsoftware practices. Web technology or web architectures are not the key factoridentifying the development projects of interest for this research. The factors thatidentify the development projects of interest for this research are non-technicaland related to the applied software practices. As part of this research, there isalso a special focus on the emergence of software practices and on how softwarepractices have to change over time in order to respond to the environment of asoftware development project.
The software practices of interest are based on oral and ambiguous commu-nication between the involved stakeholders, relying on tacit knowledge of stake-holders and the short time-to-market requirement for software applications.
2.1 Software development practice
Software development is not an isolated activity but takes place in a broadercontext, with both technical and non-technical issues. The environment wheresoftware development takes place consist of the development project, of the or-ganisations that are involved in the development project, of competing softwareprojects and of potential customers expectation and demands. These are just
158 CHAPTER 14. PAPER 8
some elements that are part of the complex environment where software is de-veloped.
The way in which software is developed is influenced by its environment.Techniques and methods that are used in the development activities are selectedbased on an assessment of the environment of a software application. This isthe case when a decision between an agile approach and a traditional approachto software development has to be made. This is also the case when a decisionhas to be made on what activities are to be included into the development of asoftware application and how they are to be performed.
The success of agile software development methods such as Scrum or eXtremProgramming provides software developers with an alternative to plan-drivenmethods. Traditional software development methods – plan-driven methods –are characterized by a ”systematic engineering approach to software that care-fully adheres to specific processes” from requirements to finished code. Agilemethods are characterised by short iterative cycles, user involvement, relying ontacit knowledge within a team and self organising teams [1]. A pragmatic viewon both plan-driven and agile software development is not exclusive favouringone of the approaches, but rather locking where each approach is suitable. Thechoice of which approach to use should depend on the characteristics of a soft-ware development project at hand. In [1] B. Boehm and R. Turner describe fivecritical factors that describe the suitability of both approaches. These factorsare size, criticality, dynamism, personnel and culture. They are used to assessthe suitability of each development approach.
More general, software practice reflects how a development approach andhow a software process is working in a given real world situation and what ac-tions are taken by developers in a project to follow the development approachor software process. This opens up for other factors than the five success factorsmentioned above, to influence the actual practice. A framework to understandand describe the emergence of a working practice is presented by C. Slappen-del [2] and used by Kautz et. al. [3] and Madsen et. al. [4] to understand theemergence of software practice. This framework consist of three perspectives:the individualist perspective, the structuralist perspective and the interactiveprocess perspective. These perspectives open for factors such as previous experi-ence, social context and the environment of a software development project. Theemergence of software practice is thus influence by its context in such a way thatthe involved developers and stakeholders are engaging in activities, methods andtechniques that – based on their experience, beliefs and opinions – will enablethem to bring a project to a successful end and be satisfied personally.
The environment of a software development project and the factors that in-fluence the emergence of software practice are not stable but will most likelychange during the course of development projects. A change in the environmentof a development project has therefore implications for the software practice.The software practice that emerged in a given context to advance a developmentproject to a successful end may not achieve this goal any more in a changed con-text. Responding to the changed environment makes it necessary to investigate
159
if the current software practice still is suitable and the right way to achieve thegoals of a development project. Assuming that the environment of a develop-ment project is changing continuously the software practice has to be changedcontinuously, too. In other words, monitoring the environment of a software de-velopment project and evaluating the current software practice with respect tothe environment is an ongoing activity. This evaluation will result in a recom-mendation to change the software practice when it is not suitable to achieve theprojects goal under the changed environment.
2.2 Web application development
To manage the development of web applications in a changing environment, itis necessary to understand both how web applications are developed and whatinternal change takes place in a web application during its lifetime. After webapplications are launched initially, new versions are added – in many cases con-tinuously –, with each new version adding new functionality. Web applicationshave therefore – like other applications – to manage requirements that are chang-ing. At the same time, as a web application becomes more mature and fulfilsthe needs of it users, the development practices used to develop these require-ments are also changing. This change has to be managed, too. The developmentpractices used at any given time, reflects an applications environment and con-text. Discovering changes in this complex environment, that may influence thedevelopment practices, is not an easy task.
A web application passes through some lifetime phases after its initial devel-opment and launch. This can be divided into two phases: an initial applicationphase – launching the web application and getting attraction from potentialusers – and a mature application phase – when the application has a strong userbase and seeks to fulfil their expectations. The transition between these phasesis a gradual change over time, but still influences the development practices.
Lifetime phases for a web application (1) Initial phase: When web applicationsare launched, there is a strong emphasise on time-to-market. A new web applica-tion needs to attract new customers with new and innovative functions and witha positive experience for the users. The overall goal is to get users to return tothis web application and to use the services that it provides. In competition withother web applications it is important to be the first with a new functionality.This can sometimes be described as a rush-to-market approach, where presenceis more important than quality. Deploying new and innovative functionality andgetting the attention of potential users is perceived as an opportunity. Using along time to develop new functionality and let the attention of potential usersbe drawn to competing applications is perceived as a risk. (2) Mature phase:When a web application becomes more mature, the focus is on keeping the ex-isting users and try not to loose them to competing web applications. This isalso reflected in the risk assessment at this time: to change the application toomuch, providing functionality that do not contribute to the users satisfaction
160 CHAPTER 14. PAPER 8
and thereby loosing existing users. At this time, understanding the needs of theusers and providing functionality that fulfils their needs becomes an opportunity.
Changing development practices Developing software with such an emphasiseon time-to-market haves an impact on the development practices [5]. The de-velopment practises are informal and implementation oriented. Requirementsare elicited and specified informally. Applications are not tested thoroughly, andmany fixes are initiated by user reports. There is a lot of implicit knowledge, andthe success of the application that is being developed is depending on the expe-rience and general domain understanding of the developers. Early in the lifetimeof an application, a lot of knowledge will be tacit with the involved stakehold-ers. Through simple communication it can be collected (see part 1 of figure 1),and this knowledge will be mostly qualitative as it represents the stakeholdersopinion, belief and expert judgement.
When it comes to manage the changing environment that web applicationsare operated within, it is important to evaluate the usefulness and appropriate-ness of the used development practices. It follows from the aforementioned thatan informal requirement specification can be appropriate at an early stage of theapplication, whereas it is not appropriate in a later stage when we have a highrisk of loosing costumers if the end-users are not satisfied. Thus, the developmentpractices have to change too, together with the web application.
3 Related Work
Web applications are developed in several contexts and environments. The pur-poses of web applications vary, and the lifetime of web applications vary, too.This is reflected on the development practises applied to develop them and bythe tools and methods that are used developing them. The work on web applica-tion development processes is influenced by the variation in purpose and lifetimeof web applications. In the last years, a number of development process for webapplication development have been proposed, where each process is addressingsome characteristics that are important in a given context. A general model toassess and improve web processes is given in [6]. This paper divides web appli-cation development processes into two groups: the authoring process, concernedwith the navigational structure of an web application, and the infrastructureprocess providing the computational infrastructure of a web applications.
Developing a web application differs from traditional application develop-ment in a number of ways. One of the most prominent differences is the needto master the multidisciplinary nature of web applications. The navigationalstructure of a web application is an important part of the multidisciplinaryside of web applications and needs special consideration. There are a numberof web application processes that addresses the construction of the navigationalstructure. Web application processes such as UWE [7], NDT [8], The Araneusapproach [9] and the Autoweb Development Process [10]. All of these processesare model-driven and depend on an intensive requirement specification. In ap-plication domains where time-to-market is important, most development teams
161
do not feel they have the time to specify the requirements as precise as is neededfor these approaches to work.
The other group of web application processes are supporting the developmentof business logic and other computational functionality. A common approach hasbeen to use general development processes and to tailor them for the needs of webapplication development. Processes that are used in web application developmentare Agile Development [11], Extreme Programming [12], Scrum [13] and theOPEN Process [14]. There are also proposals of processes custom made for webapplication development such as WebHelix [15] and the Cognitive Web BasedSoftware Development Process [16]. These processes are include issues that arespecial for web application development and that need special support, such aswebsite management, content management, short development life-cycle times,multidisciplinary development teams and small development teams.
None of these web application development processes are addressing thechanging environment that web applications are developed in, and that influ-ence the development practices that are used. This is not to say that theseprocesses should not be used. They are addressing issues that are consideredspecial for web application development and provides techniques and method-ologies to manage these issues. However, when e-commerce web applications aredeveloped with a strong emphasise on time-to-market, being aware of and man-aging the changing environment is crucial for the success of a web application.The subprocess proposed in this paper claims to be a valuable contribution tomanage the changing environment of web applications and is intended to be usedwith other web development processes.
4 The changing environment of web applicationdevelopment
Web applications are changing all the time, and this is dealt with – more or lesssuccessful – in many ways. This ever present change can be observed in manyareas. Web applications are changing the contents they deliver to its users, andthe graphical design and presentation of this content. Implementing changes re-quires the use of suitable techniques and methods. The technology that is usedto implement the functionality of web applications is changing too. New technol-ogy has to be integrated with more proven technology, introducing compatibilityproblems and other problems arising from integrating several technologies. Theend-users of web applications are also changing, both in their number and theirexpectation, and so is the risk that end-users are moving on to a competing webapplication. Other changes are related to the development practices applied todevelop a web application and the trade-offs that are performed by choosingamong these practices. The assumptions that these trade-offs are based on, arechanging, and so is the knowledge that is available to make the decisions. Man-aging a web application over time calls for changing the development culture ofa web application, so that it becomes suitable for the further development of theapplication [17].
162 CHAPTER 14. PAPER 8
Explicit knowledgeTo
From
Tacitknowledge
Tacit knowledge Socialization Externalization
Internalization CombinationExplicit knowledge
Product Function
Cu
sto
me
r S
ati
sfa
cti
on
Absent Fully Implemented
Low
High
Implicit requirements
Requir
emen
ts
Wow-factor
Surprise
1. 2.
1
2
Fig. 1. Part 1: Conversion of knowledge [18]. Part 2: Kano model
4.1 Change of knowledge
When developing a software system, we will at any time have a certain degreeof freedom to make decision and a certain amount of knowledge to base thesedecisions on. At the start of an applications development, the knowledge aboutthe software system is limited, whereas the degree of freedom is high. With noknowledge available it is hard to take advantage of this freedom. In the end of thedevelopment, the degree of freedom is low, whereas there is a lot of knowledgeabout the system. The challenge is to gain as much knowledge as possible earlyin the development process, while the degree of freedom is still high. Even whenthe available knowledge is mostly tacit, it can still be exploited, and lessons canbe learned from using it [19]. Over time, when more has been learned aboutthe application under development, this knowledge can be refined. Exploitingtacit knowledge helps to take advantage of the degrees of freedom early in thedevelopment phase.
Managing the collecting of knowledge and the change from tacit and infor-mal knowledge towards more explicit and formal knowledge requires the use ofsuitable methods and tools. The tacit knowledge must be collected and analysed,which involves the conversion from tacit to explicit knowledge. A model for theconversion of knowledge is given by I. Nonaka [18] and is shown in part 1 offigure 1. This knowledge can then be organised, sorted, assessed and analysed(as in [20], [21], [5] , and [19]), and decisions can be taken based on this analysis.
4.2 Change of opportunities and risks
The opportunities and risks that a web application is facing depend on sev-eral factors. Among them are the lifetime phase of a web application [19], thecompetitive situation of a web application, and the end-users’ expectation.
163
Web applications are changing over time as they develop and mature. Thelifetime of a web application can be divided into three phases: new application,intermediate application and mature application. There are opportunities andrisks associated with each stage. A new application has to attract new end-usersand to make visiting users return. This can be achieved by developing innovativeand attractive functions. Short development cycles are important in the strugglewith competing web applications. It is an opportunity for a web application to bethe first one that offers functionality not offered by other web applications. Beinglate is a risk. The next phase is the intermediate phase with a base of returningend-users. The opportunities and risks are changing. Still, being innovative isimportant and an opportunity. But deploying new functionality too soon, atthe cost of its quality, becomes a risk. In the last phase – mature application –keeping the end-users is important and loosing them is the risk. Changes to theweb application are planned thoroughly. Satisfy the expectation of the end-usersis the most opportunity.
The end-users expectation to a web application and the user-experience itgives them, is another factor that influences the opportunities and risk of aweb application. New features and functions can have the effect of giving theend-users a positive user-experience and being a competitive advantage. Withtime, this function will not be considered as a surprising function, and it willbe something that the end-users expect. This change is illustrated by using theKano model (see part 2 of figure 1), where surprises become requirements (1)and where requirements become implicit requirements (2).
5 Managing change in software application development
Managing software applications in a changing environment is a necessary skillto develop a successful software application. Being able to manage this changemeans to achieve the goals and objectives that are defined for an applicationthat is being developed, despite the changing environment.
Managing change is an ongoing activity that will go on as long as the softwareapplication is developed and maintained. The challenge in managing changein web application development stems from the special characteristics that arefound in web application development (see section 2).
5.1 A subprocess for managing the changing environment
Managing the changing environment requires that a certain amount of planningis introduced into an otherwise ad hoc driven development effort. The aim isto introduce a small process that can be used together with other developmentmethodologies (figure 2). The objectives for this subprocess are:
– To be used with other iterative development process, and with methods thatare suitable in a situation and known to a project team.
164 CHAPTER 14. PAPER 8
– To exploit the available knowledge, both qualitative and quantitative, todetect important changes in the development environment and to decide onthe best action.
– To collect a base of knowledge for further development of a web application,adding new knowledge with every iteration.
– To enable learning from iteration to iteration, by evaluating how decisionswhere reached, and what results they yielded.
Collect data
Analyse data
Make decision
Implement and evaluate
decision
Lessons learned
Application objectives and vision
Knowledge base
Fig. 2. Subprocess for managing changes in the development environment
The subprocess has four steps that are described in detail below. For eachstep a goal is specified. How this goal can be reached depends on which life phasethe applications is in and on the knowledge that is currently available. It is thedeveloper’s task to decide on what methods and tools are suitable to reach thedescribed goal.
5.2 The subprocess step for step
1. Collect data – The input to this step is the application vision and itsobjectives, and – when this is not the first iteration – experience from previousiterations and that is stored in the knowledge base.
Goal: Create a shared understanding of the goals that have to be addressedby the current iteration and the decisions that have to be made, and collectknowledge that the next decisions can be based on.
Activities: The activities in this step are
– Create a common understanding of the goals and objectives for the nextiteration. What are the opportunities and risks in the current iteration?
– Identify and address issues that are important for the next iteration.What needs to be done to exploit the opportunities and to mitigate therisks?
165
– Describe the available options to resolve the issues and collect knowledgethat is needed to analyse the identified options.
– Collect knowledge about the applications environment such as the end-user expectation or other competing web applications, and that can havean effect on the identified options.
The knowledge collected in this step can either be qualitative (a stakeholdersopinion on the requirements) or quantitative (a measurement of the applicationperformance).
The activities in this step can be performed by applying several methods,such as affinity diagrams [22] or simple interviews with relevant stakeholders forqualitative data, and data from measurement initiatives that have been definedby using a measurement framework such as GQM [23], for quantitative data.
2. Analyse data – This steps start with a list of issues that has to be resolvedand possible options of implementing these issues.
Goal: Assess the consequences of the available options, and the relationshipsbetween them and the web application’s objectives and goals.
Activities: To reach the goal of this step, the following activities has to beperformed:– Analyse the available options that were described in the previous step
with respect to their consequences.– Analyse the factors from the applications environment that have an effect
on the available options, or that are influenced by them.– Identify the relationships between the available options and the factors
from the environment.
When assessing the available options and the relationships with other factors,both opportunities and risks have to be identified. Other aspects that may beimportant to find a good decision, can be included, such as return of value. Theassessment will depend on the available knowledge and the experience and beliefsof the involved stakeholders. If it should turn out during these activities thattoo little is known about important factors, the previous step can be repeatedto collect more knowledge.
The collected knowledge will be the starting point for the analysis in thisstep. If new knowledge is gained during this step, it will be added to collectedknowledge. As the knowledge can be both in qualitative and quantitative form,methods for both types of knowledge are needed. There are several methods avail-able to analyse qualitative data. The SWOT analysis [24] can identify strengths,weakness’s, opportunities and threats in a simple way. The available options canbe classified accordingly, and thereby the consequences can be shown. A similarmethod is Impact estimation tables [25], that assesses the available options ac-cording to some defined criteria. The advantage of this method is that any num-ber of criteria can be defined. It is also a method that allows the combinationof qualitative and quantitative data. Still another method to use is the Quality
166 CHAPTER 14. PAPER 8
Function Deployment method [26], that can be used to assess the requirementand option against some quality factors. These methods can be combined, as in[19], where a SWOT analysis is used together with a QFD-like matrix. Whenanalysing quantitative data, there are several ways to estimate and predict thevalues of interest.
3. Decision making – Based on the collected knowledge about the availableoptions and the assessment of these options, a decision can be made.
Goal: Find and decide on the option that best pursues the applications goaland best fulfils the stakeholders interests.
Activities: The activities in this step are– Prioritise the options that have been assessed in the previous step, ac-
cording to the applications objectives or the stakeholder’s interests.– Resolve conflicts that may exists between the stakeholders and reach a
common understanding for a decision.– Document the background and motivation for the decision. This will help
evaluating the decision and learning from it later in time.
Making a decision that fulfils the interests of most stakeholders, is oftennot possible without performing a trade-off between the stakeholders. Especiallywhen little knowledge about the application is explicit, and it is hard to predictthe precise outcome of a decision, finding a decision that the stakeholders agreeon will help to find a proper balance between the stakeholders’ interests.
The options that are considered for a decision, are prioritised according to thedegree in which they help pursue the application objectives and the stakeholders’interests. In some cases, there could be a criteria such as return on value or cost-benefit, whereas in other cases there is only an opinion from the stakeholdersto which degree an option pursue the applications objectives and their interests.When qualitative information is involved, these criteria can also be expressedqualitative, as shown in [27]. When both quantitative and qualitative knowledgehas to be combined, this can be done by either coding quantitative knowledgein a qualitative way or by coding qualitative knowledge in a quantitative way.
Reaching a decision means also that conflicts has to be resolved. There areseveral ways this can be done. One way is to reach for a consensus orienteddecision, where all stakeholders agree on the chosen decision. To reconcile allconflicts a Delphi like process can be used [28]. Another way is to vote for thedecision and to let the majority of stakeholders get their will. This may alsorequire reconciliation of conflicts between stakeholders, but the goal is to find adecision that the majority of stakeholders can agree on.
4. Implement and evaluate decision – After the decision has been made andafter it has been implemented, the outcome of the decision should be evaluated.
Goal: Evaluate and analyse the outcome of the implemented decision with re-spect to the application’s objective and the motivation for the decision. De-viations from what is expected and acceptable is analysed further, to findthe root causes, potential improvements and lessons to learn for future use.
167
Activities: The activities in this step are– Observe the outcome and the results of the implemented decision.– Analyse and evaluate the outcome, with respect to the intention and
motivation for the decision. Are the objectives met and is the outcomein line with the applications objectives as stated before the decision wasmade?
– In case the result is different from what is expected, an analysis shouldbe conducted to investigate the root causes of the deviation.
– Document any lessons learned and new knowledge gained in this step forfuture use.
Observing the outcome of a decision has to be done in a way that is suitablefor the development team. A simple description with the stakeholders observa-tions of the real outcome and a description of their expected outcome would do.This would be helpful for projects with no measurement instruments defined andwhere there is little explicit knowledge available. In other cases, when measure-ment instruments are already implemented, some aspects of the result could bemeasured and compared to the specified target.
To analyse the root causes a simple Post-Mortem Analysis (PMA) [29] canbe conducted. Based on the identified root causes, one can specify improvementsactions. These are either improvements for the already taken and implementeddecision – whenever this is possible and relevant – and improvements for futureuse and decisions. They are to be used in the next iteration or in later iterations.
The lessons learned from the root cause analysis are stored in a repository,where they are available for future use. In the same way, new knowledge thathas been gained through this iteration is stored in a knowledge base.
5.3 Knowledge base and Lessons learned
For every iteration, new knowledge gained and the lessons that have been learnedwhen making a decision should be kept for future use. Analysing the knowledgethat a decision was based on and the outcome of the decision helps the devel-opment team identify where improvement is needed. Based on this analysis thedevelopment team will be able to improve its ability to chose the right actionto achieve the desired outcome. Analysing the knowledge that a decision hasbeen based on and the outcome of the decision identifies where more preciseknowledge is needed. A measurement initiatives can be initiated based on thisanalysis.
The knowledge gained in an iteration is stored in a way that is natural forthe development team. There exists methods and tools to store knowledge andexperience gained during an iteration. On such method is the Experience Factory[30], but the development team can also chose a simple approach, such as a textfile, where knowledge and experience from a project so far are organised.
At the end of each iteration, when the knowledge base is being updated, theiteration itself is evaluated. Are the development practises applied to developthe application in this iteration still appropriate?
168 CHAPTER 14. PAPER 8
5.4 Managing change by using the process
The aims of the proposed subprocess are to be easy in use so that it can be usedand integrated with other iterative development methodologies, to exploit bothqualitative and quantitative information and combining them, and to supportmanaging a changing environment. There are two necessary activities in man-aging change: Detecting change and deciding on the right response to meet thechange.
Basically, change comes in two ways. Change comes either evolutionary orrevolutionary. The later is easy to observe, such as a introduction of new andinnovative functionality or the introduction of new governmental regulation. Theevolutionary change is, however, not always easy to detect. It can only be de-tected over time by comparing an assessment of a situation at two different times.The activities in the first two steps of the process – Collect data and Analysedata – are collecting and analysing data for every iteration. When assessing acertain situation, observations on the same situation from previous iterations isavailable and can be compared with the actual iteration. If there is a differencebetween the observations and if this difference represent a trend, a change isdiagnosed. A method that can be used to detect a change is the gap analysis.
The other activity is to decide on the right response to meet the change thathas been detected. By coupling the decisions that have to be made to the ap-plications vision and environment, change can be anticipated gradually for eachiteration. The available options are assessed against the changing environment,and they are prioritised according to the degree in which they are appropriatein the given situation.
6 Conclusions and further work
This paper has looked at the changing environment that web applications haveto face, and proposed a subprocess to manage the changing environment. Theaim of the subprocess is to create an awareness for the change, to detect howit influences a web application and to search for decisions that helps the devel-opment team to exploit the opportunities that comes with change and to setup barriers for the associated risks. The process helps the development team touse knowledge from different sources and to learn from decisions that have beenmade.
References
1. Boehm, B., Turner, R.: Balancing Agility and Discipline. Addison Wesley Pub-lishing Company, Inc. (2004)
2. Slappendel, C.: Perspectives on innovation in organizations. Organization Studies17(1) (1996) 107–129
3. Kautz, K., Nielsen, P.A.: Understanding the implementation of software processimprovement innovations in software organizations. Information Systems Journal14(1) (2004) 3–22
169
4. Madsen, S., Kautz, K., Vidgen, R.T.: Method emergence in practice - influencesand consequences. In: Proceedings of the 13th European Conference on Informa-tion Systems, Information Systems in a Rapidly Changing Economy, ECIS 2005,Regensburg, Germany, May 26-28, 2005. (2005)
5. Ziemer, S., Stalhane, T.: Web application development and quality - observationsfrom interviews with companies in norway. In: Proceedings of Webist 2006. (2006)
6. Rodriguez, D., Harrison, R., Satpathy, M.: A generic model and tool supportfor assessing and improving web processes. In: Preceedings of the Eighth IEEESymposium on Software Metrics (METRICS’02). (2002)
7. Koch, N., Kraus, A., Hennicker, R.: The authoring process of the uml-based webengineering approach. In: First International Workshop on Web-oriented SoftwareTechnology (IWWOST01). (2001)
8. Escalona, M., Mejias, M., Torres, J., Reina, A.: The ndt development process. InLovelle, J.M.C., Rodrigues, B.M.G., Aguilar, L.J., Gayo, J., Ruiz, M., eds.: ICWE2003: Proceedings of the 3th international conference on Web engineering. (2003)
9. Merialdo, P., Atzeni, P., Mecca, G.: Design and development of data-intensive websites: The araneus approach. ACM Transactions on Internet Technology (TOIT)3(1) (February 2003) 49 – 92
10. Fraternali, P., Paolini, P.: Model-driven development of web applications: theautoweb system. ACM Transactions on Information Systems (TOIS) 18(4) (2000)323 – 382
11. McDonald, A., Welland, R.: Agile web engineering (awe) process. Technical report,Department of Computing Science, University of Glasgow (2001)
12. Maurer, F., Martel, S.: Extreme programming: Rapid development for web-basedapplications. IEEE Internet Computing Magazine 6(1) (2002) 86–90
13. Rising, L., Janoff, N.S.: The scrum software development process for small teams.IEEE Software 17(4) (July/August 2000) 26 – 32
14. Haire, B., Henderson-Sellers, B., Lowe, D.: Supporitng web development in theopen process: Additinal tasks. In: Computer Software and Applications Conference,2001. COMPSAC 2001. 25th Annual International. (2001) 383–389
15. Whitson, G.: Webhelix: another web engineering process. Journal of ComputingSciences in Colleges 21(5) (May 2006) 21–27
16. Kushwaha, D.S., Singh, R.K., Misra, A.K.: Cognitive web based software develop-ment process: towards a more reliable approach. SIGSOFT Software EngineeringNotes 31(4) (2006) 1–6
17. Ramesh, B., Pries-Heje, J., Baskerville, R.: Internet software engineering: A dif-ferent class of processes. Annals of Software Engineering 14 (2002) 196–195
18. Nonaka, I., Takeuchi, H.: The Knowledge Creating Company. Oxford UniversityPress (1995)
19. Ziemer, S., Stalhane, T.: Trade-off’s in web application development – how toassess a trade-off opportunity. In: Proceedings Third International Conferenceon Web Information System and Technologies (WEBIST 2007), Barcelona, Spain,March 3-6, 2007. (2007)
20. Ziemer, S., Stalhane, T.: The use of trade-offs in the development of web appli-cations. In Matera, M., Comai, S., eds.: Engineering Advanced Web Applications.Rinton Press (2004)
21. Ziemer, S., Stalhane, T., Sveen, M.: Trade-off analysis in web development. In: 3-WoSQ: Proceedings of the third workshop on Software quality, ACM Press (2005)70–75
22. Straker, D.: A Toolbook for Quality Improvement and Problem Solving. PrenticeHall (1995)
170 CHAPTER 14. PAPER 8
23. Solingen, R.V., Berghout, E.: The Goal/Question/Metric Method: A PracticalGuide for Quality Imprvement and Software Development. McGraw-Hill Interna-tional (1999)
24. Hill, T., Westbrook, R.: Swot analysis: it’s time for a product recall. Long RangePlanning 30(1) (1997) 46–52
25. Gilb, T.: Competitive Engineering: A Handbook For Systems Engineering, Re-quirements Engineering, and Software Engineering Using Planguage. Butterworth-Heinemann Ltd (2005)
26. Akao, Y.: Quality Function Deployment. Productivity Press (1990)27. Ziemer, S., Sampaio, P., Stalhane, T.: A decision modelling approach for analysing
requirements configuration trade-offs in time-constrained web application develop-ment. In: Proceedings of SEKE 2006. (2006)
28. Linstone, H., Turoff, M.: The Delphi Method - Techniques and Applications. Ad-dison Wesley Publishing Company, Inc. (1975)
29. Birk, A., Dingsoyr, T., Stalhane, T.: Postmortem: never leave a project withoutit. IEEE Software 19(3) (2002) 43–45
30. Basili, V., Caldiera, G., Romback, H.D.: Experience Factory. In: Encyclopedia ofSoftware Engineering, Vol. 1. John Wiley Sons (1994) 476–496
171
172 CHAPTER 14. PAPER 8
Appendix A
Interview guide template
For the interviews with the companies that develop web applications (conductedin activities 1 and 4, see figure 1.3) we developed an interview guide. The use ofthe qualitative research interview has been described in section 3.3.1, includingdetails on how the interview guide was developed. This appendix presents thelatest version of the interview guide template.
173
174 APPENDIX A. INTERVIEW GUIDE TEMPLATE
Interview guide Version: 0.3 Question list Date: 06 April 2005
2. Questions The questions are sorted in different categories:
– General questions
– Projects
– Requirements
– Quality
– Trade-off analysis
2.1 General questions Nr Question 1 How many employees, and how many are developers? 2 How are the development activities organised in the company?
– Development department – Product department – Other
3 What are you developing? 4 In which phases of the software lifecycle do you participate?
– Initial Development – Maintenance – Development of updates – Other
5 What are success criteria for your company? How would you grade them? – Time-to-market – User satisfaction – Reliability – Security – Other
6 What are the most important competitors for your company and/or for your products? 7 In what way do you consider Web Development to be different from traditional development?
2.2 Projects Nr Question 1 What is the typical number of people on a development project?
– 1–3 people – 3–5 people – 5–10 people – More than 10 people
2 What is the time frame for a typical development project? What is the time frame for a short and a long development project? – 1–2 weeks – 2-4 weeks – 2 months – 3 months – 4–6 months – More than 6 months
175
Interview guide Version: 0.3 Question list Date: 06 April 2005
4 Are resources outside of the project available for the development team? If yes, how would you describe the resources, and their contribution. Possible resources located outside the development team are: – Maintenance – Database Manager – Performance engineer – Usability designer – Other
5 What deployment strategy is applied by the development team? – many small deployments (i.e. every week) – few larger deployments (i.e. three for the project).
6 Can you name the stakeholders involved in a typical project? Possible stakeholders can be: – Developer – Project manager – Product responsible, sales manager – Marketing manager – Other
7 When planning the next release and deciding the content of it, who is contributing to this decision? Can you list the persons contributing and rank?. – Project manager – Sales or marketing manager – Customer – Customer support – Other
8 How is the content of the next deployment decided? How are conflicts between different stakeholder solved?
9 How are projects initiated? Who is responsible for that decision? – Customer – Marketing or sales manager – Other
10 Who is responsible for a system once the project is done? – Maintenance – Customer – Development team – Other
11 Are you using any project methodology or software process? – Rational Unified Process (RUP) – eXtreme Programming (XP) – Agile development – Other
176 APPENDIX A. INTERVIEW GUIDE TEMPLATE
Interview guide Version: 0.3 Question list Date: 06 April 2005
Nr Question 12 What are the phases in your development activities? What are the practises used in your
organisation? – Requirement – Analysis – Design – Development – Testing – Deploy – Other
13 How are projects estimated? Are you using any estimation method?
2.3 Requirements Nr Question 1 Who is specifying new requirements? 2 How are new requirements specified? Describe the practise used in your organization.
Are you using any of these techniques? – Formal – Use case – Story – Anecdotes – Other
3 How are requirements prioritised? 4 How are requirements tested? 5 How are requirements validated?
2.4 Quality Nr Question 1 What quality attributes are important for your organization? Can you range them?
Quality attributes can be among – Usability – Security – Reliability – Robustness – Portability – Performance – Scalability – Availability – Other
2 How are they “specified”? Are these quality attributes specified explicitly or are they perceived in relation with a common perception or a relating product? Describe the practise used in your organization.
3 How are quality attributes measured? 4 How are quality attributes tested?
2.5 Trade-off analysis Nr Question 1 How are conflicts between requirements detected? 2 How are the effects of the conflicting requirements expressed? 3 How are the conflicts between conflicting requirements resolved?
177
Interview guide Version: 0.3 Question list Date: 06 April 2005
E. Strategies 1) Please allocate 100 points to the following three factors: Time-to-market, reliability (defects), and system
features, according to their importance to the success of the product. 2) Time-to-market awareness:
a) Do you have awareness of time-to-market even before the development process begins by rethinking the development process and norms?
b) Have you carefully planned and scheduled the project early in the process? c) Have you experienced the pressure to meet the deadline at the end of the process by working more
intensely or adding more personnel? d) Have you made trade-off decisions at the end of the development process to meet a deadline? (Dropping
features or lowering quality standards.) 3) Reliability awareness:
a) Do you have any mechanisms for achieving reliability (quality)? (Yes) Do the mechanisms cover all areas of the product and all phases of the project?
b) Is testing the most important method for achieving reliability of the product? c) Have you experienced resetting (or extension) of a deadline due to quality problems in any phase of the
development process?
178 APPENDIX A. INTERVIEW GUIDE TEMPLATE
Appendix B
Notes from an interview
This appendix show the notes taken in one of the interviews. The name of thecompany has been removed as has any other information that may identify thecompany due to a confidentially agreement with the company. The interviewshave been conducted in Norwegian and the notes are thus in Norwegian only.
179
180 APPENDIX B. NOTES FROM AN INTERVIEW
Notes from an interview with <Company> Versjon: 0.1 Referat fra 17. februar 2005 Date: 24. Februar 2005
1. Innledning Dette dokumentet inneholder et referat fra møtet mellom <Name> og Sven Ziemer den 17. Februar 2005. Møtet fra interessant for WebSys fordi man fikk se på hvordan en liten organisasjon bruker en Web Applikasjon i sin daglige drift. Det som var iøynefallende var at det ikke ble nevnt tekniske forhold blant de viktigste suksessfaktorer. Derimot er det organisatoriske og strategiske forhold som nevnes som suksessforhold.
I slutten av møtet tok vi opp muligheten for et evt samarbeid. Resultatet ble at man ikke starter et samarbeid, men at man kan ta kontakt med hverandre på et senere tidspunkt, dersom dette måtte være av interesse.
Dokumentet er organisert som følger: I avsnitt 2 finnes et referat fra møtet. Den første del er fra en generell samtale, mens seksjonene 2.1.1 – 2.1.5 er svar på spørsmål som ble stilt fra WebSys. I avsnitt 3 stilles det noen oppfølgende spørsmål.
2. Referat fra 17. februar 2005 Sted: Mosjøen
Deltakere: <Name> (<Company>) og Sven Ziemer (WebSys)
Noen fakta om <Company>:
– 6 ansatte (2 – 3 på utvikling; 3 – 4 på support).
– 50 % av Widerøes biletter selges via WIAS, webstedet til Widerøe.
– Webstedet har en conversion rate (Lock to book factor) på 12 %; iflg. <Company> er den for de aller fleste tilsvarende Websteder på ca 2 %.
– 65 % av bilettene som omsettes av WIAS er biletter for Widerøe; resten av bilettene er for andre flyselskaper.
– Omsettingen for 2004 var ca NOK 400 millioner, med et resultat på ca NOK 7,9 millioner. Inntektskilden til <Company> er først et gebyr på NOK 25, som alle kunder betaler. Videre har WIAS en cashoverflow som man får renteinntekter på (det kan gå opptil noen måneder fra Widerøe mottar betaling for en reise til de overfører pengene til flyselskapet som utførte flyvningen; dette skjer iht faste rutinger fra internasjonal bransjeorganisasjon IATA). I tillegg får WIAS en andel på 6 % av bilettsalget for andre flyselskaper.
– Fokus på å knytte kunder til seg. Kunder skal oppfattet webstedet som stabil og transperant. Dette skal oppnås bl.a. ved at man tilbyr reiser fra andre flyselskaper enn Widerøe og samarbeidene partnere.
Suksessfaktorer for virksomheten:
– Stabil arbeidsstokk
– Integret miljø
– Fleksibilitet
– Bevist kundeprofil og –håndtering.
– Alle medarbeidere må jobbe med brukerstøtte minst en dag i uken; dette gir en god oversikt over hvordan løsningen fungerer ute hos kundene.
– For å få til 24 h brukerstøtte er en medarbeider til envher tid i Thailand.
– <Company> går nye veier og utvider tilbudet, mens andre smalner tilbudet sitt.
Design av websiden:
181
Notes from an interview with <Company> Versjon: 0.1 Referat fra 17. februar 2005 Date: 24. Februar 2005
– Lager en internet basert løsning. Det betyr også at man prøver å få all brukerstøtte over på en innebygget chattefunksjon fremfor telefonbasert brukerstøtte.
Hva «ønsker» man seg, hva vil man forbedre seg i?
– Generelt: Man sliter med å forstå brukeratferd riktig. Man har ingen parametere for den loyale bruker.
– Teknisk: Bedre scripting mot databaser, og verktøy for å gjøre dette enklere.
2.1 Svar på intervju spørsmålene:
2.1.1 General questions: Nr Question 1 How many employees, and how many
are developers? 6 ansatte.
2 How are the development activities organised in the company? – Development department – Product department – Other
<Company> er et utviklingsmiljø som eies av <Company owner>.
3 What are you developing? <Company> utvikler webstedet for <Company owner>, og prosessene for salg av biletter via webstedet.
4 In which phases of the software lifecycle do you participate? – Initial Development – Maintenance – Development of updates – Other
<Company> har et totalansvaret for webstedet og er derfor involvert i alle faser.
5 What are success criteria for your company? How would you grade them? – Time-to-market – User satisfaction – Reliability – Security – Other
<Company> anser følgende som viktige suksesskriterier: – Brukertilfredshet – Brukervenlighet – Høy servicegrad – Automatisering – Tilgjengelighet på support (24 h for
<Company>) – Rask respons på henvendelser – Tillitsbygging hos kunder – Transperans – Korte beslutningsprosesser
6 What are the most important competitors for your company and/or for your products?
Online reisebyråer og andre flyselskaper.
7 In what way do you consider Web Development to be different from traditional development?
Ingen svar.
182 APPENDIX B. NOTES FROM AN INTERVIEW
Notes from an interview with <Company> Versjon: 0.1 Referat fra 17. februar 2005 Date: 24. Februar 2005
2.1.2 Projects Nr Question 1 What is the typical number of people on
a development project? – 1–3 people – 3–5 people – 5–10 people – More than 10 people
Det er to til tre utviklere som jobber med design og utvikling.
2 What is the time frame for a typical development project? What is the time frame for a short and a long development project? – 1–2 weeks – 2-4 weeks – 2 months – 3 months – 4–6 months – More than 6 months
Webstedet kan oppdateres daglig på innhold. Feilretting skjer ved behov; kodeoppdateringer skjer normalt en til to ganger i måneden (organiseres som linjearbeid). Komplett makeover av webstedet ca alle 18 måneder. En komplett makeover tar ca 2 måneder.
3 What roles are found in the development team? – Project Manager – Developer – Database manager – Testing – Other
Alle.
4 Are resources outside of the project available for the development team? If yes, how would you describe the resources, and their contribution? Possible resources located outside the development team are: – Maintenance – Database Manager – Performance engineer – Usability designer – Other
Ressurser man bruker utenfra er Usability designer (Netlife Research, Usability specialists).
5 What deployment strategy is applied by the development team? – many small deployments (i.e. every
week) – few larger deployments (i.e. three
for the project).
Se punkt 2; begge strategier anvendes. Mtp nyutvikling (makeover) brukes allikevel typen større oppdateringer.
6 Can you name the stakeholders involved in a typical project? Possible stakeholders can be: – Developer – Project manager – Product responsible, sales manager – Marketing manager – Other
<Company> er langt på vei uavhengig. Oppdragsgiver er marketingsavdelingen i <Company Owner>; tjenesteleverandør er <Tjenesteleverandør>
183
Notes from an interview with <Company> Versjon: 0.1 Referat fra 17. februar 2005 Date: 24. Februar 2005
deciding the content of it, who is contributing to this decision? Can you list the persons contributing and range them. – Project manager – Sales or marketing manager – Customer – Customer support – Other
Beslutninger tas av daglig leder, som får en liste med ønsker om endring både fra sin support avdeling og fra et samarbeid med bl.a. <Other Company> og <Tjeneste Leverandør>.
8 How is the content of the next deployment decided? How are conflicts between different stakeholders solved?
N/A
9 How are projects initiated? Who is responsible for that decision? – Customer – Marketing or sales manager – Other
<Company>.
10 Who is responsible for a system once the project is done? – Maintenance – Customer – Development team – Other
<Company>.
11 Are you using any project methodology or software process? – Rational Unified Process (RUP) – eXtreme Programming (XP) – Agile development – Other
<Company>bruker eget prosjetverktøy som er laget i Excel.
12 How are projects estimated? Are you using any estimation method?
Estimerer ut i fra felles erfaring. For ny makeover – som lanseres i år – bruker man både en soft-launch og en hovedlansering. Oppdertingsintervaller er også avhengig av <Tjeneste Leverandør>.
2.1.3 Requirements Nr Question 1 Who is specifying new requirements? Nye krav initieres delvis på bakgrunn av erfaringene
fra brukkerstøtte.
2 How are new requirements specified? Describe the practise used in your organization. Are you using any of these techniques? – Formal – Use case – Story – Anecdotes – Other
Nye krav utarbeides i fellesmøte. Man bruker skisser på Whiteboard for å illustere kravene. For hvert møte lages et referat. Forbedringspotential kommer fra egen erfaring og fra kunder.
184 APPENDIX B. NOTES FROM AN INTERVIEW
Notes from an interview with <Company> Versjon: 0.1 Referat fra 17. februar 2005 Date: 24. Februar 2005
Nr Question 3 How are requirements prioritised? Svar for spørsmål 3 og 4:
Man har stor informasjon fra kundekontakt.
4 How are requirements tested? Test skjer ved stikkprøver; det finnes ingen full coverage. Testingen skjer så langt som tiden rekker. For testingen av bl.a. prisstruktur har man ferdige tester. <Company> bruker brukertesten og har også et panel med super-brukere som man kan bruke. Feil håndteres når de dukker opp.
5 How are requirements validated? Se punkt 3.
2.1.4 Quality Nr Question 1 What quality attributes are important for
your organization? Can you range them? Quality attributes can be among – Usability – Security – Reliability – Robustness – Portability – Performance – Scalability – Availability – Other
<Company> stiller krav til bl.a. <Tjeneste Leverandør> og hosting leverandør. Krav til hosting leverandør er 99,7 % oppetid. Krav til <Tjeneste Leverandør> er at størrelsen for grafikkfiler er under 70 K. For usability brukr man egen teft. Det ansees som ikke målbar og man bruker ikke tid på det.
2 How are they “specified”? Are these quality attributes specified explicitly or are they perceived in relation with a comon perception or a relating product? Describe the practise used in your organization.
Se punkt 1.
3 How are quality attributes measured? Se punkt 1. 4 How are quality attributes tested? Se punkt 1.
2.1.5 Trade-off analysis Nr Question 1 How are conflicts between requirements
detected? N/A
2 How are the effects of the conflicting requirements expressed?
N/A
3 How are the conflicts between conflicting requirements resolved?
N/A
Trade-off med tid: «Vi sover ikke dårlig om natten fordi vi vet at en kosmetisk feil er borte om en eller to dager». Feil vurderes som enten driftskritisk og ikke-driftskritisk. Dette har også vært en overgang for organisasjonen som må lære å forholde seg til et nytt medium.
Ved feil kan man alltid rulle tilbake til forrige versjon.
185
Notes from an interview with <Company> Versjon: 0.1 Referat fra 17. februar 2005 Date: 24. Februar 2005
– Til 2.1.1 4: hvilke faser operer man med i <Company>?
– Til 2.1.1 7: Dette spørsmålet tok vi aldri opp; kan dere svare på det?
– Til 2.1.2 11: Kan dere vise oss et eksempeldokument fra prosjektverktøyet?
– Til 2.1.3 2: Hvem og hvor mange deltar på møtene? Er det mulig at dere viser oss et slikt referat?
186 APPENDIX B. NOTES FROM AN INTERVIEW
Appendix C
Post experimentquestionnaire
This appendix presents the post experiment questionnaire used to collect datafrom the student subjects after the release planning experiments. The samequestionnaire was used for all experiments. The order of questions has beenchanged after each experiment to avoid a history bias, i.e. that the studentsremember the questions. Also positive questions were rephrased into negativequestions. The questionnaire shown in this appendix was used after the secondround of experiments.
187
188 APPENDIX C. POST EXPERIMENT QUESTIONNAIRE
Questionnaire v 1.1 Page 1 of 1
Questionnaire Group: ________________ Name: ________________________________ 31 January 2007 The following is a set of statements concerning attitudes about the release planning scenario that you have performed today. For each statement please say whether you agree strongly, agree, are neutral, disagree or disagree strongly with it. Tick the appropriate box.
1. I received new information from the other group members.
6. I feel that the information that I provided to other group members changed their point of view/belief/opinion (of what should be included in the next release).
[1] Rusli Abdullah and Mohd Hasan Selamat. Facilitating Knowledge SharingWith Groupware among Faculty Communities in Higher Learning Insti-tution. International Journal of Computer Science and Network Security(IJCSNS), 7(5):220–229, May 2007.
[2] Robert Adcock and David Collier. Measurement validity: A shared stan-dard for qualitative and quantitative research. The American PoliticalScience Review, 95(3):529–546, September 2001.
[3] Yoji Akao. Quality Function Deployment. Productivity Press, Portland,OR, USA, 1990.
[5] A. J. Bagnall, V. J. Rayward-Smith, and I. M. Whittley. The next releaseproblem. Information and Software Technology, 43(14):883–890, Decem-ber 2001.
[6] Luciano Baresi, Sebastiano Colazzo, Luca Mainetti, and Sandro Morasca.W2000: A Modelling Notation for Complex Web Applications. InEmilia Mendes and Nile Mosley, editors, Web Engineering, pages 335–364. Springer-Verlag Berlin Heidelberg, 2006.
[7] Luciano Baresi and Sandro Morasca. Three empirical studies on estimat-ing the design effort of Web applications. ACM Transactions on SoftwareEngineering and Methodology (TOSEM), 16(4):15, September 2007.
[8] Victor Basili, Gianluigi Caldiera, and Hans Dieter Rombach. Encyclopediaof Software Engineering, Vol. 1, chapter Experience Factory, pages 476–496. John Wiley Sons, 1994.
[9] Richard Baskerville, Balasubramaniam Ramesh, Linda Levine, and JanPries-Heje. High-Speed Software Development Practices: What Works,What Doesn’t. IT Professional, 8(4):29–36, 2006.
[10] Len Bass, Paul Clements, and Rick Kazman. Software Architecture inPractice. Addison-Wesley, Reading, MA, USA, 2003.
191
192 BIBLIOGRAPHY
[11] Kent Beck. Extreme programming explained. Addison-Wesley, Reading,MA, USA, 1999.
[12] Patrik Berander and Anneliese Andrews. Requirements Prioritization. InAybuke Aurum and Claes Wohlin, editors, Engineering and ManagingSoftware Requirements, pages 69–94. Springer-Verlag Berlin Heidelberg,2005.
[13] Stefan Biffl, Aybuke Aurum, Barry Boehm, Hakan Erdogmus, and PaulGrunbacher, editors. Value-Based Software Engineering. Springer-VerlagBerlin Heidelberg, 2006.
[14] Andreas Birk, Torgeir Dingsøyr, and Tor Stalhane. Postmortem: neverleave a project without it. IEEE Software, 19(3):43–45, 2002.
[15] Barry Boehm and Li Guo Huang. Value-Based Software Engineering: ACase Study. Computer, 36(3):33–41, 2003.
[16] Barry Boehm and Richard Turner. Balancing Agility and Discipline. Ad-dison Wesley Publishing Company, Inc., 2004.
[17] Par Carlshamre, Kristian Sandahl, Mikael Lindvall, Bjorn Regnell, andJohan Natt och Dag. An Industrial Survey of Requirements Interdepen-dencies in Software Product Release Planning. In 5th IEEE InternationalSymposium on Requirements Engineering (RE 2001), 27-31 August 2001,Toronto, Canada, pages 84–93, 2001.
[18] Stefano Ceri, Piero Fraternali, and Aldo Bongio. Web Modeling Lan-guage (WebML): a modeling language for designing Web sites. ComputerNetworks, 33(1–6):137–157, June 2000.
[19] Robert N Charette. Software engineering risk analysis and management.McGraw-Hill, Inc., New York, NY, USA, 1989.
[20] Thomas Chau and Frank Maurer. Knowledge Sharing in Agile SoftwareTeams. In Wolfgang Lenski, editor, Logic versus Approximation, volume3075 of Lecture Notes in Computer Science, pages 173–183. Springer-Verlag Berlin Heidelberg, 2004.
[21] Thomas Chau, Frank Maurer, and Grigori Melnik. Knowledge Shar-ing: Agile Methods vs. Tayloristic Methods. In 12th IEEE InternationalWorkshops on Enabling Technologies (WETICE 2003), Infrastructure forCollaborative Enterprises, 9-11 June 2003, Linz, Austria, pages 302–307,2003.
[22] Paul Clements, Rick Kazman, and Mark Klein. Evaluating Software Ar-chitecture. Addison-Wesley, Reading, MA, USA, 2002.
BIBLIOGRAPHY 193
[23] Ulrike Cress and Friedrich-Wilehlm Hesse. Knowledge sharing in groups:Experimental findings of how to overcome a social dilemma. In Proceedingsof the 6th international conference on Learning sciences (ICLS ’04), 22-26June 2004, Santa Monica, California, USA, pages 150–157. InternationalSociety of the Learning Sciences, 2004.
[24] John W. Creswell. Qualitative inquiry & research design (second edition).Sage Publications, Inc, 2007.
[25] Monica Dalen. Intervju som forskningsmetode – en kvalitativ tilnærming.Universitetsforlaget, 2004.
[26] Anne Dardenne, Axel van Lamsweerde, and Stephen Fickas. Goal-directedrequirements acquisition. Science of Computer Programming, 20(1–2):3–50, April 1993.
[27] M. Denne and J. Cleland-Huang. The Incremental Funding Method:Data-Driven Software Development. IEEE Software, 43(14):39–47, 2004.
[28] Yogesh Deshpande, San Murugesan, and Steve Hansen. Web Engineering:Beyond CS, IS and SE Evolutionary and Non-engineering Perspectives. InSan Murugesan and Yogesh Deshpande, editors, Web Engineering: Man-aging Diversity and Complexity of Web Application Development, volume2016 of Lecture Notes in Computer Science, pages 14–23. Springer-VerlagBerlin Heidelberg, 2001.
[29] Torgeir Dingsøyr. Postmortem reviews: purpose and approaches in soft-ware engineering. Information and Software Technology, 47(5):293–303,March 2005.
[30] Regina Dittrich, Brian Francis, Reinhold Hatzinger, and Walter Katzen-beisser. A paired comparison approach for the analysis of sets of likert-scale responses. Statistical Modelling, 7(1):3–28, April 2007.
[31] Gengshen Du, Jim McElroy, and Gunther Ruhe. Ad Hoc Versus Sys-tematic Planning of Software Releases - A Three-Staged Experiment. InJurgen Munch and Matias Vierimaa, editors, Proceedings of 7th Inter-national Conference in Product-Focused Software Process Improvement(PROFES 2006), June 12-14, 2006, Amsterdam, The Netherlands, vol-ume 4034 of Lecture Notes in Computer Science, pages 435–440. Springer-Verlag Berlin Heidelberg, 2006.
[32] Gengshen Du, Jim McElroy, and Gunther Ruhe. A family on empiricalstudies to compare informal and optimization-based planning of softwarereleases. In Guilherme Horta Travassos, Jose Carlos Maldonado, and ClaesWohlin, editors, Proceedings of the 2006 ACM/IEEE International Sym-posium Empirical Software Engineering (ISESE 2006), September 21-22,2006, Rio de Janeiro, Brazil, pages 212–221. ACM Press, 2006.
194 BIBLIOGRAPHY
[33] C. Marlene Fiol. Consensus, Diversity, and Learning in Organizations.Organization Science, 5(3):403–420, August 1994.
[34] Jr Floyd J. Fowler. Survey Research Methods, Third Edition. Sage Pub-lications, Inc, 2002.
[35] Piero Fraternali and Paolo Paolini. Model-driven development of Webapplications: the AutoWeb system. ACM Transactions on InformationSystems (TOIS), 18(4):323 – 382, 2000.
[36] Adam D. Galinsky, William W. Maddux, Debra Gilin, and Judith B.White. Why It Pays to Get Inside the Head of Your Opponent. Psycho-logical Science, 19(4):378–384, April 2008.
[37] Franca Garzotto, Luca Mainetti, and Paolo Paolini. Hypermedia Design,Analysis, and Evaluation Issues. Communications of the ACM, 38(8):74–86, August 1995.
[38] Tom Gilb. Competitive Engineering: A Handbook For Systems Engineer-ing, Requirements Engineering, and Software Engineering Using Plan-guage. Butterworth-Heinemann Ltd, 2005.
[39] Amedeo Giorgi. The Question of Validity in Qualitative Research. Journalof Phenomenlogical Psychology, 33(1):1–18, 2002.
[40] Barney Glaser and Anselm Strauss. The Discovery of Grounded Theory:Strategies for Qualitative Research. Aldine, New York, 1967.
[41] Robert L. Glass. A Story about the Creativity Involved in Software Work.IEEE Software, 18(5):96–97, September-October 2001.
[42] Des Greer and Gunther Ruhe. Software release planning: an evolutionaryand iterative approach. Information and Software Technology, 46(4):243–253, 2004.
[43] Warren Harrison. N = 1: an alternative for software engineering research?In Janice Singer, Margaret-Anne Storey, and Susan Elliott Sim, editors,Beg, borrow, or steal: using multidisciplinary approaches in empirical soft-ware engineering research. Workshop session at the 22nd InternationalConference on Software engineering (ICSE ’00), June 4-11, 2000, Limer-ick, Ireland. ACM Press, 2000.
[44] Brian Henderson-Sellers, David Lowe, and Brendan Haire. OPEN ProcessSupport for Web Development. Annals of Software Engineering, 13(1–4):163–201, June 2002.
[45] Terry Hill and Roy Westbrook. SWOT analysis: it’s time for a productrecall. Long Range Planning, 30(1):46–52, 1997.
BIBLIOGRAPHY 195
[46] Martin Host, Bjorn Regnell, and Claes Wohlin. Using Students as Subjects– Comparative Study of Students and Professionals in Lead-Time ImpactAssessment. Empirical Software Engineering, 5(3):201–204, 2000.
[47] Martin Host, Claes Wohlin, and Thomas Thelin. Experimental ContextClassification: Incentives and Experience of Subjects. In Grula-CatalinRoman, William Griswold, and Bashar Nuseibeh, editors, Proceedings ofthe 27th international conference on Software engineering (ICSE ’05),May 15-21, 2005, St Louis, Missouri, USA, 2005.
[48] European Software Insitute. 1997 Software Best Practice Questionnaire.www2.umassd.edu/swpi/esi/tr-sbpqaor3.pdf, 1997.
[49] Susan Jamieson. Likert scales: how to (ab)use them. Journal of MedicalEducation, 38(12):1217–1218, 2004.
[50] Ho-Won Jung. Optimizing Value and Cost in Requirement Analysis. IEEESoftware, 15(4):74–78, 1998.
[51] Robert S. Kaplan and David P. Norton. The Balanced Scorecard. HarvardBusiness School Press, 1996.
[52] Gerti Kappel, Birgit Proll, Siegfried Reich, and Werner Retschitzegger.Web Engineering: The Discipline of Systematic Development of Web Ap-plications. Wiley & Sons, 2006.
[53] Joachim Karlsson and Kevin Ryan. A Cost-Value Approach for Prior-itizing Requirements. IEEE Software, 14(5):67–74, September/October1997.
[54] Joachim Karlsson, Claes Wohlin, and Bjorn Regnell. An evaluation ofmethods for prioritizing software requirements. Information and SoftwareTechnology, 39(14–15):939–947, 1998.
[55] Lena Karlsson, Thomas Thelin, Bjorn Regnell, Patrik Berander, andClaes Wohlin. Pair-wise comparisons versus planning game partitioning–experiments on requirements prioritisation techniques. Empirical SoftwareEngineering, 12(1):3–33, February 2007.
[56] Rick Kazman, Jai Asundi, and Mark Klein. Making Architecture DesignDecisions: An Economic Approach. Technical Report CMU/SEI-2002-TR-035 ESC-TR-2002-035, SEI, Carnegie Mellon University, September2002.
[57] Rick Kazman, Mario Barbacci, Mark Klein, S. Jeromy Carriere, andSteven G. Woods. Experience with Performing Architecture TradeoffAnalysis. In Barry Boehm, David Garlan, and Jeff Kramer, editors,Proceedings of the 21st international conference on Software engineering(ICSE ’99), May 16-22, 1999, Los Angeles, CA, USA, pages 54–63. IEEEComputer Society Press, 1999.
196 BIBLIOGRAPHY
[58] Rick Kazman and Mark Klein. Performing Architecture Tradeoff Analysis.In Jeff N. Magee and Dewayne E. Perry, editors, Proceedings of the thirdinternational workshop on Software architecture (ISAW ’98), November01 - 05, 1998, Orlando, Florida, USA, pages 85–88. ACM Press, 1998.
[59] Barbara A. Kitchenham, Shari Lawrence Pfleeger, Lesley M. Pickard, Pe-ter W. Jones, David C. Hoaglin, Khaled El Emam, and Jarret Rosenberg.Preliminary Guidelines for Empirical Research in Software Engineering.IEEE Transaction on Software Engineering, 28(8):721–734, August 2002.
[60] Steinar Kvale. The Qualitative Research Interview. Journal of Phe-nomenological Psychology, 14(1):171–196, 1983.
[61] Steinar Kvale. The social contruction of validity. Qualitative Inquiry,1(1):19–40, March 1995.
[63] Laura Lehtola, Marjo Kauppinen, and Sari Kujala. Requirements Pri-oritization Challenges in Practice. In Frank Bomarius and Hajimu Iida,editors, 5th International Conference on Product Focused Software Pro-cess Improvement (PROFES 2004), April 5-8, 2004, Kausai Science City,Japan, volume 3009 of Lecture Notes in Computer Science, pages 497–508.Springer-Verlag Berlin Heidelberg, 2004.
[64] Markus Lindgren, Rikard Land, Christer Norstrom, and Anders Wall. KeyAspects of Software Release Planning in Industry. In Farookh KhadeerHussain and Elizabeth Chang, editors, 19th Australian Conference onSoftware Engineering, 2008 (ASWEC 2008), March 25–28, 2008, Perth,Australia, pages 320–329, 2008.
[65] Harold A. Linstone and Murray Turoff. The Delphi Method. Techniquesand Applications. Addison Wesley Publishing Company, Inc., 1975.
[66] Xiaoqing Frank Liu. A quantitative approach for assessing the priori-ties of software quality requirements. Journal of Systems and Software,42(2):105–113, August 1998.
[67] Alan MacCormack, Chris F. Kemerer, Michael Cusumano, and Bill Cran-dall. Trade-offs between Productivity and Quality in Selecting SoftwareDevelopment Practices. IEEE Software, 20(5):78–85, September 2003.
[68] Sabine Madsen, Karlheinz Kautz, and Richard T. Vidgen. Method Emer-gence in Practice - Influences and Consequences. In Proceedings of the 13thEuropean Conference on Information Systems, Information Systems in aRapidly Changing Economy (ECIS 2005), May 26-28, 2005, Regensburg,Germany, 2005.
BIBLIOGRAPHY 197
[69] Andrew McDonald and Ray Welland. A Survey of Web Engineering inPractice. Technical Report TR-2001-79, Department of Computing Sci-ence , University of Glasgow, Scotland, March 2001.
[70] Andrew McDonald and Ray Welland. Agile Web Engineering (AWE) Pro-cess. Technical Report TR-2001-98, Department of Computing Science,University of Glasgow, Scotland, December 2001.
[71] Andrew McDonald and Ray Welland. Agile Web Engineering (AWE) Pro-cess: Multidisciplinary Stakeholders and Team Communication. In JuanManuel Cueva Lovelle, Bernardo Martın Gonzales Rodrıguez, Luis Joy-anes Aguilar, Jose Emilio Labra Gayo, and Marıa del Puerto Paule Ruız,editors, 3rd International Conference on Web Engineering (ICWE 2003),Oviedo, Spain, July 14-18, 2003, volume 2722 of Lecture Notes in Com-puter Science, pages 515–518. Springer-Verlag Berlin Heidelberg, 2003.
[72] Douglas C. Montgomery. Design and Analysis of Experiments (5th edi-tion). John Wiley & Sons, Inc., 1997.
[73] San Murugesan, Yogesh Deshpande, Steve Hansen, and Athula Ginige.Web Engineering: A New Discipline for Development of Web-Based Sys-tems. In San Murugesan and Yogesh Deshpande, editors, Web Engineer-ing: Managing Diversity and Complexity of Web Application Develop-ment, volume 2016 of Lecture Notes in Computer Science, pages 3–13.Springer-Verlag Berlin Heidelberg, 2001.
[74] Michael D. Myers and Michael Newman. The qualitative interview in ISresearch: Examining the craft. Information and Organization, 17(1):2–26,January 2007.
[75] Ikujiro Nonaka. A Dynamic Theory of Organizational Knowledge Cre-ation. Organization Science, 5(1):14–37, February 1994.
[76] Ikujiro Nonaka and Hiro Takeuchi. The Knowledge Creating Company.Oxford University Press, 1995.
[77] Robert L. Nord, Mario R. Barbacci, Paul Clements, Rick Kazman, MarkKlein, Liam O’Brien, and James E. Tomayko. Integrating the Architec-ture Tradeoff Analysis Method (ATAM) with the Cost Benefit AnalysisMethod (CBAM). Technical Report CMU/SEI-2003-TN-038, SoftwareEngineering Institute, Carnegie Mellon University, Pittsburgh, December2003.
[78] Luis Olsina, Guillermo Covella, and Gustavo Rossi. Web Quality. InEmilia Mendes and Nile Mosley, editors, Web Engineering, pages 109–142. Springer-Verlag Berlin Heidelberg, 2006.
[79] Luis Olsina and Gustavo Rossi. Measuring Web Application Quality withWebQEM. IEEE MultiMedia, 9(4):20–29, October 2002.
198 BIBLIOGRAPHY
[80] Wanda J. Orlikowski and Jack J. Baroudi. Studying Information Tech-nology in Organizations: Research Approaches and Assumptions. Infor-mation Systems Research, 2(1):1–28, March 1991.
[81] Balasubramaniam Ramesh, Jan Pries-Heje, and Richard Baskerville. In-ternet Software Engineering: A Different Class of Processes. Annals ofSoftware Engineering, 14(1–4):169–195, December 2002.
[82] Jorg Rech, Christian Bogner, and Volker Haas. Using Wikis to TackleReuse in Software Projects. IEEE Software, 24(6):99–104, Novem-ber/December 2007.
[83] William Remus. Using students as subjects in experiments on decisionsupport systems. In Proceedings of the Twenty-Second Annual Hawaii In-ternational Conference on System Sciences, 1989. Vol.III: Decision Sup-port and Knowledge Based Systems Track. January 3-6, 1989, Kailua-Kona, HI, USA, pages 176–180, 1989.
[84] Colin Robson. Real World Research. Blackwell Publishers, Oxford, 2002.
[85] Gustavo Rossi and Daniel Schwabe. Model-Based Web Application De-velopment. In Emilia Mendes and Nile Mosley, editors, Web Engineering,pages 303–334. Springer-Verlag Berlin Heidelberg, 2006.
[86] Gunther Ruhe, Armin Eberlein, and Dietmar Pfahl. Trade-off analysisfor requirements selection. International Journal of Software Engineeringand Knowledge Engineering, 13(4):345–366, August 2003.
[87] Gunther Ruhe and An Ngo. Hybrid Intelligence in Software Release Plan-ning. International Journal of Hybrid Intelligent Systems, 1(2):99–110,April 2004.
[88] Gunther Ruhe and Moshood Omolade Saliu. The Art and Science of Soft-ware Release Planning. IEEE Software, 22(6):47–53, November/December2005.
[89] Per Runeson. Using students as experiment subjects – an analysis ongraduate and freshmen student data. In Proceedings 7th InternationalConference on Empirical Assessment & Evaluation in Software Engineer-ing (EASE’03), April 8-10, 2003, Keele University, Staffordshire, UK,pages 95–102, 2003.
[90] Thomas L. Saaty. The Analytic Hierarchy Process. McGraw-Hill, NewYork, 1980.
[91] Thomas L. Saaty and Luis G. Vargas. Models, Methods, Concepts & Ap-plications of the Analytic Hierarchy Process. Kluwer Academic Publishers,Norwell, MA, 2001.
BIBLIOGRAPHY 199
[92] Omolade Saliu and Gunther Ruhe. Supporting Software Release PlanningDecisions for Evolving Systems. In 29th Annual IEEE / NASA SoftwareEngineering Workshop (SEW-29 2005), 6-7 April 2005, Greenbelt, Mary-land, USA, pages 14–26. IEEE Computer Society, 2005.
[93] Daniel Schwabe, Robson Mattos Guimaraes, and Gustavo Rossi. Cohe-sive design of personalized Web applications. IEEE Internet Computing,6(3):34–43, March-April 2002.
[94] Raymond Scupin. The KJ method: A Technique for Analyzing DataDerived from Japanese Ethnology. Human Organization, 56(2):233–237,Summer 1997.
[95] Asim El Sheikh and Haroon Tarawneh. A Survey of Web EngineeringPractice in Small Jordanian Web Development Firms. In Ivica Crnkovicand Antonia Bertolino, editors, Proceedings of the 6th joint meetingof the European Software Engineering Conference and the ACM SIG-SOFT International Symposium on Foundations of Software Engineering(ESEC/SIGSOFT FSE), September 3-7, 2007, Dubrovnik, Croatia, pages481–490. ACM Press, 2007.
[96] Dag I.K. Sjøberg, Bente Anda, Erik Arisholm, Tore Dyba, MagneJørgensen, Amela Karahasanovic, Espen F. Koren, and Marek Vokac.Conducting Realistic Experiments in Software Engineering. In Proceed-ings of the 2002 International Symposium on Empirical Software Engi-neering (ISESE’02), October 3-4, 2002, Nara, Japan, pages 17–26. IEEEComputer Society, 2002.
[97] Dag I.K. Sjøberg, Bente Anda, Erik Arisholm, Tore Dyba, MagneJørgensen, Amela Karahasanovic, and Marek Vokac. Challenges andRecommendations When Increasing the Realism of Controlled SoftwareEngineering Experiments. In Reidar Conradi and Alf Inge Wang, edi-tors, Empirical Methods and Studies in Software Engineering (ESERNET2001-2003), volume 2765 of Lecture Notes in Computer Science, pages24–38. Springer-Verlag Berlin Heidelberg, 2003.
[98] Carol Slappendel. Perspectives on Innovation in Organizations. Organi-zation Studies, 17(1):107–129, 1996.
[99] Tor Stalhane, Torgeir Dingsøyr, Geir Kjetil Hanssen, and Nils Brede Moe.Post Mortem – An Assessment of Two Approaches. In Reidar Conradi andAlf Inge Wang, editors, Empirical Methods and Studies in Software Engi-neering (ESERNET 2001-2003), volume 2765 of Lecture Notes in Com-puter Science, pages 129–141. Springer-Verlag Berlin Heidelberg, 2003.
[100] Tor Stalhane and Dag Sjøberg. WebSys:Web-based Systems - Time-To-Market vs. Reliability, Research proposal, 2002.
200 BIBLIOGRAPHY
[101] David Straker. A Toolbook for Quality Improvement and Problem Solving.Prentice Hall, 1995.
[102] Anselm C. Strauss and Juliet M. Corbin. Basics of Qualitative Research:Techniques and Procedures for Developing Grounded Theory (Second Edi-tion (6 Nov 1998)). Sage Publications, Inc, 1998.
[103] Roy Suddaby. From the editors: What grounded theory is not. TheAcademy of Management Journal, 49(4):633–642, August 2006.
[104] Liping Sui and Rong-Jou Yang. Study of Knowledge Sharing and Strat-egy in Enterprise Knowledge Management. In Proceedings of the Thirty-Seventh Southeastern Symposium on System Theory, 2005 (SSST ’05),March 20-22, 2005, Tuskegee, Alabama, USA, pages 336–340, 2005.
[105] William J. Tastle and Mark J. Wierman. Consensus and dissention: Ameasure of ordinal dispersion. International Journal of Approximate Rea-soning, 45(3):531–545, August 2007.
[106] Dean Tjosvold and Richard H. G. Field. Effects of Social Context on Con-sensus and Majority Vote Decision Making. The Academy of ManagementJournal, 26(3):500–506, September 1983.
[107] Bart van den Hooff and Femke de Leeuw van Weenen. Committed toShare: Commitment and CMC Use as Antecedents of Knowledge Sharing.Knowledge and Process Management, 11(1):13–24, January/March 2004.
[108] Paul F. Velleman and Leland Wilkinson. Nominal, Ordinal, Interval, andRatio Typologies are Misleading. The American Statistican, 47(1):65–72,February 1993.
[109] Giuseppe Visaggio. Value-based decision model for renewal processes insoftware maintenance. Annals of Software Engineering, 9(1–4):215–233,2000.
[110] Ronald E. Walpole, Raymond H. Myers, Sharon L. Myers, and KeyingYe. Probability & Statistics for Engineers & Scientists, seventh edition.Pearson Education International, Prentice-Hall, Inc., 2002.
[111] Alf Inge Wang and Tor Stalhane. Using Post Mortem Analysis to EvaluateSoftware Architecture Student Projects. In Dan Port, editor, Proceedings.18th Conference on Software Engineering Education & Training, 2005(CSEE&T 2005), April 18-20, 2005, Ottawa, Canada. IEEE-CS Press,2005.
[112] Claes Wohlin, Martin Host, and Kennet Henningsson. Empirical ResearchMethods in Software Engineering. In Reidar Conradi and Alf Inge Wang,editors, Empirical Methods and Studies in Software Engineering (ESER-NET 2001-2003), volume 2765 of Lecture Notes in Computer Science,pages 7–23. Springer-Verlag Berlin Heidelberg, 2003.
BIBLIOGRAPHY 201
[113] Claes Wohlin, Per Runeson, Martin Host, Magnus C. Ohlsson, Bjorn Reg-nell, and Anders Wesslen. Experimentation in Software Engineering: AnIntroduction. Kluwer Academic Publishers, 2000.
[114] Kazuma Yamamoto and Motoshi Saeki. Attributed goal-oriented anal-ysis method for selecting alternatives of software requirements. IEICE-Transactions on Info and Systems, E91-D(4):921–932, April 2008.
[115] Ning Ye, Fan Zhi-Ping, and Feng Bo. Motivation Factors That MakeKnowledge Workers Share Their Tacit Knowledge In Universities: an Em-pirical Research. In Jian Chen, editor, 2005 International Conferenceon Services Systems and Services Management (ICSSSM ’05), Volume2, June 13-15, 2005, Chongqing, China, pages 923–927. IEEE-CS Press,2005.
[116] Robert K. Yin. Case Study Research. Design and Methods. Second Edi-tion. Sage Publications, Inc, 1994.
[117] Norazlin Yusop, Didar Zowghi, and David Lowe. The impacts of non-functional requirements in web system projects. In Zahir Irani, Omiros D.Sarikas, Juan Llopis, Reyes Gonzalez, and Jose Gasco, editors, Proceed-ing of European and Mediterranean Conference on Information Systems(EMCIS2006), July 6-7, 2006, Alicante, Spain, 2006.
[118] Meilian Zheng and Gongmin Bao. An Empirical Study on KnowledgeSharing, Affective Commitment, Perceived Task Interdependence and JobInvolvement in Chinese Accounting Firms. In Dundar F. Kocaoglu, Tim-othy R. Anderson, and Tugrul U. Daim, editors, Technology Managementfor the Global Future, 2006. PICMET 2006. July 8-13, 2006, Istanbul,Turkey, volume 3, pages 1307–1315, July 2006.
[119] Liming Zhu, Aybuke Aurum, Ian Gorton, and Ross Jeffery. Tradeoff andSensitivity Analysis in Software Architecture Evaluation Using AnalyticHierarchy Process. Software Quality Journal, 13(4):357–375, December2005.
[120] Sven Ziemer. Managing change in software development. In LechMadeyski, Miroslav Ochodek, Dawid Weiss, and Jaroslav Zendulka, edi-tors, 2nd IFIP Central and East European Conference on Software Engi-neering Techniques (CEE-SET 2007), Software Engineering in Progress.Work-in-progress session, October 10-12, 2007, Poznan, Poland, pages184–198. Wydawnictwo Nakom, Poznan (Poland), 2007.
[121] Sven Ziemer and Ilaria Canova Calori. An experiment with a releaseplanning method for web application development. In Pekka Abrahams-son, Nathan Baddoo, Tiziana Margaria, and Richard Messnarz, editors,Proceedings Software Process Improvement, 14th European Conference,EuroSPI 2007, September 26-28, 2007, Potsdam, Germany, volume 4764
202 BIBLIOGRAPHY
of Lecture Notes in Computer Science, pages 106–117. Springer-VerlagBerlin Heidelberg, 2007.
[122] Sven Ziemer and Ilaria Canova Calori. Knowledge sharing through asimple release planning method for web application development. InDaniel Cooke et al., editor, Proceeding of the Nineteenth InternationalConference on Software Engineering & Knowledge Engineering (SEKE2007), July 9-11, 2007, Boston, USA, pages 686–691. Knowledge SystemInstitute Graduate School, Skokie, IL, USA, 2007.
[123] Sven Ziemer, Pedro R.F. Sampaio, and Tor Stalhane. A decision mod-elling approach for analysing requirements configuration trade-offs in time-constrained Web Application Development. In Kang Zhang, GeorgeSpanoudakis, and Giuseppe Visaggio, editors, Proceedings Eighteenth In-ternational Conference on Software Engineering and Knowledge Engineer-ing (SEKE 2006), July 5-7, 2006, San Francisco, CA, USA, pages 144–149. Knowledge System Institute Graduate School, Skokie, IL, USA, 2006.
[124] Sven Ziemer and Tor Stalhane. The use of Trade-offs in the Developmentof Web Applications. In M. Matera and S. Comai, editors, EngineeringAdvanced Web Applications. Rinton Press, 2004.
[125] Sven Ziemer and Tor Stalhane. Web Application Development and Qual-ity – Observations from interviews with companies in Norway. In JoseA. Moinhos Cordeiro, Vitor Pedrosa, Bruno Encarnacao, and JoaquimFilipe, editors, Proceedings Second International Conference on Web In-formation Systems and Technologies: Internet Technology / Web Interfaceand Applications (WEBIST 2006), Volume 1, April 11-13, 2006, Setubal,Portugal, pages 495–498. INSTICC Press, 2006.
[126] Sven Ziemer and Tor Stalhane. Trade-off’s in Web application devel-opment – How to assess a trade-off opportunity. In Jose Cordeiro andSlimane Hammoudi, editors, Proceedings Third International Conferenceon Web Information Systems and Technologies (WEBIST 2007), March3-6, 2007, Barcelona, Spain, pages 267–274. INSTICC Press, 2007.
[127] Sven Ziemer, Tor Stalhane, and Magnar Sveen. Trade-off Analysis in WebDevelopment. In Xiaoqing (Frank) Liu, Yan Sun, Gautam Kane, YujiKyoya, and Kunio Noguchi, editors, 3-WoSQ: Proceedings of the thirdworkshop on Software quality, held at the 27th International Conferenceon Software Engineering (ICSE’05), May 17, 2005, St Louis, Missouri,USA, ISBN 1-59593-122-8., pages 70–75. ACM, 2005.
