Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=tjor20 Journal of the Operational Research Society ISSN: 0160-5682 (Print) 1476-9360 (Online) Journal homepage: https://www.tandfonline.com/loi/tjor20 Agile operational research Melina Vidoni, Laura Cunico & Aldo Vecchietti To cite this article: Melina Vidoni, Laura Cunico & Aldo Vecchietti (2020): Agile operational research, Journal of the Operational Research Society, DOI: 10.1080/01605682.2020.1718557 To link to this article: https://doi.org/10.1080/01605682.2020.1718557 Published online: 13 Feb 2020. Submit your article to this journal Article views: 25 View related articles View Crossmark data
Transcript
Full Terms & Conditions of access and use can be found athttps://www.tandfonline.com/action/journalInformation?journalCode=tjor20
To cite this article: Melina Vidoni, Laura Cunico & Aldo Vecchietti (2020): Agile operationalresearch, Journal of the Operational Research Society, DOI: 10.1080/01605682.2020.1718557
To link to this article: https://doi.org/10.1080/01605682.2020.1718557
Melina Vidonia , Laura Cunicob and Aldo Vecchiettib
aRMIT University – School of Science, Computer Science and Software Engineering, Melbourne, Australia; bInstitute of Design andDevelopment (INGAR CONICET-UTN), National Scientific and Technical Research Council, Santa Fe, Argentina
ABSTRACTAs project management has become a critical subject in modern-world organisations,Operational Research (OR) needs to incorporate mechanisms to deal with rapid, unplannedchanges as well as confusing information and stakeholders with conflicting values. Agilemethods are widely used and tested in Software Engineering (SE) to deal with problems ofthe characteristics above. Because of this, after establishing that both OR interventions, aswell as SE developments, have common stages and information evolution, this proposalaims to pose the challenge of applying agility to manage OR projects. Guidelines to adaptAgile Methodologies to OR are proposed, and a case vignette is studied as an initial test.Finally, future lines of work are considered to define how the larger project in which thisproposal is embedded will continue.
ARTICLE HISTORYReceived 25 October 2018Accepted 12 January 2020
KEYWORDSSystems thinking; soft-OR;project management;practice of OR
1. Introduction
For several decades until now, Operational Research(OR) models are known as vital instruments inmany organisations to make decisions in complexproblems. As many authors pointed out, the devel-opment and implementation of such mathematicalmodels face several difficulties: stakeholders withconflicting interest, poor definitions, confusinginformation, changing environments and problemswith constant unclear ramifications (Ackoff, 1979;Churchman, 1967).
Researchers and practitioners proposed differentmethodologies to manage OR interventions, knownas Soft-OR and Problem Structuring Methods(PSM), characterised by the inclusion of externalstakeholders, as well as the use of techniques toimprove requirement elicitation (Mingers & White,2010). Some of them are Soft Systems Methodology(Checkland & Poulter, 2010), Strategic OptionsDevelopment and Analysis (SODA) (Eden &Ackermann, 2001), and Strategic Choice Approach(Friend, 2006). This research area targeted an unre-solved issue for OR interventions; then, due to thewide range of options of these approaches, differentauthors provided frameworks to compare them. Forexample, while Smith and Shaw (2019) tried toidentify similarities in different methods with aframework structured around some traditionalassumptions (ontological, epistemological, axiologi-cal, and methodological), Midgley et al. (2013) pro-vided an approach for long-term comparisons, whileremaining locally meaningful.
Nevertheless, these methodologies only focus onthe initial stage of the process, without providing aglobal approach. Even more, their use in practice isstill a topic under discussion: while it is widelyaccepted in some academic circles (Mingers, 2011),it is shunned in others as is not based on rigorousmathematical techniques (Ackermann et al., 2009).
Few articles focus on describing the use of Soft-OR or a PSM during an OR intervention. Da SilvaFilho (2015) recommended PSM to an organisationto carry out their interventions after determiningthat their comprehension of wicked problems wasskewed. Cabrera, Cabrera, Powers, Solin, andKushner (2018) used a Soft-OR approach to show-case the impact of a given design in CommunityOR while discussing its consequences and lessonslearned. Finally, Schramm and Schramm (2018)used a group-decision approach like SODA to sup-port different decision-making processes in Brazilianwatershed committees.
Nonetheless, it is more frequent to find articlesdescribing the mathematical background and detailsof an intervention to the detriment of the applica-tion itself and the whole system project (Ackermannet al., 2009; Ormerod, 2014). OR still centres inmathematical models and algorithms, instead of itsability to formulate management problems, solveand implement them (Ackoff, 1979; Ormerod,2014). As a result, it has continually focused on themathematical representations to the situationaddressed at the expense of systems thinking(Mingers & White, 2010). In many cases, this led to
CONTACT Melina Vidoni [email protected] RMIT University – School of Science, Computer Science and Software Engineering,Melbourne, Australia� Operational Research Society 2020
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETYhttps://doi.org/10.1080/01605682.2020.1718557
solving a situation that is widely different to reality(Ackermann, 2012).
In general, in an OR intervention, the teamneeded to implement a decision-making model con-sists of – at least – a mathematical modeller and asoftware engineer. They work in the whole processwith ideas, goals, and definitions of other stakehold-ers such as end-users, supervisors, and managers.The generated mathematical model must be linkedto the organisation’s information system to get dataand to provide results. From this perspective, an ORintervention is similar to the development andimplementation of a software-intensive system. Itslifecycle starts with ideas, followed by design, execu-tion, tuning, and maintenance. In this regard, theyneed to be managed as systems.
Software Engineering (SE) has several accepted,tested proposals for dealing with information sys-tems. SE moved from focusing on how to developsoftware (Kneuper, 2017) to establishing processmanagement methodologies under the name of life-cycles (Birrell & Ould, 1985; Boehm, 1986; Royce,1970). However, the emergence of the Internet,the requirements for shorter time-to-market and theincrease in changing requirements demonstrated theneed for more lightweight methods. These were latergrouped under the concept of agile methods (AM),founded in the Agile Manifesto (Beck et al., 2001).AMs are widely accepted in the SE community(Dingsøyr, Nerur, Balijepally, & Moe, 2012; Melo,Cruzes, Kon, & Conradi, 2011).
This article analyses the use of SE agility for ORinterventions. For this purpose, the characteristics ofthe main stages of an OR intervention are identifiedtogether with the information managed at eachstage. This evidences a match between OR interven-tions and agile lifecycles stages, showing that agilitycan be applied to project management in OR. Thisis further elaborated in three-step guidelines, whichinclude selecting a methodology, organising a pro-ject, and representing the information in artefacts.
Finally, a case study vignette supports the viabilityof this approach.
This article is organised as follows. Section 2 dis-cusses why agility is considered an option, whileSection 3 proposes a solution to manage the infor-mation and divide the project into stages, creatingthe empirical context for using agility in OR.Section 4 provides practical guidelines to do so, dis-cussed using a case vignette in Section 5. Finally,Sections 6 and 7 present discussions andconclusions.
2. Agile: Why?
AM is a type of project management process. Theyanticipate changes and allow a high degree of flexi-bility – in both project and source code – to rapidlyadapt them. Because of the process and tools thatAMs provide, stakeholders can make small objectivechanges without considerable amendments to thebudget or schedule (Dingsøyr et al., 2012). Thisconcept is also applied to manufacturing and supplychain management (Gligor, Esmark, &Holcomb, 2015).
In SE, the rise of AMs was powered through thegeneration of the Agile Manifesto (Beck et al.,2001): a document establishing the goals and phil-osophy for agility. It established four values andtwelve principles that any method must have to beagile; these can be seen in Table 1. Even more, theAgile Manifesto’s influence contributed to AMs’increasing acceptance (Kneuper, 2017; Tarhan &Yilmaz, 2014).
Many of these values and principles have a directcorrelation to OR interventions. For example:
� (V4) Responding to change over following aplan. The emergence of the Internet made devel-opments to face a shorter time-to-market and aneed to better adapt to unclear and changingrequirements (Boehm, 2006). Thus, agility statesthat a plan must: (a) be flexible enough to allow
Table 1. Agility values and principles.Code Definition
Values V1 Individuals and interactions over processes and tools.V2 Working software over comprehensive documentation.V3 Customer collaboration over contract negotiation.V4 Responding to change over following a plan.
Principles P1 Satisfy the customer through the early and continuous delivery of valuable software.P2 Changing requirements, even in late phases, to enhance customer’s competitive advantage.P3 Frequently delivery of working software, preferring shorter timescales.P4 Business people and developers must work together throughout the project.P5 Build projects around motivated individuals.P6 Conveying information on development teams through face-to-face conversation.P7 To use working software as the primary measure of progress.P8 Sustainable development: everyone should be able to maintain a constant pace indefinitely.P9 To have continuous attention to technical excellence and sound design.P10 Simplicity is essential.P11 Self-organising teams produce the best designs and architectures.P12 The team must regularly reflect on how to become more effective.
2 M. VIDONI ET AL.
adapting the system to the required changes, and(b) provide artefacts and processes to do so.Usually, an OR model is not able to adapt to thestakeholders’ needs, because its development iscentred in technical, mathematical properties anddoes not consider future changes; then it loses itsusability as it drifts away from the real, targetedsituation to solve a more ideal, outdated case(Checkland & Poulter, 2010).
� (P2) Changing requirements, even in latephases to enhance the customer’s competitiveadvantage. Related to value V4, requirementscan also change in late stages of developments –i.e., when the model is reaching completion or isin-use in the organisation. Having a lifecycle thatprovides tools and management process toreview and modify the source code when achange is detected, can reduce the rate of newerrors. This may encourage the reuse of themathematical code.
� (P10) Simplicity is essential. Complex, difficult-to-read source code or documentation is moreprone to create misunderstandings; even more, itcan increase the cost of modifying an existentmodel to the point where it is less expensive tostart over than salvaging what already exists(Ahmed, Ahmad, Ehsan, Mirza, & Sarwar, 2010;Dingsøyr et al., 2012). Agility aims to simplifythe code, even if it means rewriting it until itmaximises it performances and readability. If thesource code is easy to read, then it is possible toreuse it in new problems (Haefliger, von Krogh,& Spaeth, 2008), reducing developing times andcosts, while at the same time refining solutions(Frakes & Kang, 2005). This property is crucialfor OR models in changing environments.
� (P12) Regularly reflect on how to become moreeffective. Agility proposes that teams should beable to recognise what they did wrong in a pro-ject, to learn from their mistakes and apply thatnew knowledge in future interventions (Dingsøyret al., 2012). This allows refining practices andprocess, to improve steadily.
Current PSM and Soft-OR methodologies implycapturing and visually representing the stakeholders’points of view (Mingers & White, 2010). Thus,PSMs focus on the most “social” aspects of agility,such as V1, V3, P2, P4, and P6 of Table 1. Theremaining properties are left behind even when they
are a prominent part of an OR project’s process(Ackermann, 2012).
Creating a project and source code that it is eas-ily modified without negatively affecting its qualityis not an easy task. This becomes even more chal-lenging as the size and complexity of projectsincrease. AMs apply a technique known as divide-and-conquer, which implies fragmenting a set ofrequirements, selecting a few, and incrementallybuilding the system by iteratively adding morerequirements to it (Beck et al., 2001). This enablesdevelopers to frequently deliver working code,reducing the return-of-investment time, and testingthe system-model “on-site” and interoperating withthe others (Boehm, 1986). This is an intrinsicallyagile characteristic reflected in the properties P1, P3,P7, P8, and P10 (see Table 1).
3. Agile and or: Aspects in common
Managing OR interventions implies distinguishingthe elements affecting the process, the emerging sys-tems and people’s rationales (Mingers & White,2010); as a result, it requires to also focus on othersactivities beyond writing mathematical code. Thissection establishes the empirical context that will beused in Section 4, identifying the states that repre-sent the evolution of information within an OR pro-ject and that defines its lifecycle.
3.1. Information evolution
In an OR intervention, information evolves andgrows during a project, becoming more refined(Ormerod, 2008). Figure 1 summarises the evolutionof the information states of an OR project lifecycle.This proposal showcases the similarities between ORand SE projects, aiming to facilitate the adaptationof AMs.
Although the process for OR is sequentially pic-tured, it is considered a progressive elaboration: con-tinuously improving and adding details, as morespecific, accurate information becomes availablewhile the project progresses (Project ManagementInstitute, 2017). Whereas it is possible to go back toseveral states earlier in order to add more detail, itis not tolerable to “skip” more refined states whenmoving forward. For example, it is feasible to goback from “Mathematical Model” to “FormalSpecification,” but after completing the changes, the
Figure 1. The process of information evolution during an OR project.
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 3
progression must improve the “MathematicalDesign” before addressing the “MathematicalModel” once again.
Regarding each state, in particular, any projectstarts with “Ideas,” which defines that a given needshould be satisfied, while “Requirements” formalisethe conditions and capabilities limiting the solutionto that need. Current PSM and Soft-OR methodolo-gies aim to elicit these two states of information andagreements about them.
“Formal Specification” aims to structure“Requirements” to discover specific information,and prioritise functionalities of the model. This aimsto shape the project process in incremental, iterativecycles, by following the three steps of Figure 2:
“Mathematical Design” is the primary step beforecoding the model. It consists of diagrams and docu-mentation that communicate its structure to differ-ent stakeholders by using the Software Architectureconcept of points-of-view: elements that documentthe same model from unique perspectives but withcomplementary specifications (ISO/IEC/IEEE, 2011).
Finally, the “Mathematical Model” is the sourcefiles with the model coded in the selected mathem-atical programming language, while the “Answers”are reports of results such as charts, files, spread-sheets, etc, derived from raw results, and also relatedanalysis, and other internal process.
3.2. Lifecycle stages
The process of managing and evolving this informa-tion defines different periods of a project’s lifecycle.Each of these phases has a defined goal regardingthe information to be used, and the refinement it
produces. The output generates the artefacts (seeSection 4.3) used as input on the next stage, as theinformation evolves with the project.
Phases are related to short-term goals of the pro-ject and not to the situation it aims to solve. SEestablished that it is more productive to move insmaller steps: understanding the problem, designingand evaluating solutions, and then coding. This isdone instead of attempting to perform all the stepsat the same time – i.e., coding while requesting add-itional, improved data.
Figure 3 presents the proposed phases and corre-sponding information states, adding brief definitionson the phases. The use of stage names established inSE and nomenclature that is known to both practi-tioners and academics, simplify the adaptation ofexisting methodologies.
“Analysis” focuses on defining who is part of theproject (clients and developers/modellers), what theproject is about, and what the clients genuinelyneed. It is essential to understand the value of theproject as a whole, integrate the participants’ know-ledge and favour the synergy of collective work.PSM and Soft-OR methodologies can be appliedduring this phase (Checkland & Poulter, 2010; Eden& Ackermann, 2001; Friend, 2006).
“Design” aims to structure the requisites for for-mal modelling that does not require mathematicalcode. This is used as documentation, composed ofthe “Formal Specification” and the “MathematicalDesign,” and is a base upon which accountable peo-ple can be defined, requisites prioritised, and howthey will be translated to the model.
Any misunderstandings, lack of detail or missingagreements dragged on from the “Analysis” phasewill negatively affect the “Design,” and cause defect-ive refinements of information. In the “Design”phase, it is documented how decisions were made,how the code is organized, which parts of the codecontain each functionality, and how changes shouldbe handled and addressed. If possible modificationsare considered at the “Design,” later changes in the
Figure 2. Steps to refine “Requirements” into a “FormalSpecification.”
Figure 3. Proposed lifecycle stages for OR projects.
4 M. VIDONI ET AL.
“Mathematical Model” become less “traumatic”; thecost of reworking the code and the chance of intro-ducing new faults are lowered.
The following step is “Development,” and its goalis to code the model in a given mathematical lan-guage, following the specification created during the“Design” phase: it is the most traditional and coreactivity of any OR project. It also includes the gen-eration of answers, testing regarding the inputs, andits validation, compared to the specifications of“Requirements,” and to the goals of the project.
The final stage is “Implementation.” Figure 3depicts it together with the “Development” as it usesthe same states of information. However, its goal isto deploy the model in the client’s organisation, toensure its adequate interaction with existing sys-tems, and to train users in its use. Often this stagedoes not receive the importance it should, especiallyin projects that generate models used to assist in asingle decision. Putting a model into use is essentialto its success.
4. Agile: How?
Making an OR intervention agile implies three steps,visible in Figure 4. The following subsections pro-pose guidelines for each of them.
4.1. Selecting a method
Choosing the right method to arrange interventionsallows the management sequence to be defined.Since AMs have been used in SE for almost twodecades, many authors have compiled experiences todefine specific recommendations using different
parameters; the authors Qumer and Henderson-Sellers (2008) provided an extensive report about it,according to different parameters such as teamsizes, project length, geographical distribution,among others.
Though there are several papers along these linesin SE, they are not directly applicable to OR, assome parameters – such as project or team size –are not the same in both disciplines. Furthermore,offering such detailed selection guide requires hav-ing several real-world application reports specific toOR, as was the case in SE (Qumer & Henderson-Sellers, 2008). Therefore, until such data have beencollected, this section only states an initial selectionguideline, to kickstart its application in differentinterventions.
Overall, AMs can be grouped into two categories:
a. Code-focused methods are especially suited forsmaller teams and projects. Though these AMsprovide a specific reduced project managementframe, most of their practices, activities, andprocesses are concentrated in coding. Examplesare XP and Crystal.
b. Project-focused methods offer practices, artefacts,and activities linked with the global vision ofthe project. They are usually targeted to mid/large project or team sizes, or situations inwhich an overall organization is mandatory. Asa result, they can often be merged with otherAMs to obtain a more thorough methodology.Examples are Scrum, Kanban, and Lean.
This categorisation narrows the search, by evalu-ating internal and external characteristics. The firstset has perks related to the team itself, its training,background, and organisation, while the secondgroup involves restrictions coming from thecontracting organisation. Figure 5 summarisesthis process.Figure 4. Steps to apply agility to OR interventions.
Figure 5. Overview of the AM selection process.
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 5
Regarding internal characteristics, smaller teamswith lower-to-none experience in applying agility toOR should use Type A methods; thus the change intheir practices is less significant, while at the sametime includes elements of project management.Otherwise, larger and distributed teams shouldmove towards Type B, as these AMs imply a moreradical change, being harder to implement if themodellers are not trained or present a higher resist-ance to adjusting their practices. Concerning exter-nal characteristics, longer projects are moredependent of managerial activities; other elementsto consider the provision of documentation as partof the agreement and the selection of an AM alreadyused at the client’s ICT department, among others.
As can be seen in Figure 5, internal and externalcharacteristics are not excluding, and need to beconsidered at the same time. Which one weighsmore towards a final decision ultimately depends onthe specific characteristics of the situation. Toremain in scope, this article works with four of the
most currently accepted methodologies; Figure 6summarises why they are selected.
The parallelism showed in Section 3 between theinformation managed in OR and SE projects, andtheir lifecycle stages allow the definition of whichprocesses of AM are parts of each stage. Table 2shows this correspondence. It is noteworthy that,even though they have different organisations, theyalways fit their activities into the lifecycle stages pre-sented previously.
4.2. Project organization
Organising a project through an AM requires defin-ing intermediate goals, to establish iterations: transi-tional steps that incrementally build the leadingtowards the final product. Each one yields animproved, more refined version of the previousrelease, and the last one should be the final version.
Therefore, it is vital to determine how many iter-ations there would be, when will they start and end,
Figure 6. Selected Agile Methodologies, and reasons for selection.
Table 2. Comparison of the proposed OR project stages, and AMs processes.Scrum FDD XP Crystal
Delivery: Handle theaccepted code to users.Reflect on thelessons learned.
6 M. VIDONI ET AL.
and which functionalities they will develop. To dothis, it is required to perform an initial “Analysis,”outside any iteration. Though AMs allocate proc-esses for doing this – i.e., Scrum’s Initiate, FDD’sDevelop an Overall Model, XP’s Listening orCrystal’s Project – PSMs can be used for the samegoal. Mixing them with AMs is entirely possible asagility is not rigid: it only suggests techniques. Thus,the general steps can be seen in Figure 7.
Formalising the requirements implies document-ing them. How this is done depends on the specificAMs that have been selected; i.e., using FDD leadstowards a feature list, while if using XP the teambuilds user stories. Section 4.3 will discuss the infor-mation representation.
Prioritising requirements involve deciding whichones should be developed first and why; the reasonsare usually dependencies between them, the effortneeded to code, and its impact on the system’s finalfunctionality. Requirements should also have an esti-mated development time; this is calculated based onexperience and, if the team tracks it through metrics– as done in SE (Achimugu, Selamat, Ibrahim, &Mahrin, 2014) – this value can be impartiallyobtained. This is done to divide a project intosmaller iterations, where each one has an overallgoal – i.e., making a specific part of the whole pro-cess – and a list of the included requirements.Those with a higher priority must be included atearlier iterations. This is done considering a max-imum time limit so that all iterations have a similarlength and work-load.
4.3 Information representation
Artefacts materialise the information evolution pre-sented in Section 3.1. They are tangible by-productsthat describe a given aspect of the system, whichcan be represented using different notations (IEEEComputer Society, 2014). Each AM often offers dif-ferent artefacts tuned to its specific proposals andactivities. All of them can be framed in the statesseen in Figure 1.
Table 3 condenses these relationships, but it isnot an exhaustive list, only disclosing the most rele-vant or commonly used artefacts; descriptions arenot included to keep it concise. However, no arte-facts are included for “Ideas” as they are often pro-vided as an artefact by the client; thus, even thoughAMs acknowledge it, they do not provide specificrepresentations for it.
Many of these artefacts are generic elementsdirected towards information specific to the projectmanagement; examples are “User Stories,” “Product/Sprint Backlog,” “Features, Feature Sets,” “UseCases,” “Iteration Plan.” Adapting them to an ORproject becomes straightforward. Something similarhappens to the code: AMs do not prescribe how thecode is written or structured, though some of themsuggest best practices – i.e., XP’s coding in pairs-; asa result, they can also be translated to OR.
Other artefacts may prove to be more challeng-ing, such as writing test cases or generating a“Mathematical Design” as architecture, in the termsknown to SE.
Figure 7. General steps to organise an OR intervention in iterations, following an agile philosophy.
Table 3. Match between the information states and AMs artefacts for their representation.
Requirements User Stories User Stories Features Requirements FileFeature Sets Use Cases
Formal Specification Product Backlog Release Plan Feature List Project Map & Release PlanSprint Backlog Iteration Plan Selected Feature List Iteration Plan
Mathematical Design Acceptance TestUnit Tests Test Cases
Extensions Overall Model ArchitectureMathematical Model Product Increment Working Source Code Release Components & BuildAnswers End User Material Help Training & Manual
Extensions Statistics
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 7
OR modellers are not used to creating this type ofinformation refinement before coding (Ackermann,2012). However, an advantage of AMs is that theyallow building them iteratively and incrementally: itstarts as a general overview (i.e., FDD’s OverallModel), and it is cyclically refined and expanded ateach iteration through activities like Implement,Design by Feature. More importantly, it is not neces-sary to create new artefacts since there is currentlyan extensive gallery to choose from. This reduces theresistance to incorporate the new practice.
5. Case vignette: Retail stores chain
This section presents a case study through a reverseengineering approach of an academic collaborationperformed with a local retail stores chain – hereinnamed RSC. This was originally carried out using atraditional unstructured approach, without followingany project-driven lifecycle as a guide; as a result,several problems appeared throughout the venture.This process is analysed, issues are recognised, anddifferent possible agile solutions are presented. Thecontrast between the approach originally used andthe Agile Operational Research highlights how agilitycan prevent or solve the occurrence of this type ofproblems. It is worth noting that this does not implythat problems need to be known before starting aproject because, in most cases, this is not feasible.
RSC has retail shops in almost half the countryand supplies them through two different ware-houses. All the stores used SAP – a proprietary ERP– as their central enterprise system. The companyasked for an evaluation of changes in the structureof its supply chain in a ten-year period, concerningspecific warehouses, and to minimise its operatingcosts. The development of a mathematical modelwas proposed to estimate costs and benefits of thethree available options: removing the warehouse,keeping it as-is, or transforming it to cross-docking.
The intervention started in December 2016 andcompleted by August 2017. Even when the result
was positive for RSC, it presented some projectmanagement difficulties. In the following para-graphs, some of the showcased texts are extractedfrom official communications and reports, meetingrecordings, emails and modellers’ notes, amongothers. Figure 8 shows the overall process of theRSC intervention, using the codes referencedbetween brackets that appear in the figure – i.e.,[C1] – to relate that stage to the textual description.
At the first meeting, the representatives com-mented on their intentions to change the warehouse,and the need to evaluate the feasibility and conveni-ence of carrying it forward or not. After that meet-ing, modellers rushed to the model developmentusing it to represent the current situation [C1].Because the data required were sent in emails with asignificant delay, and without the correspondingdocumentation [C2], an additional effort wasneeded to understand the information received; thisresulted in assumptions made to move for-ward early.
However, while the coding advanced, modellersrequired more data [C3]. The issue was that, regard-less of using an SAP, RSC had inconsistent informa-tion and many sectors were isolated from theprimary system, using local databases and generat-ing information gaps. The following excerpts of dif-ferent emails of the managers over a three-monthperiod exemplify that situation:
� “[… ] I apologise, but we were not able toobtain this data. I could give you only thefollowing items [… ]. We will need to startcollecting the rest of it, as most of themcannot be obtained through the enterprisesystem, and we need to inquire theDistribution centre about them [… ]”.
� “[… ] The inconsistency in the cubic metersseen in December is caused by a wrong inputdata from some products. We are trying tolocate which one is, so we can study anddecide how to fix it [… ]”.
Figure 8. Original process for the RSC intervention.
8 M. VIDONI ET AL.
Then, RSC’s representatives included mid-linemanagers in the project, without any formal intro-duction, and tasked them with preparing therequired information [C5]. This caused delays in thegeneration of data and misunderstandings in com-munication. The following are two differ-ent requests:
� A modeller requested: “We need the number oftruck travels (both for your trucks and those out-sourced), on each period (week or month) foreach sale point [… ]”. The logistic representativeanswered: “[… ] This question is too ambiguous.Could you be more specific?”.
� In another case, a modeller wrote to a mid-linemanager: “[… ] To improve the model, we needthe supply routes you are actually using. Is itpossible to obtain that information? [… ]”.
As RSC attempted to fix their databases, manag-ers became interested in new requirements [C5].These were communicated through email to thedevelopers. The code was continuously adapted toinclude each request [C4] and, as there were no for-mally specified requirements, changes were notrecorded, and the model was not versioned. Hence,what should have been final releases became proto-types with intermediate results [C6], affecting theprovided results, and requesting new constraintsand data to be coded [C7].
Near the end of the intervention, at a meetingscheduled to show preliminary results, the modellersdiscovered that they had dealt with middle manag-ers, without any decision-making power. Newrequirements appeared when presenting the finalreport to the company’s heads, some of which com-promised the very purpose of the project. This defi-ciency in the elicitation and selection of
stakeholders delayed the project’s completion farbeyond schedule.
After this summary of the project life flow, it isvisible that there was no consistent stream of activ-ities, and that any form of project management wasdisregarded in the intervention, by both modellersand clients.
5.1. Methodology selection and processorganization
Many AMs can face the same problem differently,but still, provide a global vision and a satisfactoryoutcome. To show there is no single, perfect choicefor each project, two proposals are made regardinghow the RSC situation could have been managed.The main problem, in this case, is the lack of dataand the database’s inconsistencies [C2, C3, C5] thatdelayed the project and forced developers to fix themodel as new, verified data were provided continu-ously [C4].
The first proposal uses Scrum as AM, and itsprocess can be seen in Figure 9.
The generation of the product backlog throughan elicitation method in “Analysis” phase, makesvisible the lack of data [C1–C5]. Thus, the first iter-ation is dedicated to determining which data arerequired for the model, while RSC reorganises theirdatabases. Then, the outcome is the complete set ofverified information to be used as an input in themodel. The second and third iteration, develop themathematical models through incremental sprints[C6], first for the current conditions, and then forthe other decision options. New ideas [C7] can beincluded on each sprint after detection on theweekly meetings/reports. They are translated to userstories-fragments of functionalities to be
Figure 9. Proposed management of RSC’s intervention using Scrum.
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 9
implemented (see examples in Table 4) and addedto the backlogs to be included on versioned models.
The second proposal uses Extreme Programming(XP), and the overall process can be seen in Figure10. XP provides a code-centric and more straightfor-ward approach, but still adding project managementto the intervention. As it focuses on creating thetests for the functionalities before developing them(Beck, 1999), instead of moving directly to coding,the modellers first prepare the tests using the pro-vided data. With this, RSC’s data deficiencies wouldhave been evident sooner, giving RSC representa-tives time to organise a process of databaseunification.
Regarding the project’s organisation, a couple ofinitial meetings allow delineating user stories pre-venting gaps in the requirements specifications andestablishing a baseline upon which to work [C1, C2,C7]; examples are shown in Section 5.2. This isrevisited through weekly meetings, acknowledgingthe current progress, and re-discovering newrequirements.
XP process could group its iterations to releaseversions for modelling each situation. The interven-tion starts building the model that reflects keepingthe warehouse as-is so that RSC representatives canconduct the database unification and reorganisation.During that process, the information can be handledin stacks to the modellers. Each stack represents asmall-iteration of the lifecycle and allows workingon specific features: coding only those functionalitiesthat have verified data and tests. This produces aversioned release (of the specific situation),
accompanied by documentation registering eachfunctionality and change committed to the code anda formal results report.
As a result, the “Development” would have pro-gressed in parallel with RSC’s enterprise systemreorganisation, producing an iteration for each stackof data, until finishing the group for the as-is situ-ation. Then, further iterations would only require aminimal amount of additional data and would beable to skip the first two steps from Figure 10.
A critical difference between de AMs used is thatXP provides an overall structure but performssmaller iterations in which execute the five activities(data collection and verification, testing scenariosand models, and coding), while Scrum has a pre-fixed amount of more significant iterations (eventhe first one is dedicated to defining the requireddata). Thus, while this XP organisation poses aneven workload for both client and modellers, theScrum proposal makes the first iteration moredependent on the client’s database reorganisation.
Table 4. Examples of “User Stories” for RSC’s case vignette.Code Story Relevance Estimate Dependencies
E01 As a manager, I want to compare the operational costs of maintaining thewarehouse as is, changing it to cross-docking, or closing the location.
5 E02, E03, E04
E02 As a manager, I want to know the operational costs (salaries, products transportcosts, earning from sales, warehousing fees, and taxes) for maintaining thewarehouse as-is, for the next five years.
4 US12, US30,US34, US40
E03 As a manager, I want to know the operational costs (redundancy payments,products distribution costs, earnings for sales, trucks outsourcing costs, andtaxes) for closing the warehouse.
4 US13, US24, US40
E04 As a manager, I want to know the operational costs (salaries, products transport,and distribution costs, earnings from sales, and taxes) for changing thewarehouse to a cross-docking location and keeping it for the next five years.
4 US14, US23,US30, US40
US9 As a modeller, I want to have data (sales average, sales points locations, anddistances between them, latest salaries, additional outsourcing costs, latesttaxes) from the past five years in CSV format.
4 5 hours US30
US12 US13US14
As a modeller, I want to create input data by inferring the costs related totravelling from one sales point to another, in the scenario of [keeping thewarehouse, changing to cross-docking, closing the warehouse].
2 1 day
US23 As a modeller, I want to have constraints to define which sales points are suppliedby the cross-docking location.
2 2 days
US24 As a modeller, I want to have constraints to determine the number of products tobe distributed from the central warehouse to each sales point location.
4 3 days US14, US30
US30 As a database manager, I want to retrieve the sales volume, organised by monthand by sales point, and export it to a CSV.
2 3 days
US34 As a modeller, I want to minimise the operational costs to infer earnings forkeeping the warehouse during the next five years.
4 3 hours
US40 As a manager I want the model to export the results of each scenario to an Excelfile, with charts and simple tables.
3 2 days
Figure 10. Proposed management of CRS’s interventionusing XP.
10 M. VIDONI ET AL.
5.2. Artefacts examples (XP)
Two different possibilities were shown for RSC: XPand Scrum. To keep this article in focus, only frag-ments of the artefacts created for XP are presented,to act as illustrations.
The first artefacts are “User Stories,” used to for-malise the “Ideas” into “Requirements.” They shouldbe prioritised, to later define the iterations. Table 4proposes examples, along with three prioritisationcriterions: estimated development time, if it dependson other stories and relevance for the client (withfive being the most relevant). The codes of Table 4are deliberately designated to skip numbers: this isto show that there should be many more stories.
It is worth noting that user stories are alwayswritten from a specific stakeholder role’s point ofview. They are often detailed while developers andstakeholders brainstorm together and structured tobe easy to read to the latter. They must be simpleenough to portray the requirements straightfor-wardly but reducing possible misunderstandings.Some interesting points can be discussed inthese examples:
� There can be epics: stories that are too genericand need to be specified into smaller stories.[E01] and its subdivisions in [E02], [E03], and[E04] are examples these. The subdivisions canalso be considered epics, but are used for group-ing the iterations to build each specific model, asmentioned in Section 5.1.
� There should be a story for each input data thatneeds to be inferred from the existing data. Thestories [US12], [US13], and [US14] show thatdata inference must also be documented for eachscenario that needs to be developed.
� Likewise, there should also be stories for eachgroup of constraints on each scenario (such as
[US23] and [US24]) and stories for each object-ive to be used (i.e., [US34]).
� The expected type of reports to be generated alsoneed to be addressed as stories, as in many casesthis needs to be coded as part of the model;[US40] is an example of this.
� Obtaining and organising the input data wasvital. It is suitable to represent each data stackrequirement as a story itself. This is done fromthe database manager’s perspective (as in[US30]) and the modellers’ standpoint(i.e., [US9]).
With this, it is possible to document the itera-tions through a “Release Plan,” to indicate wheneach iteration should be completed, and the“Iteration Plans” for each one of them. This can bedone using many of the available online tools exist-ing for software development projects.
In particular, Figure 11 shows a “Release Plan”created with FeatureMap (Salience, 2018), which canbe used with most AMs. Furthermore, most of thesetools allow collaboratively updating of the diagrams,reflecting changes to the iterations. As the agile val-ues state, it is better to respond to change than tofollow a plan (value V4) strictly.
Regarding the “Mathematical Design,” Section 3.1defined that it should be generated following theconcept of points-of-view; this implies showcasingthe model from the perspective of different stake-holders. Therefore, it also aligns with how the userstories are written. An important element of the“Mathematical Design” is that it is also incremen-tally developed: only a general structure is presentedat the beginning of the project, and then it is itera-tively refined at each iteration; this results on a final,detailed structure, created alongside the projectand model.
Figure 11. Example of “Release Plan” created using FeatureMap.
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 11
To keep the article in scope, only two possibleviews of the case vignette are presented; both ofthem belong to a general, pre-coding state.
In particular, Figure 12 shows a general structurediagram, similar to SE packages; it is targeted atmodellers and project leaders and shows whichprincipal components are present in the model.Here, as there is a considerable number of inputdata, only those relevant to understand the generalstructure are part of the diagram. Each situation hasan input data that defines what each location is (awarehouse, a sales point, a cross-docking, andothers) and if it is active or not; constraints use thatspecific input data in order to simulate each situ-ation. This is possible because these types of inputdata are written with the same structure (as repre-sented by the arrows). Furthermore, variables arenot present because, at this stage, there is no needto declare them concretely.
However, a decision-maker stakeholder is notusually interested in understanding coding details.Therefore, their point-of-view of the “MathematicalDesign” is different, as seen in Figure 13. Here, theysee how the models are used to obtain the answersthey need: each situation is simulated to obtain indi-vidual results, and then they are compared and pre-sented as a report. In particular, this is using BPMN
(Business Process Model Notation), which is stand-ard nomenclature for showcasing business processes.
6. Discussions and limitations ofthe proposal
Accepting and validating AMs in SE was a long pro-cess, which took around 10-years, requiring practi-tioners to apply these methods in practice andresearchers to study them (Cardozo, Ara�ujo Neto,Barza, Franca, & da Silva, 2010; Dingsøyr et al.,2012; Kneuper, 2017). As a result, ensuring the suc-cess of this article’s proposal would be hurried, sinceit would require applying the same process, by usingagility in many OR interventions (Ormerod, 2008).Although AM advantages in SE are verifiable, itsapplicability to OR interventions may encounter the
Figure 12. “Mathematical Design” for the case vignette, from a developers point-of-view. This is an initial model, previous toany project iteration.
Figure 13. “Mathematical Design” for the case vignette,from a decision-maker client point-of-view. This is an initialmodel, previous to any project iteration.
12 M. VIDONI ET AL.
resistance to move to new practices. Furthermore, itshould be noted that poor project management doesnot necessarily imply its failure or that of its solu-tion. Because of this, this article introduces AMs toOR by stating the feasibility of the adaptation, aim-ing to have more reports on their use and results.
It is important to mention that AMs are compat-ible with existent PSM or Soft-OR techniques, asthey do not prescribe specific techniques. Moreover,AMs-based proposed techniques are not mandatoryeither, as these lifecycles support process flexibilityabove all.
An advantage of the proposal is converting theinformation into tangible artefacts and using this inOR potentially increase its adoption in differentorganisations. SE artefacts and AMs are extensivelyused in Information and CommunicationsTechnologies (ICT) departments as well (Cardozoet al., 2010), and can reinforce the integration, byproviding a seamless link to the use of the newesttechniques (Ranyard, Fildes, & Hu, 2015).
On the other hand, many of the concepts intro-duced here may be considered as basic for SE.Nonetheless, their use and modification to fit ORconcepts and practices is novel. This is done bydefining concepts in common, and clear steps toadopt any AM to OR based on the underlying prin-ciples. As changes usually require a transition periodwhere more effort is made to accept them, then,though AMs are based on the same values and prin-ciples, it is possible that code-centric methods -suchas the XP or Crystal- appeal more to practitioners,as they are closer to current practices.
Because each OR intervention has its intrinsiccharacteristics that depend on the situation beinganalysed, it is not possible to provide a universal,specific sequence on how to adapt every AM, as thiswill ignore the changeable nature that characterisesmost situations. Therefore, training practitioners inthe selected AMs techniques is critical to being ableto use them on OR interventions. Although this isnot part of the scope of the article, the authors haveevaluated the possibility of working in coordinationwith another academics specialized in OR to applythis proposal in future interventions to local indus-tries, to not only train them on AM but also applythese methodologies in practice.
7. Conclusions
This article acknowledges that project managementhas become critical to modern organisations andthat, as a result, OR models should be created takinginto account the continuously changing environ-ments, their role in the decision-making process andthe constant unclear ramifications the target
situations may have. This perspective differentiatesfrom traditional model-focused processes, also study-ing its integration with existing systems. In addition,it contributes to addressing models as systems per se.
SE agility is particularly strong for taking care ofincreasingly demanding projects, favouring the abil-ity to adapt to changes and establishing short deliv-ery deadlines that imply faster investment returns.However, current PSM and Soft-OR focus only onunderstanding the problem and ignoring the otherstages without acknowledging several aspects of afull lifecycle.
As a result, this article proposes the challenge ofadapting and using SE agility to manage OR interven-tions. As it was in SE, an extensive, widespread use ofAMs in OR interventions is mandatory to obtain datageneralizable enough to discuss advantages and draw-backs in more detail. Therefore, this article introducesAMs to OR by stating the feasibility of the adaptation,aiming to have more reports on their use. This isfounded on the fact that many concepts can beadapted from SE to OR, as both disciplines have asimilar project lifecycle, as well as an equivalent evo-lution of information. General guidelines are providedto instruct on how to port Agile Methods to OR proj-ects. This is exemplified in a case study vignette,using both Extreme Programming and Scrum.
However, this is only the front end of a largerproject, enabling possibly future works. The mostrelevant is using AM such as Scrum or XP inreal-world interventions with local industries fromdifferent backgrounds. In addition, other relatedconcepts can be explored, such as formalisingrequirements and quality attributes for OR, as wellas studying the impact that code and design smellshave in this type of projects. Moreover, other spe-cific agile techniques can be explored in the contextof OR interventions, such as estimations, prioritisa-tions and sprint retros, among others. Finally, asagility derives from SE, it would be possible in thefuture -after several agile OR interventions areperformed- to study how both disciplines can beintegrated through this shared project manage-ment process.
Acknowledgements
The authors gratefully acknowledge the anonymousreviewers whose comments and suggestions helpedimprove and clarify this manuscript. Furthermore,authors are especially thankful to Prof Richard Ormerodfor his contributions and insight.
Disclosure statement
No potential conflict of interest was reported bythe authors.
Achimugu, P., Selamat, A., Ibrahim, R., & Mahrin, M.(2014). A systematic literature review of softwarerequirements prioritization research. Information andSoftware Technology, 56(6), 568–585. doi:10.1016/j.infsof.2014.02.001
Ackermann, F. (2012). Problem structuring methods ‘inthe Dock’: Arguing the case for Soft OR. EuropeanJournal of Operational Research, 219(3), 652–658. doi:10.1016/j.ejor.2011.11.014
Ackermann, F., Bawden, R., Bosch, O., Brocklesby, J.,Bryant, J., Buede, D., … White, L. (2009). The case forSoft O.R. INFORMS Pubs Online.
Ackoff, R. (1979). The future of operational research ispast. Journal of the Operational Research Society, 30(2),93–104. doi:10.1057/jors.1979.22
Ahmed, A., Ahmad, S., Ehsan, N., Mirza, E., & Sarwar, S.(2010). Agile software development: Impact on prod-uctivity and quality. IEEE International Conference onManagement of Innovation and Technology (ICMIT)(pp. 287–291). Singapore: IEEE.
Beck, K., Beedle, M., Bennekum, A. C., Cunningham, W.,Fowler, M., Grenning, J., … Thomas, D. (2001).Manifesto for Agile Software Development. Retrieved2018, from http://www.agilemanifesto.org/.
Birrell, N., & Ould, M. (1985). A practical handbook forsoftware development (1st ed.). New York, USA:Cambridge University Press.
Boehm, B. (1986). A spiral model of software develop-ment and enhancement. ACM SIGSOFT SoftwareEngineering Notes, 11(4), 22–24. doi:10.1145/12944.12948
Boehm, B. (2006). A view of 20th and 21st century soft-ware engineering. 28th International Conference onSoftware Engineering (ICSE) (pp. 12–29). Shanghai,China: ACM New York. doi:10.1145/1134285.1134288
Cabrera, D., Cabrera, L., Powers, E., Solin, J., & Kushner,J. (2018). Applying systems thinking models of organ-izational design and change in community operationalresearch. European Journal of Operational Research,3(1), 932–945. doi:10.1016/j.ejor.2017.11.006
Cardozo, E., Ara�ujo Neto, J., Barza, A., Franca, A., & daSilva, F. (2010). Scrum and Productivity in SoftwareProjects: A Systematic Literature Review. 14thInternational Conference on Evaluation andAssessment in Software Engineering (EASE) (pp.131–134). Swindon, UK: BCS Learning & DevelopmentLtd. doi:10.14236/ewic/EASE2010.16
Checkland, P., & Poulter, J. (2010). Soft systems method-ology. In M. Reynolds & S. Holwell (Eds.), Systemsapproaches to managing change: A practical guide (1sted., pp. 191–242). London, UK: Springer London.
Churchman, C. (1967). Wicked problems. ManagementScience, 14(4), B-141–B-146. doi:10.1287/mnsc.14.4.B141
Cockburn, A. (2004). In A. Cockburn & J. Highsmith(Eds.), Crystal clear: A human-powered methodology for
small teams (1st ed., Chapters 2 and 4). USA: Addison-Wesley Professional.
da Silva Filho, M. (2015). Problem structuring methodsrecommendation for a public organization of the Riode Janeiro State. Procedia Computer Science, 55,196–202. doi:10.1016/j.procs.2015.07.033
Dingsøyr, T., Nerur, S., Balijepally, V., & Moe, N. (2012).A decade of agile methodologies: Towards explainingagile software development. Journal of Systems andSoftware, 85(6), 1213–1221. doi:10.1016/j.jss.2012.02.033
Doshi, V., & Patil, V. (2016). Competitor driven develop-ment: Hybrid of extreme programming and featuredriven reuse development. International Conference onEmerging Trends in Engineering, Technology andScience (ICETETS) (pp. 1–6). Pudukkottai, India: IEEE.
Eden, C., & Ackermann, F. (2001). SODA— The princi-ples. In J. Rosenhead & J. Mingers (Eds.), Rational ana-lysis for a problematic world revisited: Problemstructuring methods for complexity (2nd ed., pp. 21–41).New York, United States: John Wiley & Sons.
Frakes, W., & Kang, K. (2005). Software reuse research:Status and future. IEEE Transactions on SoftwareEngineering, 35(7), 529–536. doi:10.1109/TSE.2005.85
Friend, J. (2006). Labels, methodologies and strategicdecision support. Journal of the Operational ResearchSociety, 57(7), 772–775. doi:10.1057/palgrave.jors.2602089
Gligor, D., Esmark, C., & Holcomb, M. (2015).Performance outcomes of supply chain agility: Whenshould you be agile? Journal of OperationsManagement, 33–34, 71–82. doi:10.1016/j.jom.2014.10.008
Haefliger, S., von Krogh, G., & Spaeth, S. (2008). Codereuse in open source software. Management Science,54(1), 180–193. doi:10.1287/mnsc.1070.0748
Hunt, J. (2006). Feature-driven development. En AgileSoftware Construction (Primera ed., pp. 161–182).Londres, Inglaterra: Springer-Verlag London Limited.
IEEE Computer Society. (2014). Guide to the softwareengineering body of knowledge (3rd ed.). IEEE.
ISO/IEC/IEEE. (2011). 42010:2011 – Systems and softwareengineering – Architecture description. In C. I.Engineering (Ed.), Switzerland: InternationalStandardization Organization.
Kneuper, R. (2017). Sixty years of software developmentlife cycle models. IEEE Annals of the History ofComputing, 39(3), 41–54. doi:10.1109/MAHC.2017.3481346
Livermore, J. (2007). Factors that impact implementingan agile software development methodology. IEEESoutheastCon. (pp. 82–86). Richmond, VA, USA: IEEE.
Melo, C., Cruzes, D., Kon, F., & Conradi, R. (2011). Agileteam perceptions of productivity factors. AgileConference (AGILE) (pp. 57–66). Salt Lake City, UT,USA: IEEE.
Midgley, G., Cavana, R., Brocklesby, J., Foote, J., Wood,D., & Ahuriri-Driscoll, A. (2013). Towards a newframework for evaluating systemic problem structuringmethods. European Journal of Operational Research,229(1), 143–154. doi:10.1016/j.ejor.2013.01.047
Mingers, J. (2011). Soft OR comes of age—but not every-where!. Omega, 39(6), 729–741. doi:10.1016/j.omega.2011.01.005
Mingers, J., & White, L. (2010). A review of the recentcontribution of systems thinking to operationalresearch and management science. European Journal of
Ormerod, R. (2008). The transformation competence per-spective. Journal of the Operational Research Society,59(11), 1435–1448. doi:10.1057/palgrave.jors.2602482
Ormerod, R. (2014). The mangle of OR practice: Towardsmore informative case studies of ‘technical’ projects.Journal of the Operational Research Society, 65(8),1245–1260. doi:10.1057/jors.2013.78
Project Management Institute. (2017). A guide to the pro-ject management body of knowledge (PMBOKVR guide)(6th ed.). USA: Project Management Institute, Inc.
Qumer, A., & Henderson-Sellers, B. (2008). An evaluationof the degree of agility in six agile methods and itsapplicability for method engineering. Information andSoftware Technology, 50(4), 280–295. doi:10.1016/j.infsof.2007.02.002
Ranyard, J., Fildes, R., & Hu, T. (2015). Reassessing thescope of OR practice: The influences of problem struc-turing methods and the analytics movement. EuropeanJournal of Operational Research, 245(1), 1–13. doi:10.1016/j.ejor.2015.01.058
Royce, W. (1970). Managing the development of largesoftware systems. Proceedings of the IEEE WESCON(pp. 328–338). IEEE.
Schramm, V., & Schramm, F. (2018). An approach forsupporting problem structuring in water resourcesmanagement and planning. Water ResourcesManagement, 32(9), 2955–2968. doi:10.1007/s11269-018-1966-9
Schwaber, K. (1995). SCRUM development process. In J.Sutherland, C. Casanave, J. Miller, P. Patel, & G.Hollowell (Eds.), Business object design and implemen-tation (1st ed., pp. 117–134). Austin, Texas, USA:Springer, London.
Smith, C., & Shaw, D. (2019). The characteristics of prob-lem structuring methods: A literature review. EuropeanJournal of Operational Research, 274(2), 403–416. doi:10.1016/j.ejor.2018.05.003
Tarhan, A., & Yilmaz, S. (2014). Systematic analyses andcomparison of development performance and productquality of Incremental Process and Agile Process.Information and Software Technology, 56(5), 477–494.doi:10.1016/j.infsof.2013.12.002
Vlaanderen, K., Jansen, S., Brinkkemper, S., & Jaspers, E.(2011). The agile requirements refinery: ApplyingSCRUM principles to software product management.Information and Software Technology, 53(1), 58–70.doi:10.1016/j.infsof.2010.08.004
