Theory Building in RE - The NaPiRE Initiative

transcript

Technische Universität München

Theory building in RE - The NaPiRE* Initiative

* Naming the Pain in Requirements Engineering (www.re-survey.org)

Daniel MéndezTechnical University of Munich

www.mendezfe.org@mendezfe

RE@40 Seminar, 2017Kappel am Albis, Switzerland

“[…] judging a theory by assessing the number, faith, and vocal energy of its supporters […] basic political credo of contemporary religious maniacs”

— Imre Lakatos, 1970

‣ In Software Engineering, too, truth often lies in power rather than in empirical evidence.

“[…] judging a theory by assessing the number, faith, and vocal energy of its supporters […] basic political credo of contemporary religious maniacs”

— Imre Lakatos, 1970

RE research and practice often dominated by conventional wisdom

For a problem-driven research, we nee a reliable body of knowledge about industrial practices and problems.

Naming the Pain in Requirements Engineering (NaPiRE)

‣Foundation for problem-driven research

Overall goal: (First) Theory on industrial practices and problems in Requirements Engineering

201722 countries

51 researchers from 43 institutions

201410 countries

23 researchers

20121 country,

2 researchers

Overall approach: By the community for the community

Bi-yearly replicated, globally distributed family of surveys (w/ synthesis) on practices and problems‣Collaborative instrument design‣ Independent surveys in different countries‣Collaborative synthesis and publication

www.re-survey.org

Principles

‣Theory-centric‣Anonymity, but closed‣ Full open science practices

201722 countries

201410 countries

23 researchers

www.re-survey.org

Principles

201722 countries

www.re-survey.org

Principles

Excerpt results NapiRE 2015/16

Country # Companies(full survey completion)

Response rate

Austria 14 72 %Germany 41 36,8 %UK (Northern Ireland) 18 39,7 %Canada 13 75 %USA 15 36,2 %Estonia 8 25,7 %Finland 15 37,5 %Norway 10 70,8 %Sweden 20 51,8 %Brazil 74 35,3 %Total 228

crosscutting (60)

public (28)

sector (28)

enterprise (20)

finance (26)

planning (20)

resource (23)

automotive (15)

healthcare (14)

management (16)

business (9)

energy (11)

insurance (11)

logistics (8)

telecom (9) transportation (8)

aerospace (5)

education (5)

gas (5) geoprocessing (4)

informed (5)

intelligence (4) oil (5)

process (5)

aviation (3)

communication (3) construction (3)

defence (3) e-commerce (3)

entertainment (3) games (3)

human (3)

shipping (3)

workforce (3)

agriculture (2)

chemicals (2)

customer (2)

electronic (2)

railway (2) relationship (2)

scientific (2) software (2)

automation (1)

government (1)

funerary (1)

goods (1) industrial (1)

networking (1)

pulp (1)

steel (1)

Distributed over industry sectors

Status quo in practices used [1]‣Requirements Elicitation‣Requirements Documentation‣Requirements Change and Alignment ‣Requirements Engineering Standards‣Requirements Engineering Improvement

Problems in RE [2]‣Problems and criticality‣Causes‣Effects

[2] Mendez, Wagner, et al. Naming the pain in requirements engineering Contemporary problems, causes, and effects in practice , 2017[1] Wagner, Mendez, et al. Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys, 2017

Req Elicitation TechniqueInterviewScenarioPrototypingFacilitated MeetingsObservation

Req Documentation TechniqueStructured req listDomain/business process modelUse case modelGoal modelData modelNon-functional reqTextualSemi-formalFormal

Technology

Req Test Alignment ApproachReq review by testerCoverage by testsAcceptance criteriaTest derivation from models

Req Change ApproachProduct backlog updateChange requestsTrace management Impact analysis

Activity

Req Elicitation

Req Documentation

Req Change Management

Req Test Alignment

P 6-13

P 14-20

P 21-24

Req Engineer

Test Engineer

Req Standard ApplicationPracticeControlTailoring

Req Eng Process Standard

P 25-28

Req Standard DefintionComplianceDevelopmentTool supportQuality assuranceProject managementKnowledge transferProcess complexityCommunication demandWilligness to changePossibility of standardisation

P 26-45

Req Improvement MeansContinuous improvementStrengths/weaknessesOwn business unit/role

Req Eng ImprovementP 46--48

Status quo in practices used [1]‣Requirements Elicitation‣Requirements Documentation‣Requirements Change and Alignment ‣Requirements Engineering Standards‣Requirements Engineering Improvement

Problems in RE [2]‣Problems and criticality‣Causes‣Effects

[2] Mendez, Wagner, et al. Naming the pain in requirements engineering Contemporary problems, causes, and effects in practice , 2017[1] Wagner, Mendez, et al. Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys, 2017

Req Elicitation TechniqueInterviewScenarioPrototypingFacilitated MeetingsObservation

Req Documentation TechniqueStructured req listDomain/business process modelUse case modelGoal modelData modelNon-functional reqTextualSemi-formalFormal

Technology

Req Test Alignment ApproachReq review by testerCoverage by testsAcceptance criteriaTest derivation from models

Req Change ApproachProduct backlog updateChange requestsTrace management Impact analysis

Activity

Req Elicitation

Req Documentation

Req Change Management

Req Test Alignment

P 6-13

P 14-20

P 21-24

Req Engineer

Test Engineer

Req Standard ApplicationPracticeControlTailoring

Req Eng Process Standard

P 25-28

Req Standard DefintionComplianceDevelopmentTool supportQuality assuranceProject managementKnowledge transferProcess complexityCommunication demandWilligness to changePossibility of standardisation

P 26-45

Req Improvement MeansContinuous improvementStrengths/weaknessesOwn business unit/role

Req Eng ImprovementP 46--48

Teaser

In scope of artefact-based standard (reference) models

Unclearrolesandresponsabilitiesatcustomerside

WeakqualificationofREteammembers

Subjectiveinterpretations

Stakeholderslackbusinessvisionandunderstanding

(1.28%)Customerdoesnotknowwhathewants

(6.41%)

LackofexperienceofREteammembers

Complexityofdomain

Insufficientresources

Lackoftime

Conflictingstakeholderviewpoints

(8.97%)

Stricttimeschedulebycustomer

Missingreq.specificationtemplate

MissingITprojectexperienceatcustomerside

Weakqualificationofstakeholders

(6.41%)

Input(33%)

(3.85%)

(6.41%)

(2.56%)

(1.28%)

(2.56%)

(10.26%)

(2.56%)

(1.28%)

(1.28%)(1.28%)

(1.10%)

Method(38%)

Req.tooabstract

Missingcompletenesscheckofrequirements

Insufficientagility

Communicationflawsbetweenteamandcustomer

Lackofawell-definedREprocess

ComplexityofRE

Unclearterminology

(11.54%)

(1.28%)(1.28%)(2.56%) (2.56%)

(1.28%) (1.28%)

People(17%)

Tools(3%)

(1.28%)

Communicationflawscustomer

Missingengagementbycustomer

Missingcustomerinvolvement (2.56%)

(1.28%)

(1.28%) Policyrestrictions (1.28%)

(1.28%)MissingREawarenessatcustomerside (1.28%)

Weakmanagementatcustomerside

(1.28%)

Missingdirectcommunicationtocustomer

Missinginvolvementofdev. (2.56%)

Poorprojectmanagement

Missingsolutionapproach

Organization(9%)

Toohighteamdistribution

Manycustomers

LanguageBarriers

Notfollowing thecommunicationplan

Missingtoolsupport (1.28%)

Customer(19%)

DesignorImplementation(9%)

Product(22%)

ValidationorVerification(3%)

Increasednumberofreq.changes

(16.18%)

IncompleteReq.

(2.94%)

Decreaseduseracceptance

Customerdissatisfaction

(2.94%)

Needforpostimplementation

PoorproductqualityDecreasedbusinessvalue

(10.29%)

(8.82%)(2.94%)

Increasedcommunication

Implementationofirrelevantreq.

Effortoverrun

Timeoverrun

Budgetoverrun

Decreasedefficiency(overall)

Increasednumberofchangerequests

Wrongestimates

Increaseddifficultyofreq.elicitation

Validationofreq.becomesdifficult

ProjectorOrganization(47%)

(2.94%)

(1.47%)

(13.24%)

(5.88%)

(4.41%)

(2.94%)(2.94%)

(2.94%)

(1.47%) (1.47%)

(2.94%)

Poor(system)designquality (1.47%)

Misunderstanding(overall)

Latedecision (2.94%)

(2.94%) (2.94%)

Overalldemotivation

InformationgetslostIncreasedcomplexityinRE

(6.41%)

Complexityofdomain

Lackoftime

(8.97%)

(6.41%)

Input(33%)

(3.85%)

(6.41%)

(2.56%)

(1.28%)

(2.56%)

(10.26%)

(2.56%)

(1.28%)

(1.28%)(1.28%)

(1.10%)

Method(38%)

Req.tooabstract

Insufficientagility

ComplexityofRE

Unclearterminology

(11.54%)

(1.28%)(1.28%)(2.56%) (2.56%)

(1.28%) (1.28%)

People(17%)

Tools(3%)

(1.28%)

Organization(9%)

Manycustomers

LanguageBarriers

Customer(19%)

Product(22%)

(16.18%)

IncompleteReq.

(2.94%)

(10.29%)

(8.82%)(2.94%)

Effortoverrun

Timeoverrun

Budgetoverrun

Wrongestimates

(2.94%)

(1.47%)

(13.24%)

(5.88%)

(4.41%)

(2.94%)(2.94%)

(2.94%)

(1.47%) (1.47%)

(2.94%)

(2.94%) (2.94%)

Overalldemotivation

(6.41%)

Complexityofdomain

Lackoftime

(8.97%)

(6.41%)

Input(33%)

(3.85%)

(6.41%)

(2.56%)

(1.28%)

(2.56%)

(10.26%)

(2.56%)

(1.28%)

(1.28%)(1.28%)

(1.10%)

Method(38%)

Req.tooabstract

Insufficientagility

ComplexityofRE

Unclearterminology

(11.54%)

(1.28%)(1.28%)(2.56%) (2.56%)

(1.28%) (1.28%)

People(17%)

Tools(3%)

(1.28%)

Organization(9%)

Manycustomers

LanguageBarriers

Customer(19%)

Product(22%)

(16.18%)

IncompleteReq.

(2.94%)

(10.29%)

(8.82%)(2.94%)

Effortoverrun

Timeoverrun

Budgetoverrun

Wrongestimates

(2.94%)

(1.47%)

(13.24%)

(5.88%)

(4.41%)

(2.94%)(2.94%)

(2.94%)

(1.47%) (1.47%)

(2.94%)

(2.94%) (2.94%)

Overalldemotivation

(6.41%)

Complexityofdomain

Lackoftime

(8.97%)

(6.41%)

Input(33%)

(3.85%)

(6.41%)

(2.56%)

(1.28%)

(2.56%)

(10.26%)

(2.56%)

(1.28%)

(1.28%)(1.28%)

(1.10%)

Method(38%)

Req.tooabstract

Insufficientagility

ComplexityofRE

Unclearterminology

(11.54%)

(1.28%)(1.28%)(2.56%) (2.56%)

(1.28%) (1.28%)

People(17%)

Tools(3%)

(1.28%)

Organization(9%)

Manycustomers

LanguageBarriers

Customer(19%)

Product(22%)

(16.18%)

IncompleteReq.

(2.94%)

(10.29%)

(8.82%)(2.94%)

Effortoverrun

Timeoverrun

Budgetoverrun

Wrongestimates

(2.94%)

(1.47%)

(13.24%)

(5.88%)

(4.41%)

(2.94%)(2.94%)

(2.94%)

(1.47%) (1.47%)

(2.94%)

(2.94%) (2.94%)

Overalldemotivation

InformationgetslostIncreasedcomplexityinRECauses rather not due to missing guidance via reference

models, but due to organisational and even social issues.

Do teams applying agile practices avoid these issues?

(6.41%)

Complexityofdomain

Lackoftime

(8.97%)

(6.41%)

Input(33%)

(3.85%)

(6.41%)

(2.56%)

(1.28%)

(2.56%)

(10.26%)

(2.56%)

(1.28%)

(1.28%)(1.28%)

(1.10%)

Method(38%)

Req.tooabstract

Insufficientagility

ComplexityofRE

Unclearterminology

(11.54%)

(1.28%)(1.28%)(2.56%) (2.56%)

(1.28%) (1.28%)

People(17%)

Tools(3%)

(1.28%)

Organization(9%)

Manycustomers

LanguageBarriers

Customer(19%)

Product(22%)

(16.18%)

IncompleteReq.

(2.94%)

(10.29%)

(8.82%)(2.94%)

Effortoverrun

Timeoverrun

Budgetoverrun

Wrongestimates

(2.94%)

(1.47%)

(13.24%)

(5.88%)

(4.41%)

(2.94%)(2.94%)

(2.94%)

(1.47%) (1.47%)

(2.94%)

(2.94%) (2.94%)

Overalldemotivation

Small companies(1-50)

Medium companies (51-250)

Large companies(> 251)

#1 Problem Incomplete/hidden requirements

Communication flaws customer

Incomplete/hidden requirements

#2 Problem Communication flaws customer

Moving targets

#3 Problem Underspecified requirements

Communication flaws project team

#4 Problem Communication flaws project team

Separation reqs. / solutions

Time boxing

#5 Problem Time boxing Weak access customer needs

Underspecified requirements

Do teams applying agile practices avoid these issues?

Moving targets

Time boxing

No.Do teams applying agile practices avoid these issues?

Moving targets

Time boxing

Not at all.Do teams applying agile practices avoid these issues?

How can we make use of such insights?

How can we make use of such insights?‣ Support problem-driven research

Raw data‣ Anonymised‣ Sanitised‣ Curated (w/ codebooks)

‣ Support problem-driven research

Open data repository

tera-PROMISE http://openscience.us/repo/ (will probably change)

Open data repository

tera-PROMISE http://openscience.us/repo/ (will probably change)

Problem-driven research

supports

Phenomena and measurements

How can we make use of such insights?‣ Immediate operationalisation for practical environments

Example: evidence-based Risk

Management*

* Based on two MSc. theses: Priscilla Mafra at UFF, Brazil; Michaela Tießler at TUM, Germany

1 Build holistic model with cause-effect relationships

‣ Immediate operationalisation for practical environments

Implement Bayesian network including context factors2

3 Operationalise as a risk assessment approach

Build holistic model with cause-effect relationships1

2 Implement Bayesian network including context factors

How can we make use of such insights?‣ Immediate operationalisation for practical environments

4 Offer situation-spec. guidelines (corrective/preventive measures)

Preparation

Where do we stand right now?

• Preparation instruments about to be completedNext steps• Data disclosure previous runs• Survey implementation & pilot phase next run‣ Start independent data collections;

(before summer break)

ISERN 2017 (October)• Discussion 1st Results• Analysis strategy

Survey / Data Collection(Coordinated local replications)

Survey ++Data Analysis and Publication

(Synthesis, planning, joint publication)

We are here...

ISERN 2018• Data analysis and publication (cont.)

You are cordially invited to join us!www.re-survey.org

Thank you!

Daniel MéndezTechnical University of Munich

www.mendezfe.org

@mendezfe