Recommendation Systems for Issue Management

transcript

Lund University - cs.lth.se/markus_borg/

Recommendation Systems for Issue ManagementMARKUS BORG, SOFTWARE ENGINEERING RESEARCH GROUP

• PhD student 2010-• Software engineering• Requirements-test alignment• Member of SWELL

• Supervisor: Per Runeson

Markus Borg

Per Runeson

• MSc CS and Eng. 2007• Developer, ABB (2007-2010)

• Compiler and editor development• Safety-critical systems• Embedded development

Agenda

• Background– Bug report inflow in large projects– Recommendation systems

• Overview of some ongoing bug research– Duplicate detection– Team assignment– Effort prediction– Change impact analysis

There Will Be Bugs!

Issue Reports as Batons in the Development “Relay Race”

Release

Change Board

Project managerTester

Issue handover

Developer

Developer Tester

Developer

Architect

Analyst

Designer

Challenges in the Bug Overflow

• 10 000s of bug reports in large projects– Continous inflow of bug reports– Limited resources to deal with it

• Q1. Which issues do we already know about?• Q2. Who should investigate the issue?• Q3. How long time will it take to correct this bug?• Q4. If we make a corrective change, what is the impact?

Recommendation System

• General definition

“a software application that aim to support users in their decision-making while interacting with large information spaces”

– Amazon, Netflix, RateBeer, Filmtipset...

• In software engineering”a software application that provides information items estimated to be valuable for a software engineering task in a given context”

Two main approaches to recommendations

Bug example

Q1. Which issues do we already know about?

Duplicate Detection

Issue Duplicate Detection

Network Analysis of Bug Reports

Subgraph Analysis in Bug Networks

Bug star

One central bug report pointing at several others

Dense ringMost bug reports are connected

Textual Similarity Analysis

1. Study natural language content

2. Calculate overlap

3. Recommend duplicates

Calculating Textual Similarity

Issue reports

Represent

Calculate angles in vector space

Reported Results

• Results typically reported as:– ”% found among top 10 recommendations”

• Sony Ericsson 40%• Firefox 60%, 90%• Eclipse 40%, 70%• Open Office 60%, 65%• Mozilla Foundation 45%, 65%• Android 60% (Jens?)

• Are these good results? Good question.

Q2. Who should investigate the issue?

Team assignment

Bug Assignment

Bug tossing!

Machine Learning to Find Patterns

• Bugs found in India late in testing phase are severe?• Bugs reports containing term ”termination” are costly?• Bugs reported on fridays with attached stack traces tend to

be deferred to later releases?

Automated Bug Assignment – Supervised Machine Learning

Machine learning

Recommend team

How to Represent a Bug Report?

• Function• Severity• System Version• Submit Date• Submitter Location

etc.• … And the text! Title and description.

Experimental Setup

Ericsson

Company A

Machine learning

Pre-processing & feature selection

25-35 teams to choose between!

Results

• Prediction accuracy using state-of-the-art machine learning:– Ericsson 30-70%– Company A 30%

• At Ericsson– In line with accuracy of human assignment– Deploying tool in selected projects

• Textual features improve predictions, but training times increase a lot!– Working on finding a good balance

Q3. How long time will it take to correct this bug?

Effort prediction

Bug Report Clustering – Unsupervised Machine Learning

Automatically cluster bug reports based on natural language in report

Prediction of Bug Resolution Time

1. A new bug is submitted

2. Put it in the right cluster

3. Predict its resolution time

Results

• Clustered bug reports based on textual content– Android– Eclipse– Company A

• Average resolution times in bug clusters are different– Statistically significant

• Is it a good way to predict resolution times?– No, at least not yet…

Q4. If we make a corrective change, what is the impact?

Change impact analysis

Recommendations for Impact Analysis

Mandated by safety standards!

Reqs.Tests

Networks of Impacted Artifacts

Data mining in archive of previous impact analysis reports

Identification of Key Artifacts

Recommend Possible Impact

1. Textual similarity analysis

2. Follow links in network

Recommend Possible Impact

3. Rank recommendations– Network statistics– Textual similarity

Results

• Experiments in a laboratory setting– 30% of previous impact among top 5 recommendations– 40% among top 10 recommendations

• Prototype evaluation in industrial setting ongoing...

Summary

• Issue inflow in large projects can be daunting

• On the other hand, issue reports contain important information!

• With the right tools, the information overflow can be tamed to recommendations!

Recommendation Systems for Issue Management

Science