Transfer defect learning

transcript

Transfer Defect Learning

Jaechang Nam The Hong Kong Un iver s i t y o f Sc ience and Techno logy , Ch ina

Sinno Jialian Pan I n s t i t u te for I n focomm Research , S in gapore

Sunghun Kim The Hong Kong Un iver s i t y o f Sc ience and Techno logy , Ch ina

Defect Prediction

• Hassan et al.@ICSE`09, Predicting Faults Using the Complexity of Code Changes

• D’Ambros et al.@MSR`10, An Extensive Comparison of Bug Prediction Approaches

• Rahman et al.@ICSE`12, Recalling the Impression of Cross-Project Defect Prediction

• Hata et al.@ICSE`12, Bug Prediction based on Fine -grained Module histories

• …

Program Prediction Model (Machine learning)

Future defects

Training prediction model

Test set

Training set

Training prediction model

Test set

Training set

M1 M2 … M19 M20 Class

11 5 … 53 78 Buggy

… … … … … …

1 1 … 3 9 Clean

M1 M2 … M19 M20 Class

2 1 … 2 8 ?

… … … … … …

13 6 … 45 69 ?

Cross prediction model

Target project (Test set)

Source project (Training set)

Cross-project Defect Prediction

“Training data is often not available, either

because a company is too small or it is the first

release of a product”

Zimmerman et al.@FSE`09, Cross-project Defect Prediction

Cross-project Defect Prediction

“Training data is often not available, either

because a company is too small or it is the first

release of a product”

Zimmerman et al.@FSE`09, Cross-project Defect Prediction

“For many new projects we may not have enough

historical data to train prediction models.”

Rahman, Posnett, and Devanbu @ICSE`12, Recalling the

“Imprecision” of Cross-project Defect Prediction

Cross-project defect prediction

• Zimmerman et al.@FSE`09

– “We ran 622 cross-project predictions and found

only 3.4% actually worked.”

Worked, 3.4%

Not worked, 96.6%

Cross-company defect prediction

• Turhan and Menzies et al.@ESEJ`09

– “Within-company data models are still the best”

Cross Cross with a NN

filter

Within

Avg. F-measure

• Rahman, Posnett, and Devanbu@FSE`12

Cross Within

Avg. F-measure

Cross prediction results

F-measure

Cross Within Cross Within Cross Within

Equinox JDT Lucene

Approaches of Transfer Defect Learning

Normalization TCA

• Data preprocessing for training and test data Normalization

• A state-of-the art transfer learning algorithm

• Transfer Component Analysis TCA

• Adapted TCA for cross-project defect prediction

• Decision rules to select a suitable data normalization option TCA+

Data Normalization

• Adjust all feature values in the same scale

– E.g., Make Mean = 0 and Std = 1

• Known to be helpful for classification

algorithms to improve prediction

performance [Han et al. 2012].

Normalization Options

• N1: Min-max Normalization (max=1, min=0)

[Han et al., 2012]

• N2: Z-score Normalization (mean=0, std=1)

[Han et al., 2012]

• N3: Z-score Normalization only using source

mean and standard deviation

• N4: Z-score Normalization only using target

mean and standard deviation

• Adapted TCA for cross-project defect prediction

• Decision rules to select a suitable data normalization option TCA+

Transfer Learning

Traditional Machine Learning (ML)

Learning

System

Learning

System

Transfer Learning

Learning

System

Learning

System

Transfer Learning

Learning

System

Learning

System

Knowledge

Transfer

Transfer Learning

Learning

System

Learning

System

Transfer Learning

Learning

System

Learning

System

Knowledge

Transfer

A Common Assumption in

Traditional ML

Pan and Yang@TKDE`10, Survey on Transfer Learning

• Same distribution

Traditional ML

Cross Prediction

Traditional ML

Transfer Learning

Transfer Component Analysis

• Unsupervised Transfer learning

– Target project labels are not known.

• Must have the same feature space

• Make distribution difference between

training and test datasets similar

Pan et al.@TNN`10, Domain Adaptation via Transfer Component Analysis

Transfer Component Analysis (cont.)

• Feature extraction approach

– Dimensionality reduction

– Projection

• Map original data

in a lower-dimensional feature space

– Projection

2-dimensional feature space

– Projection

– C.f. Principal Component Analysis (PCA)

Target domain data Source domain data

PCA TCA

Preliminary Results using TCA

F-measure

21 *Baseline: Cross-project defect prediction without TCA and normalization

Baseline NoN N1 N2 N3 N4 Baseline NoN N1 N2 N3 N4

Safe Apache Apache Safe

Preliminary Results using TCA

F-measure

21 *Baseline: Cross-project defect prediction without TCA and normalization

Prediction performance of TCA

varies according to different

normalization options!

Baseline NoN N1 N2 N3 N4 Baseline NoN N1 N2 N3 N4

Safe Apache Apache Safe

• Adapted TCA for cross-project defect prediction • Decision rules to select a suitable data

normalization option TCA+

TCA+: Decision rules

• Find a suitable normalization for TCA

• Steps

– #1: Characterize a dataset

– #2: Measure similarity

between source and target datasets

– #3: Decision rules

#1: Characterize a dataset

Dataset A Dataset B

DIST={dij : i,j, 1 ≤ i < n, 1 < j ≤ n, i < j}

#2: Measure Similarity between source and target

• Minimum (min) and maximum (max) values of

• Mean and standard deviation (std) of DIST

• The number of instances

#3: Decision Rules

• Rule #1

– Mean and Std are same NoN

• Rule #2

– Max and Min are different N1 (max=1, min=0)

• Rule #3,#4

– Std and # of instances are different

N3 or N4 (src/tgt mean=0, std=1)

• Rule #5

– Default N2 (mean=0, std=1)

EVALUATION

Experimental Setup

• 8 software subjects

• Machine learning algorithm

– Logistic regression

ReLink (Wu et al.@FSE`11)

Projects # of metrics

(features)

Apache 26

(Source code) Safe

AEEEM (D’Ambros et al.@MSR`10)

Projects # of metrics

(features)

Apache Lucene (LC)

(Source code,

Churn,

Entropy,…)

Equinox (EQ)

Eclipse JDT

Eclipse PDE UI

Mylyn (ML)

Experimental Design

Test set

Training set

Within-project defect prediction

Experimental Design

Cross-project defect prediction with TCA/TCA+

TCA/TCA+

RESULTS

ReLink Result

33 *Baseline: Cross-project defect prediction without TCA/TCA+

F-measure

Baseline TCA TCA+ Within

Safe Apache Apache Safe Safe ZXing

Baseline TCA TCA+ Within Baseline TCA TCA+ Within

ReLink Result F-measure

Source Target

Safe Apache

Zxing Apache

Apache Safe

Zxing Safe

Apache ZXing

Safe ZXing

Average

Baseline

Within

Target Target

*Baseline: Cross-project defect prediction without TCA/TCA+

AEEEM Result

35 *Baseline: Cross-project defect prediction without TCA/TCA+

F-measure

Baseline TCA TCA+ Within

JDT EQ PDE LC PDE ML

Baseline TCA TCA+ Within Baseline TCA TCA+ Within

AEEEM Result F-measure

Cross Source Target

JDT EQ

PDE LC

JDT ML

PDE ML

Average

Baseline

Within

Source Target

Threats to Validity

• Systems are open-source projects.

• Experimental results may not be

generalizable.

• Decision rules in TCA+ may not be

generalizable.

Future Work

• Transfer defect learning on different

feature space

– e.g., ReLink AEEEM

AEEEM ReLink

• Local models using Transfer Learning

• Adapt Transfer learning in other Software

Engineering (SE) problems

– e.g., Knowledge from mailing lists

Bug triage problem

Conclusion

• TCA+

– TCA

• Make distributions of source and target similar

– Decision rules to improve TCA

– Significantly improved cross-project defect prediction performance

• Transfer Learning in SE

– Transfer learning may benefit other

prediction and recommendation systems in

SE domains.

Transfer defect learning

Documents