Transfer Defect Learning

Jaechang Nam The Hong Kong Un iver s i t y o f Sc ience and Techno logy , Ch ina

Sinno Jialian Pan I n s t i t u te for I n focomm Research , S in gapore

Sunghun Kim The Hong Kong Un iver s i t y o f Sc ience and Techno logy , Ch ina

Defect Prediction

• Hassan et al.@ICSE`09, Predicting Faults Using the Complexity of Code Changes

• D’Ambros et al.@MSR`10, An Extensive Comparison of Bug Prediction Approaches

• Rahman et al.@ICSE`12, Recalling the Impression of Cross-Project Defect Prediction

• Hata et al.@ICSE`12, Bug Prediction based on Fine -grained Module histories

• …

2

Program Prediction Model (Machine learning)

Future defects

Training prediction model

3

Test set

Training set

Training prediction model

3

Test set

Training set

M1 M2 … M19 M20 Class

11 5 … 53 78 Buggy

… … … … … …

1 1 … 3 9 Clean

M1 M2 … M19 M20 Class

2 1 … 2 8 ?

… … … … … …

13 6 … 45 69 ?

Cross prediction model

4

Target project (Test set)

Source project (Training set)

Cross-project Defect Prediction

5

“Training data is often not available, either

because a company is too small or it is the first

release of a product”

Zimmerman et al.@FSE`09, Cross-project Defect Prediction

Cross-project Defect Prediction

5

“Training data is often not available, either

because a company is too small or it is the first

release of a product”

Zimmerman et al.@FSE`09, Cross-project Defect Prediction

“For many new projects we may not have enough

historical data to train prediction models.”

Rahman, Posnett, and Devanbu @ICSE`12, Recalling the

“Imprecision” of Cross-project Defect Prediction

Cross-project defect prediction

• Zimmerman et al.@FSE`09

– “We ran 622 cross-project predictions and found

only 3.4% actually worked.”

6

Worked, 3.4%

Not worked, 96.6%

Cross-company defect prediction

• Turhan and Menzies et al.@ESEJ`09

– “Within-company data models are still the best”

7

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Cross Cross with a NN

filter

Within

Avg. F-measure

Cross-project defect prediction

• Rahman, Posnett, and Devanbu@FSE`12

8

0

0.1

0.2

0.3

0.4

0.5

0.6

Cross Within

Avg. F-measure

Cross prediction results

9

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

F-measure

Cross Within Cross Within Cross Within

Equinox JDT Lucene

Approaches of Transfer Defect Learning

10

Normalization TCA

TCA+

11

• Data preprocessing for training and test data Normalization

• A state-of-the art transfer learning algorithm

• Transfer Component Analysis TCA

• Adapted TCA for cross-project defect prediction

• Decision rules to select a suitable data normalization option TCA+

Approaches of Transfer Defect Learning

Data Normalization

• Adjust all feature values in the same scale

– E.g., Make Mean = 0 and Std = 1

• Known to be helpful for classification

algorithms to improve prediction

performance [Han et al. 2012].

12

Normalization Options

• N1: Min-max Normalization (max=1, min=0)

[Han et al., 2012]

• N2: Z-score Normalization (mean=0, std=1)

[Han et al., 2012]

• N3: Z-score Normalization only using source

mean and standard deviation

• N4: Z-score Normalization only using target

mean and standard deviation

13

14

• Data preprocessing for training and test data Normalization

• A state-of-the art transfer learning algorithm

• Transfer Component Analysis TCA

• Adapted TCA for cross-project defect prediction

• Decision rules to select a suitable data normalization option TCA+

Approaches of Transfer Defect Learning

Transfer Learning

15

Transfer Learning

15

Traditional Machine Learning (ML)

Learning

System

Learning

System

Transfer Learning

15

Traditional Machine Learning (ML)

Learning

System

Learning

System

Transfer Learning

Learning

System

Learning

System

Knowledge

Transfer

Transfer Learning

15

Traditional Machine Learning (ML)

Learning

System

Learning

System

Transfer Learning

Learning

System

Learning

System

Knowledge

Transfer

A Common Assumption in

Traditional ML

16

Pan and Yang@TKDE`10, Survey on Transfer Learning

• Same distribution

A Common Assumption in

Traditional ML

16

Pan and Yang@TKDE`10, Survey on Transfer Learning

• Same distribution

Cross Prediction

A Common Assumption in

Traditional ML

16

Pan and Yang@TKDE`10, Survey on Transfer Learning

• Same distribution

Transfer Learning

Transfer Component Analysis

• Unsupervised Transfer learning

– Target project labels are not known.

• Must have the same feature space

• Make distribution difference between

training and test datasets similar

17

Pan et al.@TNN`10, Domain Adaptation via Transfer Component Analysis

Transfer Component Analysis (cont.)

• Feature extraction approach

– Dimensionality reduction

– Projection

• Map original data

in a lower-dimensional feature space

18

Transfer Component Analysis (cont.)

• Feature extraction approach

– Dimensionality reduction

– Projection

• Map original data

in a lower-dimensional feature space

18

2-dimensional feature space

Transfer Component Analysis (cont.)

• Feature extraction approach

– Dimensionality reduction

– Projection

• Map original data

in a lower-dimensional feature space

18

1-dimensional feature space

Transfer Component Analysis (cont.)

• Feature extraction approach

– Dimensionality reduction

– Projection

• Map original data

in a lower-dimensional feature space

18

1-dimensional feature space

Transfer Component Analysis (cont.)

• Feature extraction approach

– Dimensionality reduction

– Projection

• Map original data

in a lower-dimensional feature space

18

1-dimensional feature space

2-dimensional feature space

Transfer Component Analysis (cont.)

• Feature extraction approach

– Dimensionality reduction

– Projection

• Map original data

in a lower-dimensional feature space

– C.f. Principal Component Analysis (PCA)

18

1-dimensional feature space

Transfer Component Analysis (cont.)

19

Pan et al.@TNN`10, Domain Adaptation via Transfer Component Analysis

Target domain data Source domain data

Transfer Component Analysis (cont.)

20

PCA TCA

Pan et al.@TNN`10, Domain Adaptation via Transfer Component Analysis

Preliminary Results using TCA

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

F-measure

21 *Baseline: Cross-project defect prediction without TCA and normalization

Baseline NoN N1 N2 N3 N4 Baseline NoN N1 N2 N3 N4

Safe Apache Apache Safe

Preliminary Results using TCA

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

F-measure

21 *Baseline: Cross-project defect prediction without TCA and normalization

Prediction performance of TCA

varies according to different

normalization options!

Baseline NoN N1 N2 N3 N4 Baseline NoN N1 N2 N3 N4

Safe Apache Apache Safe

22

• Data preprocessing for training and test data Normalization

• A state-of-the art transfer learning algorithm

• Transfer Component Analysis TCA

• Adapted TCA for cross-project defect prediction • Decision rules to select a suitable data

normalization option TCA+

Approaches of Transfer Defect Learning

TCA+: Decision rules

• Find a suitable normalization for TCA

• Steps

– #1: Characterize a dataset

– #2: Measure similarity

between source and target datasets

– #3: Decision rules

23

#1: Characterize a dataset

24

3

1

…

Dataset A Dataset B

2

4

5

8

9

6

11

d1,2

d1,5

d1,3

d3,11

3

1

…

2 4

5

8

9

6 11

d2,6

d1,2

d1,3

d3,11

DIST={dij : i,j, 1 ≤ i < n, 1 < j ≤ n, i < j}

A

#2: Measure Similarity between source and target

• Minimum (min) and maximum (max) values of

DIST

• Mean and standard deviation (std) of DIST

• The number of instances

25

#3: Decision Rules

• Rule #1

– Mean and Std are same NoN

• Rule #2

– Max and Min are different N1 (max=1, min=0)

• Rule #3,#4

– Std and # of instances are different

N3 or N4 (src/tgt mean=0, std=1)

• Rule #5

– Default N2 (mean=0, std=1)

26

EVALUATION

27

Experimental Setup

• 8 software subjects

• Machine learning algorithm

– Logistic regression

28

ReLink (Wu et al.@FSE`11)

Projects # of metrics

(features)

Apache 26

(Source code) Safe

ZXing

AEEEM (D’Ambros et al.@MSR`10)

Projects # of metrics

(features)

Apache Lucene (LC)

61

(Source code,

Churn,

Entropy,…)

Equinox (EQ)

Eclipse JDT

Eclipse PDE UI

Mylyn (ML)

Experimental Design

29

Test set

(50%)

Training set

(50%)

Within-project defect prediction

Experimental Design

30

Target project (Test set)

Source project (Training set)

Cross-project defect prediction

Experimental Design

31

Target project (Test set)

Source project (Training set)

Cross-project defect prediction with TCA/TCA+

TCA/TCA+

RESULTS

32

ReLink Result

33 *Baseline: Cross-project defect prediction without TCA/TCA+

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

F-measure

Baseline TCA TCA+ Within

Safe Apache Apache Safe Safe ZXing

Baseline TCA TCA+ Within Baseline TCA TCA+ Within

ReLink Result F-measure

34

Cross

Source Target

Safe Apache

Zxing Apache

Apache Safe

Zxing Safe

Apache ZXing

Safe ZXing

Average

Baseline

0.52

0.69

0.49

0.59

0.46

0.10

0.49

TCA

0.64

0.64

0.72

0.70

0.45

0.42

0.59

TCA+

0.64

0.72

0.72

0.64

0.49

0.53

0.61

Within

Target Target

0.64

0.62

0.33

0.53

*Baseline: Cross-project defect prediction without TCA/TCA+

AEEEM Result

35 *Baseline: Cross-project defect prediction without TCA/TCA+

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

F-measure

Baseline TCA TCA+ Within

JDT EQ PDE LC PDE ML

Baseline TCA TCA+ Within Baseline TCA TCA+ Within

AEEEM Result F-measure

36

Cross Source Target

JDT EQ

LC EQ

ML EQ

…

PDE LC

EQ ML

JDT ML

LC ML

PDE ML

…

Average

Baseline

0.31

0.50

0.24

…

0.33

0.19

0.27

0.20

0.27

…

0.32

TCA

0.59

0.62

0.56

…

0.27

0.62

0.56

0.58

0.48

…

0.41

TCA+

0.60

0.62

0.56

…

0.33

0.62

0.56

0.60

0.54

…

0.41

Within

Source Target

0.58

…

0.37

0.30

…

0.42

Threats to Validity

• Systems are open-source projects.

• Experimental results may not be

generalizable.

• Decision rules in TCA+ may not be

generalizable.

37

Future Work

• Transfer defect learning on different

feature space

– e.g., ReLink AEEEM

AEEEM ReLink

• Local models using Transfer Learning

• Adapt Transfer learning in other Software

Engineering (SE) problems

– e.g., Knowledge from mailing lists

Bug triage problem

38

Conclusion

• TCA+

– TCA

• Make distributions of source and target similar

– Decision rules to improve TCA

– Significantly improved cross-project defect prediction performance

• Transfer Learning in SE

– Transfer learning may benefit other

prediction and recommendation systems in

SE domains.

39