Scalable Statistical Scalable Statistical Bug IsolationBug Isolation
Scalable Statistical Scalable Statistical Bug IsolationBug Isolation
Authors: B. Liblit, M. Naik, A.X. Zheng, A. Authors: B. Liblit, M. Naik, A.X. Zheng, A. Aiken, M. I. JordanAiken, M. I. JordanPresented by S. Li Presented by S. Li
Outline• Background
– Definitions– Motivations
• Contributions– Algorithm– Visualization the algorithm– Experiments
• Personal Comments
Definitions• A bug predicate is denoted by P
– A P is associated with a particular program point. For instance, If (ptr == NULL) , or int a = 10;
– , if P1 , P2 are associated with the same
program point. – P is observed to be true at least once when running
R, denoted by R(P) = 1, otherwise R(P) = 0
• A bug is denoted by B (or b)• A bug profile is denoted by
21 PP
Β
Definitions (cont.)• A bug profile includes a set of failure
runs, which share b as the cause.– more than one bug occur in failure runs–
• R(P) = 1 indicates that P is a bug predictor, Likely,
•
ji BΒ
Β
BR
bugs. all of predicates hasSuch that
.predicates all ofset theis where,
S
SSS pp
Motivations • A traditional technique involved in
statistical bug prediction – Regularized logistic regression, which
select predicates that best predicate outcome of every run
• Scalability problems lie in large-scale programs
Motivations cont.• Scalability problems of
Regularized logistic regression– The set P is logically redundant– It’s difficult to achieve the actual
important predicates associated with specific bugs causing different failure.
Contributions• To Highlight Contributions
– To propose a statistical debugging algorithm• To isolate bugs that includes multiple undiagnosed
bugs
– To perform better than earlier corresponding algorithms
– To validate the algorithm by experiments– To reveal circumstances for bugs to happen
as well as frequencies of failure runs
Statistical Debugging algorithm
• To automatically isolating multiple bugs
• To select S • To rank the predicators in S from the
most to least important. • To let predicators in S and the
associated metrics be available to help fix the most serious bugs
Statistical Debugging algorithm cont.
• Steps:– Identify the most important bug B
• Not bug B but a predicate P closely correlated with its bug profile
– Fix B, and repeat• Simulating the program’s behavior without bug b
Β
Statistical Debugging algorithm cont.
• To identify the bug • To select predicates that are the most
likely to correspond to its bug profile– P1,P2,P3, …, P, ranked in the order of
importance– R(P) = 1– Bug profiles , unknown size and
membership
Β
Β
Statistical Debugging algorithm cont.
• To repeat to fix bug B– To discard any run such that R(P) = 1– To recursively apply the algorithm to
the remaining runs
– To prune P = {P1,P2,P3, …, PB} by:• Reducing the importance of predictors of B• Re-ranking predictors P, for instance,
allowing other predicators to rise to the top in subsequent iterations.
Statistical Debugging algorithm cont.
• To analyze simple codes to introduce equations in the algorithm. Consider the following C code:– f = …; Line (a)– if (f == NULL) { Line (b)
• X = 0; Line (c)• *f; } Line (d)
The bugs in this exampleis deterministic ,
because…
Statistical Debugging algorithm cont.
• Non-deterministic bugs, considering the following codes, – f = …; Line (a)– if (f == NULL) { Line (b)
• X = 0; Line (c)• if (….) f =.. // some valid pointer… • *f; } Line (d)
The bugs in this exampleis non- deterministic
With respect to (b)
Statistical Debugging algorithm cont.
• The probability that P being true implies failure. F(P): the number of failing runs in which P is observed to be true. S(P): the number of successful runs in which P is observed to be true.
) truebe toobserved |Pr()( PCrashPFailure
)()(
)()(
:Estimated
PFPS
PFPFailure
Statistical Debugging algorithm cont.
• Failure(P) = 1.0, a bug is deterministic for P, equivalently, P is never observed to be true in a successful run, S(P)=0
• Failure(P) < 1.0, non-deterministic
)()(
)()(
PFPS
PFPFailure
Statistical Debugging algorithm cont.
• Failure(P) is not enough, considering…– f = …; Line (a)– if (f == NULL) { Line (b)
• X = 0; Line (c)• *f; } Line (d)
• Failure(f == NULL) =1.0 , good• As well, Failure(x == 0) =1.0 , why?
x==0 always true, only failures reach it
Statistical Debugging algorithm cont.
• Thus, just because Failure(P) is high does not mean P is the cause of a bug, only means this predicate is checked on a path of failures.
• In the case of (x==0), the condition causing failure is made earlier, e.g.
(f == NULL)
Statistical Debugging algorithm cont.
• It is introduced to address the issue. Not only by the chance that it implies failure, but also how much difference of the P is observed to be true vs. simply reaching it where the P is checked.
• To eliminate the predicates irrelevant to the bug, like (x==0) in the above example
)observed |Pr()( PCrashPContext
Statistical Debugging algorithm cont.
)()()( PContextPFailurePIncrease
• In the above example Failure(x==0)=Context(x==0)=1.0 and so Increase(x==0)=0;
• Conclusion: a predicate P with• no useful for predication and be discarded.
0)( PIncrease
Visualization of Algorithm
• Thermometer is used for visualization of experiments • The length of the thermometer: # of runs where a
predicate is observed. • Black band on the left: Context(P); red band: Increase
(P); white band: # of successful runs; S(P)
Visualization of Algorithm cont.
• It shows F(P) after discarded the negative increase(P)
• The large white band reveals these predicates are non-deterministic.
• The very narrow red band indicate that Increase scores are small. With high increase scores
Super-bug predicate, combining Multiple bugs
Visualization of Algorithm cont.
Visualization of Algorithm cont.
• The following suggestions of metric of predicates are made from the above observation
)log())(log(
1)(
12
)(Improtance
);()(Importance
);()(Importance
NumFPFpIncrese
P
PIncreaseP
PFP
Experiments • To validate the statistic debugging
algorithm in five case studies. • To determine how many runs needed ,
let importanceN(P) be the importance of P using N runs. So,
Importance32,000(P) – ImportanceN(P)<0.2
Experiments
Personal Comments • Likes
– Well structure, problems addressed, then proposed solutions addressed, step by step
– Using a real and simple example to explain problems and difficulties that lies in research
– Giving statistical interpretation by visualization, using their observation to explain the abstract mathematic equations
Personal Comments• Dislikes
– They do not mention whether their research could be extended for isolation potential bugs, e.g. bugs with less importance, which probably cause failure in future
– The dark (red) band and grey (pink) dark band in pictures are not very clear if this paper is only white/ black.
