50.530: Software Engineering Sun Jun SUTD. Week 8: Race Detection.

transcript

50.530: Software Engineering

Sun JunSUTD

Week 8: Race Detection

Agenda

• A remainder of concurrent bugs.• Common ways of coordinating

threads/processes• How to detect potential concurrent bugs?

Concurrent Bugs

Thread10

Thread2

count++ count++

count = 0

count = 1 count = 1

count = 2

count should be 2?

Concurrent Bugs

Thread1

Thread2

02 11 20

03 12 21 30

22 3113

w2count=1

Is it OK if it could be 1 or 2?

02 11 20

03 12 21 30

22 3113

What do we do to make sure the red transitions won’t happen?

Coordinate the threads based on timing

“You wait for 5 minutes, I will start right away”

Using Thread.sleep()

• Example: a program with two threads, one printing 1,2,3,5 and the other printing 4 repeatedly such that the print out is

1,2,3,4,51,2,3,4,51,2,3,4,5…

Using Thread.sleep()

Thread1

0print 1,2,3

wait for 10 seconds

Thread2

0wait for 5 seconds

print 4

print 5

Expected BehaviorRightThreadLeftThread

timeprint 1,2,3

sleeping

sleeping print 4

print 5,1,2,3 sleeping

sleeping print 4

This would work except it doesn’t SleepExample.java

02 11 20

03 12 21 30

22 3113

Coordinate the threads based on locking

If an object is to be shared by multiple threads, make sure the object is locked and there is only one key to unlock it.

Using LocksThread1

ThirdBlood.java

0acquire lock

release lock

Thread2

0 acquire lock

release lock

Rules for Locks

• Update related state variables in a single atomic operation

• For each mutable variable that may be accessed by more than one thread, all assesses to that variable must be performed with the same lock held.

• Every shared, mutable variable should be guarded by exactly one lock. Make it clear to maintainers which lock that is.

• For every invariant that involves more than one variable, all the variables involved in that invariant must be guarded by the same lock.

02 11 20

03 12 21 30

22 3113

Coordinate the threads by sending messages

“I am working on this. I will send you a message when I am done.”

wait() and nofity()

Producer/Consumer Pattern

Producer Thread 1Producer Thread 2…

Consumer Thread 1Consumer Thread 2…

BoundedBuffer

addItem removeItem

BufferFixed.java

Race Conditions

A race condition occurs when two or more threads can access shared data at the same time and at least one of the threads is writing.

Thread 1 Thread 2

count++ count++

Deadlockpublic class LeftRightDeadlock { private final Object left = new Object (); private final Object right = new Object ();

public void leftRight () {synchronized (left) { synchronized (right) { doSomething(); }}

public void rightLeft () {synchronized (right) { synchronized (left) {doSomethingElse(); }}

Thread A Thread B

lock leftlock right

try to lock right wait for lock left

wait foreverwait forever

Examplepublic void transferMoney (Account from, Account to, int amount) { synchronized (from) {

synchronized (to) { if (from.getBalance() < amount) {

//raiseException } else {

from.debit(amount);to.credit(amount)

Is it deadlocking?

Examplepublic void transferMoney (Account from, Account to, int amount) { synchronized (from) {

synchronized (to) { if (from.getBalance() < amount) {

//raiseException } else {

from.debit(amount);to.credit(amount)

How can transferMoney deadlock?Thread A: transferMoney(myAccount, yourAccount, 1)Thread B: transferMoney(yourAccount, myAccount, 1)Check out: DemonstrateDeadlock.java

Research Discussion

Would Delta Debugging work for concurrent programs?Or the bug localization methods?

ERASER: A DYNAMIC DATA RACE DETECTOR FOR MULTI-THREADED PROGRAMS

Savage et al. SOSP’97

Static vs. Dynamic Analysis

Static (program) analysis: is the analysis of computer programs that is performed without actually executing the programs.– Often would result in false alarms

Dynamic (program) analysis: is the analysis of computer programs that is performed by executing programs on a real or virtual processor. – Often not exhaustive

Dynamic Analysis

Instrument the program

Obtain execution traces

Analyze the trace and wonder:

Is there a potential problem like race condition?

e.g., adding print statement everywhere

e.g., which are sequences of statements

Example

1. Thread 1: count++

Can we spot any problem from this trace?

Answer: Yes, it seemed that nothing would prevent us from reordering statement 1 and 2. Since they modify the same variable, a race condition is found!

Example

1. Thread 1: lock(mu)

3. Thread 1: unlock(mu)

4. Thread 2: lock(mu)

6. Thread 2: unlock(mu)

Answer: No, because of the happens-before relationship.

Happens-before

ExampleTrace

2. Thread 1: send message to thread 2

3. Thread 1: v = v + x;

4. Thread 2: y = z + x;

5. Thread 2: get message from thread 1

Answer: No, because of the happens-before relationship.

Happens-before

Happens-Before Approach

Input: a sequence of statements <s1, s2, …>Output: true if there is a race condition

For any pair of statements s and t (assuming s is “earlier”) if (s and t are from two different threads and s and t access some shared memory and either s modifies the memory or t does) {

if (there is no happens-before relation from s to t) {report potential race condition;

Example

Thread 1: obj1.methodA()

Thread 2: obj2.methodB()

It depends on whether obj1 and obj2 are disjoint.

Example

Thread 1: obj1.methodA()

Thread 2: obj2.methodB()

The question is: whether A and B are disjoint?

Define Happens-Before

Given a sequence of statements<a, b, c, …>

• If statement f and u (u is after f in the sequence) are from the same thread, f happens-before u.

• If f sends a message and u receives the message, f happens-before u.

• If f unlocks an object and u locks the same object, f happens-before u.

• …• If f happens before u and u happens-before y, f happens-

before y. Is this complete?

Subtle Happens-Before

Do these form a race condition?

Thread 1: count++

Thread 2: while (x <= 0) {

Thread 2: Thread.yield()

Thread 2: endwhile

Thread 1: x++

Thread 2: endwhile

Thread 2: count++

Subtle Happens-Before

Thread 1: count++

Thread 2: Thread.yield()

Thread 2: endwhile

Thread 1: x++

Thread 2: endwhile

Thread 2: count++

Happens-Before Algorithm

• In order to get precise happens-before relation, we often need to analyze the whole program.

• Race condition detection based on happens-before relation is very costly – no known efficient implementation so far.

• Race condition detection based on happens-before relation is not exhaustive even with a perfect happens-before relation.

Exercise 1Trace

Thread 1: count++

Thread 1: lock(mu)

Thread 1: x++

Thread 1: unlock(mu)

Thread 2: lock(mu)

Thread 2: x++

Thread 2: count++

Is there a race condition according to the happens-before approach?

Is there a race condition?

Lockset Algorithm

Recall (rules for locks): • For each mutable variable that may be

accessed by more than one thread, all assesses to that variable must be performed with the same lock held.

The lockset algorithm is designed to check if the rules is properly implemented. If not, it reports potential race condition.

Lockset Algorithm

For each shared variable x set lockset(x) = *;Endfor

For each access to x by thread t lockset(x) = {locks held by t at the time} intersect lockset(x) if (lockset(x) is an empty set) {

report race condition; }}

* is a special flag denoting the set of all possible locks.

Example

Trace Locks held by thread 1 and 2 Lockset(count)

Thread 1: lock(mu1) {} for thread 1; {} for 2 {mu1, mu2}

Thread 1: count++ {mu1} for thread 1; {} for 2 {mu1}

Thread 1: unlock(mu1) {mu1} for thread 1; {} for 2 {mu1}

Thread 2: lock(mu2) {} for thread 1; {} for 2 {mu1}

Thread 2: count++ {} for thread 1; {mu2} for 2 {}

Thread 2: unlock(mu2)

Exercise 2

Thread 1: count++

Thread 1: lock(mu)

Thread 1: x++

Thread 2: lock(mu)

Thread 2: x++

Thread 2: count++

How does Lockset find the race in this example?

Patching Lockset

Locking may not be necessary in some scenarios• Initialization: shared variables are frequently

initialized without holding a lock• Read-share data: Some shared variables are written

during initialization only and are read-only thereafter. These can be safely accessed without locks.

• Read-write locks: Read-write locks allow multiple readers to access a shared variable, but allow only a single writer to do so.

Improving Lockset

For each shared variable, we only monitor its status according to the state machine and check for race condition only at state “Shared-Modified”

Improving Lockset

When a variable is in Shared-Modified state, we do the checking slightly differently.

For each read-access of x by thread t { lockset(x) = {locks held by t at the time} intersect lockset(x) if (lockset(x) is an empty set) {

For each write-access of x by thread t { lockset(x) = {locks held by t at the time in write mode} intersect lockset(x) if (lockset(x) is an empty set) {

Example: On Variable count

Trace Locks held Write Locks held Status Lockset(count)

T1: lock(mu, R) {} for T1 and T2 {} for T1 and T2 Virgin *

T1: x = count+1 {mu} for T1; {} for T2

{} for T1; {} for T2

Virgin *

T1: unlock(mu) {mu} for T1;{} for T2

{} for T1;{} for T2

Virgin *

T2: lock(mu, W) {} for T1;{} for T2

{} for T1;{} for T2

Virgin *

T2: count++ {} for T1;{mu} for T2

{} for T1;{mu} for T2

Exclusive *

T2: unlock(mu) {} for T1;{mu} for T2

{} for T1;{mu} for T2

Exclusive *

Exercise 3

Thread 1: count++

Thread 1: lock(mu)

Thread 1: x++

Thread 2: lock(mu)

Thread 2: x++

Thread 2: count++

Draw the table for this example.

Efficiency of Lockset

• Implemented at the binary level• Maintain a status and lockset for each

memory location• Performance penalty: applications with

lockset implementation slows down 10 to 30 times.

Effectiveness

False Alarms are produced in some scenarios.• Memory reuse: the same memory is reused

for different purposes later (with different locking policy).

• Private locks: user defines their locks without using the ones from the library

• Benign races: Race condition is found but deemed to be benign.

False Alarm: Example

Which is worse for users: false alarms or missing some true races?

HYBRID DYNAMIC DATA RACE DETECTION

O’Callahan et al. PPoPP’03

Exampleclass Main { int globalFlag; ChildThread childThread; void execute () {

globalFlag = 1;childThread = new ChildThread(this);childThread.start();…synchronized (this) { childThread.interrupt();}

class ChildThread extends Thread { Main main;

ChildThread (Main main) {this.main = main;

void run() {if (main.globalFlag == 1) …;…main.childThread = null;

Is there a race condition?

Example: Lockset Algorithmclass Main { int globalFlag; ChildThread childThread; void execute () {

What would the lockset algorithm report?

Example: Lockset Algorithmclass Main { int globalFlag; ChildThread childThread; void execute () {

What would the (not improved) lockset algorithm report?

Two potential races: one globalFlag and the other on childThread.The former is false alarm, why?

The Idea

Race condition detection with Lockset

Define a limited easy-to-check happens-before relation

Filter false alarms with the happens-before relation

Potential race conditions

Filtered race conditions

Exercise 4class Main { int globalFlag; ChildThread childThread; void execute () {

} void run() {

if (main.globalFlag == 1) …;synchronized (main) { main.childThread = null;}

What would the hybrid approach report?

RACE DIRECTED RANDOM TESTING OF CONCURRENT PROGRAMS

Koushik Sen. PLDI’08

Motivation

Hybrid dynamic race detection:• 39 out of 51 reported race conditions for

tomcat are false alarms

Static race detection (from Stanford, 2006)• 13 out of 19 reported race conditions for hedc

(a software) are false alarms

Would you be happy to use these tools?

Motivation

Thread1

Thread2

02 11 20

03 12 21 30

22 3113

w2count=1

Even if we are sure there is a race condition, users might not be convinced if you can’t show it!

Question

Thread1

Thread 2

Thread k

How many possible traces are there?

How about we show the problematic trace by testing the system many times?

Scheduling

threads

Scheduler

Thread1 Thread2 Thread3 Thread4

How can we control the scheduler so that it would show the error, if there is?

Proposal: PaceFuzzer

1. Use hybrid dynamic race detection to find potential race conditions, in the form of pairs of statements (s, t).

2. PaceFuzzer executes the program with a random scheduler

3. Control the scheduler such that it keeps delaying s and t

4. Report race condition if s and t can be executed by different thread next to each other

Example

What are the race conditions according to the hybrid algorithm?Assume there are two test cases, one in which thread 1 runs to finish first and the other in which thread 2 runs to finish first.

Example

• Test 1: thread 1 runs first and then thread 2– Lockset based algorithm: • race condition between statement 1 and 10• race condition between statement 5 and 7

– happen-before filtering• The race condition between 1 and 10 is removed before

statement 4 happens-before 8

Exercise 5

• Test 2: thread 2 runs first and then thread 1– Lockset based algorithm: – happen-before filtering:

What are the race conditions reported in this case?

Example: Case (1, 10)

Statement 1 is enabled, so delay it and let thread 2 go.

Thread 2 has finished; thread 1 proceeds to execute statement 1 and more

Report: No real race condition found! False alarm!

Report: Real race condition identified!

The AlgorithmFor each pair of statements (s, t) { postponed = {} while (some thread is enabled) {

randomly pick one enabled thread which is not in postponed;if execute the picked thread would execute s or t { if a thread in postponed is about to execute the other statement {

report race condition and terminate; } else {

add the thread to postponed; }}else { execute the picked thread for one step;}

} report deadlock if some thread is not finished; }

Exercise 6

lock(L);x = 1;

Apply the algorithm with this modified example for (2, 10).

A Supportive Example

Without using RaceBuzzer• Unlikely x = 1 will be

delayed for long and hence no ERROR will be see.

• With high-probability 8 and 10 will be separated by lock(L) and hence happens-before will fail.

Empirical Study

Conclusion

• There are proposals for detecting race conditions.

• The algorithms/tools must work with large programs.

• The algorithms/tools are often neither sound nor complete– Sound: a race condition found is always a real-one.– Complete: all race conditions are found.

Example

Initially: x = y = 0

thread 1 { x = random.nextInt(1,100000); count++;}

thread 2 { if (x==3771) { count++; }}

Would any dynamic analysis approach be able to find the race condition? And be sure it is not a false alarm?

Can We Do Better?

• Recall the rules– Update related state variables in a single atomic operation– For each mutable variable that may be accessed by more

than one thread, all assesses to that variable must be performed with the same lock held.

– Every shared, mutable variable should be guarded by exactly one lock. Make it clear to maintainers which lock that is.

– For every invariant that involves more than one variable, all the variables involved in that invariant must be guarded by the same lock.

– …Can we do better using other rules?

Example

Thread 1

lock(mu)x++;y++;unlock(mu)

Thread 2

lock(mu)x++;unlock(mu)lock(mu)y++;unlock(mu)

Assume that we have an invariant x==y.

50.530: Software Engineering Sun Jun SUTD. Week 8: Race Detection.

Documents