How not to do Java Concurrency And how to find if you did it wrong Mark Winterrowd 2014-09-30.

transcript

How not to do Java ConcurrencyAnd how to find if you did it wrong

Mark Winterrowd

2014-09-30

Introductions

• Coverity

• Me

• You!

• Bob

Copyright Coverity, Inc., 20142

Meet Bob

Bob's development philosophy:

• Speed is everything

• Clever tweaks are their own reward

• Trying it out once or twice is plenty testing

Your manager has assigned him to help you out!

Your Mission: Keep your code correct!• Bob will write code mirroring mistakes seen in the

• For novices: preparation for handling your own Bob

• For experts: Examples occurring in real code

• Modeled off of code seen in Tomcat, Eclipse and Android

Patterns discussed here are low level• Prefer to use higher level structures

• e.g., java.util.concurrent

• Hide uses behind higher level calls or objects

• Sometimes unavoidable

Multithreaded Lazy Initialization

Or, sacrificing correctness for speed

Task: Lazily initialize a singleton

In a single-threaded program:

private static MyObj inst = null;

public static MyObj getInst() {

if(inst == null) {

inst = new MyObj();

return inst;

This won't work in a multithreaded program.

Simple Fix:

private static MyObj inst = null;

static final Object initLock = new Object();

synchronized(initLock) {

if(inst == null) {

inst = new MyObj();

return inst;

Bob objects: "This requires you to wait to get the lock every time!"

Bob's Performance Optimization

inst = new MyObj();

return inst;

"Since we only need to set inst the first time, synchronizing to check for null is unnecessary!"

if(inst == null) {

Is Bob right?Is his code correct?

Bob's optimized code in action

Thread 1 Thread 2

if(inst == null)

synchronized(initLock)inst = new MyObj();return inst;

No longer a singleton!

The JVM can start and stop threads at will• Holding a lock prevents another thread from

executing code guarded by the same lock object

• Bob says "In that case, I have an idea!"

"Also check in the synchronized block!"private static MyObj inst = null;

static final Object initLock = new Object();

if(inst == null)

"Now, we can't have inst be initialized multiple times!"

} return inst;}

if(inst == null)

inst = new MyObj();

Is Bob right? Is his code correct?

Let's take a look at this line

inst = new MyObj();

MyObj() {

this.field1 = someVal1;

this.field2 = someVal2;

A raw MyObj is created before running the constructor

(raw MyObj).field1 = someVal1;(raw MyObj).field2 = someVal2;inst = (raw MyObj);

JVM re-orders code at runtime!• Must have same result in current thread

• Cannot move code out of or through a synchronized block• Reordering inside or into a synchronized is fine!

public static MyObj getInst() { if(inst == null) synchronized(initLock) { if(inst == null) {

inst = new MyObj();

} } return inst;}

inst = (raw MyObj)inst.field1 = someValinst.field2 = someVal2

This instruction ordering in action:

Thread 1 Thread 2

if(inst == null)

synchronized(initLock)

inst = (raw MyObj);

return inst;Returned an uninitialized MyObj!

What is the name of this bad pattern?

if(inst == null)

Preventing reordering bugs

• Hold the lock while checking the field for null

if(inst == null)

inst = new MyObj();

return inst;

if(inst == null)Back to our original solution!

"This problem seems unlikely. Is fixing it

worth the slowdown?"

What slowdown? How unlikely?

• Uncontested lock acquisition is fast in modern JVMs

• In computing, unlikely can happen often

• In 2004, a developer opened Eclipse bug 50178 with this comment:

"I know I'm picky, but [...] getManifest has an example of what is known as "Double-Checked Locking problem [...] BundleLoader.createClassLoader has the same problem"

In the wild: Eclipse bug 50178Reply in 2006:"Synchronizing the places where we lazily create ClassLoader/BundleLoaderProxy objects will cause measurable slowdown. To fix this properly would require at least three additional syncs [...] I added todo comments in the code where double checks are currently being done."A month later:"We are currently having a very hard to reproduce bug when loading preference pages of our RCP application [...] I traced the problem to getBundleLoader [...] When looking at [two methods], I saw they both contained the TODO for this bug.

Isn't it better to have slightly less performance instead of possible threading bugs?"

Bug 50178 as reported in Coverity Connect

Premature Optimization...

• Especially dangerous in concurrency

• Rare bugs may be likely in a user's workflow

• Slow locking may be unnecessary contention• Is the lock protecting too much data?

• Is the lock protecting unrelated critical sections?

• Can some expensive operations be safely moved outside of a locked region?

Choosing your locks

Or rather, what not to choose

Make this code thread safe

static Object[] items;

void update(Object newItem) {

"Synchronize on items so another

thread can't change items." Object[] oldArr = items;

items = new Object[items.length + 1]

items[items.length - 1] = newItem;

for(int i=0; i < oldArr.length; i++){

items[i] = oldArr[i];

synchronized(items) {

What does synchronizing on a field do? • Does not block accesses from other threads to that

• Acquires a lock on the field's contents

Thread 1

synchronized(items)

items = new Object[...]

synchronized(items)items = new Object[...]

Different contents!

Object[ ] oldArr = items

items[...] = newItem

Thread 2

Don't lock on mutable fields

static Object[] items;

void update(Object newItem) {

synchronized(

Object[] newAr = new Object[items.length+1];

newAr[newAr.length - 1] = newItem;

for(int i=0; i < items.length; i++){

newArr[i] = items[i]

items = newAr;

private static final Object lock = new Object();

itemslock

Tomcat bug 46990: Locking a mutable field

"Whilst there aren't any

explicit bugs caused by this,

it may be behind some of the harder to

reproduce bugs."

Bob sends you the following code for review

class AppCtx { private static SysCtx sCtx;

public synchronized SysCtx getSysCtx() { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; }}

"synchronized ensures this

code is thread-safe!"

What does the synchronized modifier do?

public synchronized SysCtx getSysCtx() {

if(sCtx == null) { sCtx = new SysCtx();}return sCtx;

synchronized(this) {

Two AppCtx, each in their own thread

AppCtx 1 AppCtx 2

sCtxnull

unlocked

synchronized(this)Static AppCtx

Members

AppCtx1 AppCtx2

if(sCtx == null)

synchronized(this)

if(sCtx == null)

sCtx = new SysCtx()

InitializedInitialized

return sCtx;

sCtx = new SysCtx()return sCtx;

No longer a singleton!

Guard static members with static locks

public SysCtx getSysCtx() { synchronized( ) { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; } }}

private static final Object lock=new Object();

thislock

In the wild: Android bug 12015587

Caused loss of display info, user name, and permissions

Wait and Notify

and threads communicating poorly

Thread 1 consumes results from Thread 2LinkedList<QueueItem> jobQueue;

// Called in Thread 1

void processItemFromQueue() {

QueueItem item = null;

synchronized(qLock) {

item = jobQueue.remove();

processItem(item);

// Called in Thread 2void putItemInQueue(QueueItem item) { synchronized(qLock) { jobQueue.add(item); }}

What if jobQueue is empty?

The wait/notify pattern

• lock.wait( ): current thread stops running, releases lock, and is placed on lock's wait set

• lock.notifyAll( ): Starts up all threads in the wait set, which attempt to re-acquire lock.

• Call wait when your thread needs another thread to satisfy some condition

• Call notifyAll when your thread has satisfied a condition for another thread

wait and notify in actionExecuting

Instructionslock

Locked By

Unlocked

Wait Set

synchronized(lock)

lock.wait( )

Blocked on lock

synchronized(lock)

lock.wait( )

synchronized(lock)

lock.wait( )

lock.notifyAll( )

release lock

synchronized(lock)

release lock

You see this in a review of Bob's code:void processItemFromQueue() {

QueueItem item = null;

// Hold the lock as briefly as possible!

if(jobQueue.empty())

qLock.wait();

processItem(item);

void addItemToQueue(QueueItem item) {

jobQueue.add(item);

qLock.notifyAll();

Is Bob's code correct?

Bob's code in action:Executing

Instructions

Locked By

Unlocked

Wait Set

Consumer

qLock.wait()

Blocked on qLocksynchronized(qLock)

Consumer Producer

synchronized(qLock)

ProducerjobQueue.add(item)

release qLock

qLock.notifyAll()

No waiting threads to notify

returns true

Consumer is waiting while items are in the queue!

Waiting on a stale condition is a bug• Another thread can change the state between the

check and the wait

• At best, causes unnecessary delays• What if future notifications were blocked on this task?

• Always check wait condition while holding lock

Bob comes back with a second version...

Code review round #2

QueueItem item;

Bob: "Now the wait will only occur when the queue is empty!"

if(jobQueue.empty()) {

processItem(item);

qLock.wait();

Is Bob right?Is his code correct?

"A thread can also wake up without being notified, interrupted, or timing out, a so-called spurious wakeup."-Object.wait() documentation

Bob's code in action:

Executing Instructions

Locked By

Unlocked

Wait Set

ConsumerqLock.wait()

Blocked on qLock

synchronized(qLock)

Consumer Producer

A spurious wakeup occurs!

jobQueue.remove()

jobQueue empty!

Check your wait condition inside a loop!

QueueItem item;

qLock.wait();

processItem(item);

whileif(jobQueue.empty())

In the wild: Eclipse bug 366048

• Deadlock when ctrl-x was used to cut text.

• Diagnosed as a wait occurring and never waking up.

• Remains unfixed, timeout to wait was set to 30s.

The bug (in 2004 code)

In code from 2005This bug was "fixed" with a 30 second wait timeout in 2013.

Preventing new concurrency bugs

and fixing the ones you have

Don't get overly clever!

• Do you need a multithreaded program?

• Do you need lazy initialization?• Initialize using static{}

• Is there a higher level structure to replace wait/notify?• Our example: java.util.concurrent.BlockingQueue

• Sometimes it's unavoidable, or hard to remove.

Standard methods help somewhat

• Education

• Code reviews

• Testing

Nondeterminism limits effectiveness

Is there anything else we can do?

Don’t fix pre-existing problems

Manual Inspection• Responsible for finding many of our sampled

fixed defects

• One tip we found in Bugzilla:

• No guarantee anyone will notice right away• Most of the fixed bugs found by inspection had

been present for over 3 years.

• Why not have a machine inspect your code instead?

is Slow

"Search[ing] for the 'synchronized' keyword goes a long way"

Static Analysis: Automatic code inspection• Open source static analysis FindBugs™ finds many

issues

• But Coverity's static analysis finds• more real bugs

• with better explanations

• and a lower rate of false reports

• Also, we integrate with FindBugs

Analyze your code for free!

• Coverity's SCAN program provides free defect reports for open source programs.• http://scan.coverity.com

• Coverity's CodeSpotter: free-to-use SaaS analysis• http://www.code-spotter.com

• Also a commercial enterprise solution for proprietary software• http://www.coverity.com

More about Java Concurrency

• Java Language Specification

• "Java Concurrency in Practice" by Goetz et al

• "Java theory and practice: Are all stateful Web applications broken?" by Goetz• http://www.ibm.com/developerworks/library/j-jtp09238/

• "Double checked locking is broken", many signers.• http://www.cs.umd.edu/~

pugh/java/memoryModel/DoubleCheckedLocking.html

How not to do Java Concurrency And how to find if you did it wrong Mark Winterrowd 2014-09-30.

Documents