Non-blocking Michael-Scott queue algorithm

Alexey FyodorovJUG.ru Group

• Programming

• Algorithms

• Concurrency

What is this talk about?

• Programming

• Algorithms

• Concurrency

Areyousureyouneedit?

What is this talk about?

For concurrency beginners

SorryPlease go to another room

For concurrency beginners

SorryPlease go to another room

For non-blocking programming beginners

A short introduction

For concurrency beginners

SorryPlease go to another room

For non-blocking programming beginners

A short introduction

For advanced concurrent programmers

CAS-based queue algorithm

You have another room!

12:10Non-blocking Michael-Scott queue algorithm

Alexey Fyodorov

Easily scale enterprise applications

using distributed data gridsOndrej Mihaly

Main Models

Shared Memory

write + read

Similar to how we program it

Concurrent Programming

Main Models

Shared Memory Messaging

write + read send + onReceive

Similar to how we program it

Similar to how a real hardware works

Distributed Programming

Concurrent Programming

Advantages of ParallelismResource utilization Utilization of several cores/CPUs

aka PERFORMANCE

Advantages of ParallelismResource utilization

Simplicity Complexity goes to magic frameworks• ArrayBlockingQueue• ConcurrentHashMap• Akka

Utilization of several cores/CPUsaka PERFORMANCE

Advantages of ParallelismResource utilization

Async handling

Simplicity

Utilization of several cores/CPUsaka PERFORMANCE

Complexity goes to magic frameworks• ArrayBlockingQueue• ConcurrentHashMap• Akka

Responsible services, Responsible UI

Disadvantages of Locking

• Deadlocks

Disadvantages of Locking

• Deadlocks

• Priority Inversion

Disadvantages of Locking

• Deadlocks

• Priority Inversion

• Reliability• What will happen if lock owner die?

Disadvantages of Locking

• Deadlocks

• Priority Inversion

• Reliability• What will happen if lock owner die?

• Performance• Scheduler can push lock owner out• No parallelism inside a critical section!

Amdahl’s Law

α non-parallelizable part of the computation

1-α parallelizable part of the computation

p number of threads

Amdahl’s Law

α non-parallelizable part of the computation

1-α parallelizable part of the computation

p number of threads

S = #

α$%&α'

If-Modify-Write

volatile int value = 0;

Can we run it in multithreaded environment?

if (value == 0) {value = 42;

}

If-Modify-Write

volatile int value = 0;

No atomicityif (value == 0) {

value = 42;}

}

Compare-And-Set

int value = 0;

LOCKif (value == 0) {

value = 42;}

UNLOCK

Introducing a Magic Operation

value.compareAndSet(0, 42);

int value = 0;

Simulated CASlong value;

synchronized long get() { return value;

}

synchronized long compareAndSwap(long expected, long newValue) {long oldValue = value;if (oldValue == expected) {

value = newValue;}return oldValue;

}

synchronized boolean compareAndSet(long expected, long newValue) {return expected == compareAndSwap(expected, newValue);

}

Simulated CASlong value;

synchronized long get() { return value;

}

synchronized long compareAndSwap(long expected, long newValue) {long oldValue = value;if (oldValue == expected) {

value = newValue;}return oldValue;

}

synchronized boolean compareAndSet(long expected, long newValue) {return expected == compareAndSwap(expected, newValue);

}

Simulated CASlong value;

synchronized long get() { return value;

}

synchronized long compareAndSwap(long expected, long newValue) {long oldValue = value;if (oldValue == expected) {

value = newValue;}return oldValue;

}

synchronized boolean compareAndSet(long expected, long newValue) {return expected == compareAndSwap(expected, newValue);

}

Simulated CASlong value;

synchronized long get() { return value;

}

synchronized long compareAndSwap(long expected, long newValue) {long oldValue = value;if (oldValue == expected) {

value = newValue;}return oldValue;

}

synchronized boolean compareAndSet(long expected, long newValue){return expected == compareAndSwap(expected, newValue);

}

Compare and Swap — Hardware Support

compare-and-swapCAS

load-link / store-conditionalLL/SC

cmpxchg ldrex/strex lwarx/stwcx

Atomics in JDK

AtomicReference• ref.get()

• ref.compareAndSet(v1, v2)

• ...

AtomicLong• i.get()

• i.compareAndSet(42, 43)

• i.incrementAndGet(1)

• i.getAndAdd(5)

• ...

java.util.concurrent.atomic

Atomics in JDK

AtomicReference• ref.get()• ref.compareAndSet(v1, v2)

• ...

AtomicLong• i.get()• i.compareAndSet(42, 43)

• i.incrementAndGet(1)

• i.getAndAdd(5)

• ...

java.util.concurrent.atomic

Example. Atomic CounterAtomicLong value = new AtomicLong();

long get() {return value.get();

}

void increment() {long v;do {

v = value.get();} while (!value.compareAndSet(v, v + 1));

}

AtomicLong value = new AtomicLong();

long get() {return value.get();

}

void increment() {long v;do {

v = value.get();} while (!value.compareAndSet(v, v + 1));

}

Example. Atomic Counter

Atomics.Questions?

Non-blocking Guarantees

Wait-Free Per-thread progress is guaranteed

Non-blocking Guarantees

Wait-Free Per-thread progress is guaranteed

Lock-Free Overall progress is guaranteed

Non-blocking Guarantees

Wait-Free Per-thread progress is guaranteed

Lock-Free Overall progress is guaranteed

Obstruction-Free Overall progress is guaranteed if threads don’t interfere with each other

CAS-loopdo {

v = value.get();} while (!value.compareAndSet(v, v + 1));

A. Wait-FreeB. Lock-FreeC. Obstruction-Free

CAS-loopdo {

v = value.get();} while (!value.compareAndSet(v, v + 1));

A. Wait-FreeB. Lock-FreeC. Obstruction-Free

*for modern hardware supporting CAS or LL/SC

Stack & Concurrency

class Node<E> {

final E item;

Node<E> next;

Node(E item) {this.item = item;

}

}

...

class Node<E> {

final E item;

Node<E> next;

Node(E item) {this.item = item;

}

}

E3

E1

E2

E3

E1

E2

top

E3

E1

E2

top

item1

Thread 1

E3

E1

E2

top

item1

Thread 1

E3

E1

E2

top

item2item1

Thread 1 Thread 2

E3

E1

E2

top

item2item1

Thread 1 Thread 2

E3

E1

E2

item2item1

Thread 1 Thread 2top

E3

E1

E2

item2item1

Thread 1 Thread 2

We need a synchronization

top

Non-blocking Stack

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

AtomicReference<Node<E>> top;E3

E1

E2

top

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

AtomicReference<Node<E>> top;E3

E1

E2

item

top

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

E3

E1

E2

item

AtomicReference<Node<E>> top;top

newHead

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

E3

E1

E2

AtomicReference<Node<E>> top;

item

top

newHead

oldHead

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

AtomicReference<Node<E>> top;E3

E1

E2

item

top

newHead

oldHead

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

AtomicReference<Node<E>> top;E3

E1

E2

item

top

newHead

oldHead

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

AtomicReference<Node<E>> top;E3

E1

E2

item

top

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

AtomicReference<Node<E>> top;E3

E1

E2

item

top

newHead

oldHead

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

E3

E1

E2

AtomicReference<Node<E>> top;top

itemnewHead

oldHead

void push(E item) {Node<E> newHead = new Node<>(item);Node<E> oldHead;do {

oldHead = top.get();newHead.next = oldHead;

} while (!top.compareAndSet(oldHead, newHead));}

E3

E1

E2

AtomicReference<Node<E>> top;top

item

E pop() {Node<E> newHead;Node<E> oldHead;do {

oldHead = top.get();if (oldHead == null) return null;newHead = oldHead.next;

} while (!top.compareAndSet(oldHead, newHead));return oldHead.item;

}

E3

E1

E2

top

Non-blocking Stack.Questions?

Non-blocking Queue

Michael and Scott, 1996https://www.research.ibm.com/people/m/michael/podc-1996.pdf

Threads help each other

Non-blocking queue

class LinkedQueue<E> {

static class Node<E> {E item;AtomicReference<Node<E>> next;

Node(E item, AtomicReference<Node<E>> next) {this.item = item;this.next = next;

}}

Node<E> dummy = new Node<>(null, null);AtomicReference<Node<E>> head = new AtomicReference<>(dummy);AtomicReference<Node<E>> tail = new AtomicReference<>(dummy);

}

class LinkedQueue<E> {

static class Node<E> {E item;AtomicReference<Node<E>> next;

Node(E item, AtomicReference<Node<E>> next) {this.item = item;this.next = next;

}}

Node<E> dummy = new Node<>(null, null);AtomicReference<Node<E>> head = new AtomicReference<>(dummy);AtomicReference<Node<E>> tail = new AtomicReference<>(dummy);

}

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.compareAndSet(null, newNode); tail.compareAndSet(curTail, curTail.next.get());

} while (!success); }

tail

dummy 1 2

head

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.compareAndSet(null, newNode); tail.compareAndSet(curTail, curTail.next.get());

} while (!success); }

tail

dummy 1 2 item

head

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.compareAndSet(null, newNode); tail.compareAndSet(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNode

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.compareAndSet(null, newNode); tail.compareAndSet(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.compareAndSet(null, newNode);tail.compareAndSet(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.compareAndSet(null, newNode); tail.compareAndSet(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.compareAndSet(null, newNode); tail.compareAndSet(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.compareAndSet(null, newNode); tail.compareAndSet(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); tail.CAS(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); tail.CAS(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); // true tail.CAS(curTail, curTail.next.get()); // true

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); tail.CAS(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); // true tail.CAS(curTail, curTail.next.get()); // false

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); tail.CAS(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); // false tail.CAS(curTail, curTail.next.get()); // false

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

another

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); tail.CAS(curTail, curTail.next.get());

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); // false tail.CAS(curTail, curTail.next.get()); // true

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

another

void put(E item) {Node<E> newNode = new Node<>(item, null);boolean success;do {

Node<E> curTail = tail.get();success = curTail.next.CAS(null, newNode); // false tail.CAS(curTail, curTail.next.get()); // true

} while (!success); }

tailhead

dummy 1 2 item

newNodecurTail

anotherHELP

Synchronization

Blocking

lock + unlock

Invariant: before & after

lock-based

Synchronization

Blocking Non-blocking

lock + unlock CAS-loop

Invariant: before & after Semi-invariant

CAS-basedlock-based

public void put(E item) {Node<E> newNode = new Node<>(item, null);while (true) {

Node<E> currentTail = tail.get();Node<E> tailNext = currentTail.next.get();if (currentTail == tail.get()) {

if (tailNext != null) {tail.compareAndSet(currentTail, tailNext);

} else {if (currentTail.next.compareAndSet(null, newNode)) {

tail.compareAndSet(currentTail, newNode);return;

}}

}}

}

public E poll() {while (true) {

Node<E> first = head.get();Node<E> last = tail.get();Node<E> next = first.next.get();if (first == head.get()) {

if (first == last) {if (next == null) return null;tail.compareAndSet(last, next);

} else {E item = next.item;if (head.compareAndSet(first, next))

return item;}

}}

}

Non-blocking Queue in JDK

ConcurrentLinkedQueue is based on Michael-Scott queue

� based on CAS-like operations

� use CAS-loop pattern

� threads help one another

Non-blocking algorithms. Summary

Non-blocking Queue.Questions?

ArrayBlockingQueue

ArrayBlockingQueue

0 1 2 3 4 N-1...

void put(E e) throws InterruptedException {checkNotNull(e);final ReentrantLock lock = this.lock;lock.lockInterruptibly();try {

while (count == items.length)notFull.await();

final Object[] items = this.items;items[putIndex] = x;if (++putIndex == items.length)

putIndex = 0;count++;notEmpty.signal();

} finally {lock.unlock();

}}

ArrayBlockingQueue.put()

void put(E e) throws InterruptedException {checkNotNull(e);final ReentrantLock lock = this.lock;lock.lockInterruptibly();try {

while (count == items.length)notFull.await();

final Object[] items = this.items;items[putIndex] = x;if (++putIndex == items.length)

putIndex = 0;count++;notEmpty.signal();

} finally {lock.unlock();

}}

ArrayBlockingQueue.put()

Modifications

Ladan-Mozes, Shavit, 2004, 2008Key IDEA: use Doubly Linked List to avoid 2nd CAS

OptimisticApproach

http://people.csail.mit.edu/edya/publications/OptimisticFIFOQueue-journal.pdf

Hoffman, Shalev, Shavit, 2007

BasketsQueue

http://people.csail.mit.edu/shanir/publications/Baskets%20Queue.pdf

� Throughput is better

� no FIFO any more� usually you don’t need strong FIFO in real life

Baskets Queue

Summary

� Non-blocking algorithms are complicated� Blocking algorithms are easier

� correctness checking is difficult� difficult to support

� Sometimes it has better performance

Summary

� Non-blocking algorithms are complicated� Blocking algorithms are easier

� correctness checking is difficult� difficult to support

� Sometimes it has better performance

Summary

� Non-blocking algorithms are complicated� Blocking algorithms are easier

� correctness checking is difficult� difficult to support

� Sometimes it has better performance

Summary

Engineering is the art of trade-offs

Links & Books

Books

Links

• Nitsan Wakart — http://psy-lob-saw.blogspot.com/• AlexeyShipilev— https://shipilev.net/• concurrency-interestmailinglist:

http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest

Q & A