Clojure Reducers / clj-syd Aug 2012

transcript

ReducersA library and model for collection processing in Clojure

Leonardo Borges@leonardo_borgeshttp://www.leonardoborges.comhttp://www.thoughtworks.com

Thursday, 30 August 12

ReducersA library and model for collection processing in Clojure

...in 20 mins or le

Reducers huh? Here’s the gist

You get parallel versions of reduce, map and filter

Ta-da! I’m done!

and well under my 20 min limit :)

Alright, alright I’m kidding

How do reducers make parallelism possible?

• JVM’s Fork/Join framework• Reduction Transformers

How do reducers make parallelism possible?

Java requirements

• Fork/Join framework• Java 7 [1] or• Java 6 + the JSR166 jar [2]

Clojure requirements

• 1.5.0-* (this is still MASTER on github [3] as of 30/08/2012)

[1] - http://jdk7.java.net/[2] - http://gee.cs.oswego.edu/dl/jsr166/dist/jsr166.jar[3] - https://github.com/clojure/clojure

Before we start - this is bleeding edge stuff

The Fork/Join Framework

•Based on divide and conquer

•Based on divide and conquer•Work stealing algorithm

•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.

•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.•Progressively divides the workload into tasks, up to a threshold

•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.•Progressively divides the workload into tasks, up to a threshold•Once it finished one task, it pops another one form its deque

•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.•Progressively divides the workload into tasks, up to a threshold•Once it finished one task, it pops another one form its deque•After at least two tasks have finished, results can be combined/joined

•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.•Progressively divides the workload into tasks, up to a threshold•Once it finished one task, it pops another one form its deque•After at least two tasks have finished, results can be combined/joined•Idle workers can pop tasks from the deques of workers which fall behind

Text is boring

Fork/Join algorithm - simplified view

Workload is put in “deques”

...and progressively halved

...up to a configured threshold

Worker 1 Worker 2

Combine

Worker 1 Worker 2

Combine

Worker 1 Worker 2

Combine Combine

Worker 1 Worker 2

Combine

Worker 1 Worker 2

Combine

Worker 1

Combine

Worker 2

Worker 1 Worker 2

Idle workers can “steal” items from other workersThursday, 30 August 12

Worker 1 Worker 2

Combine Combine

Worker 1 Worker 2

Combine

Worker 1 Worker 2

Combine

Worker 1 Worker 2

Combine

Worker 1 Worker 2

Final result

Let’s talk about Reducers

Motivations

• Performance• via less allocation• via parallelism (leverage Fork/Join)

Motivations

• Performance• via less allocation• via parallelism (leverage Fork/Join)

Issues

• Lists and Seqs are sequential• map / filter implies order

A closer look at what map does

;; a naive map implementation(defn map [f coll] (if (seq coll) (cons (f (first coll)) (map f (rest coll))) '()))

• Recursion

• Recursion• Order

• Recursion• Order• Laziness (not shown)

• Recursion• Order• Laziness (not shown)• Consumes List

• Recursion• Order• Laziness (not shown)• Consumes List• Builds List

Oh, and it also applies the functionto each item before putting the result into the new list

This is what mapping means!

Reduction Transformers

• Idea is to build map / filter on top of reduce to break from sequentiality

• Idea is to build map / filter on top of reduce to break from sequentiality• map / filter then builds nothing and consumes nothing

• Idea is to build map / filter on top of reduce to break from sequentiality• map / filter then builds nothing and consumes nothing• It changes what reduce means to the collection by transforming the reducing functions

What map is really all about

(defn mapping [f] (fn [f1] (fn [result input] (f1 result (f input)))))

But wait! If map doesn’t consume the list any longer, who does?

• reduce does!• Since Clojure 1.4 reduce lets the collection reduce itself (through the CollReduce / CollFold protocols)• Think of what this means for tree-like structures such as vectors• This is key to leveraging the Fork/Join framework

Now we can use mapping to create reducing functions

(reduce ((mapping inc) +) 0 [1 2 3 4]) ;; 14

(fn [result input] (+ result (inc input)))

(reduce ((mapping inc) conj) [] [1 2 3 4]);; [2 3 4 5]

(fn [result input] (conj result (inc input)))

But it feels awkward to use it in this form

What do we have so far?

• Performance has been improved due to less allocations• No intermediary lists need to be built (see Haskell’s StreamFusion [4])• However reduce is still sequential

[4] - http://bit.ly/streamFusionThursday, 30 August 12

Enters fold

• Takes the sequentiality out or foldl, foldr and reduce

Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)

Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)

Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)• Segments the collection

Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)• Segments the collection• Runs multiple reduces in parallel

Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)• Segments the collection• Runs multiple reduces in parallel• Uses a combining function to join/reduce results

Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)• Segments the collection• Runs multiple reduces in parallel• Uses a combining function to join/reduce results

(defn fold [combinef reducef coll] ...)

The combining function is a monoid

• A binary function with an identity element• All the following functions are equivalent monoids

+(+ 2 3) ; 5(+) ; 0

(defn my-+ ([] 0) ([a b] (+ a b)))

(my-+ 2 3) ; 5(my-+) ; 0

(require ‘[clojure.core.reducers :as r])

(def my-+ (r/monoid + (fn [] 0)))

(my-+ 2 3) ; 5(my-+) ; 0

fold by examples

;; all examples assume the reducers library is available as r(ns reducers-playground.core (:require [clojure.core.reducers :as r]))

fold by examples:increment all even positive integers up to 10 million

and add them all up

and add them all up;; these were taken from Rich’s reducers talk

and add them all up;; these were taken from Rich’s reducers talk(def my-vector (into [] (range 10000000)))

(time (reduce + (map inc (filter even? my-vector))))

(time (reduce + (map inc (filter even? my-vector)))) ;; 500msecs

(time (reduce + (r/map inc (r/filter even? my-vector))))

(time (reduce + (r/map inc (r/filter even? my-vector)))) ;; 260msecs

(time (r/fold + (r/map inc (r/filter even? my-vector))))

(time (r/fold + (r/map inc (r/filter even? my-vector)))) ;; 130msecs

fold by examples:standard word count

(def wiki-dump (slurp "subset-wiki-dump50")) ;50 MB

(defn count-words [text] (reduce (fn [memo word] (assoc memo word (inc (get memo word 0)))) {} (map #(.toLowerCase %) (into [] (re-seq #"\w+" text)))))

fold by examples:standard word count

(defn count-words [text] (reduce (fn [memo word] (assoc memo word (inc (get memo word 0)))) {} (map #(.toLowerCase %) (into [] (re-seq #"\w+" text)))))

(time (count-words wiki-dump)) ;; 45 secs

fold by examples:parallel word count

(defn p-count-words [text] (r/fold (r/monoid (partial merge-with +) hash-map) (fn [memo word] (assoc memo word (inc (get memo word 0)))) (r/map #(.toLowerCase %) (into [] (re-seq #"\w+" text)))))

Combining fn

Will be called at the leaves to merge the partial computations

Will be called with no arguments to provide a seed value

(time (p-count-words wiki-dump)) ;; 30 secs

fold by examples:Load 100k records into PostgreSQL

(def records (into [] (line-seq (BufferedReader. (FileReader. "dump.txt")))))

(time (doseq [record records] (let [tokens (clojure.string/split record #"\t" )] (insert users/users (values { :account-id (nth tokens 0) ... })))))

;; 90 secsThursday, 30 August 12

fold by examples:Load 100k records into PostgreSQL in parallel

(time (r/fold + (r/map (fn [record] (let [tokens (clojure.string/split record #"\t" )] (do (insert users/users (values { :account-id (nth tokens 0) ... })) 1))) records)))

fold by examples:Load 100k records into PostgreSQL in parallel

;; 50 secs

(time (r/fold + (r/map (fn [record] (let [tokens (clojure.string/split record #"\t" )] (do (insert users/users (values { :account-id (nth tokens 0) ... })) 1))) records)))

When to use it

• Exploring decision trees

When to use it

• Exploring decision trees• Image processing

When to use it

• Exploring decision trees• Image processing• As a building block for bigger, distributed systems such as Datomic and Cascalog (maybe around parallel agregators)

When to use it

• Exploring decision trees• Image processing• As a building block for bigger, distributed systems such as Datomic and Cascalog (maybe around parallel agregators)• Basically any list intensive program

When to use it

• Exploring decision trees• Image processing• As a building block for bigger, distributed systems such as Datomic and Cascalog (maybe around parallel agregators)• Basically any list intensive program

But the tools are available to anyone so be creative!

Resources

• The Anatomy of a Reducer - http://bit.ly/anatomyReducers• Rich’s announcement post on Reducers - http://bit.ly/reducersANN• Rich Hickey - Reducers - EuroClojure 2012 - http://bit.ly/reducersVideo (this presentation was heavily inspired by this video)• The Source on github - http://bit.ly/reducersCore

Thanks!

Questions?

Leonardo Borges@leonardo_borges

http://www.leonardoborges.comhttp://www.thoughtworks.com

Clojure Reducers / clj-syd Aug 2012

Technology