Date post: | 15-Jul-2015 |
Category: |
Documents |
Upload: | konrad-malawski |
View: | 3,743 times |
Download: | 3 times |
Konrad 'ktoso' MalawskiGeeCON 2014 @ Kraków, PL
Konrad `@ktosopl` Malawski
streams
reactive stream processingwith
Konrad `@ktosopl` Malawski
typesafe.comgeecon.org
Java.pl / KrakowScala.plsckrk.com / meetup.com/Paper-Cup @ London
GDGKrakow.pl meetup.com/Lambda-Lounge-Krakow
hAkker @
Agenda
• Reactive Streams • Background and Specification• Protocol details
• Akka Streams • Concepts and goals• Building Blocks• Akka Streams in Action
• Q & A
Streams“You cannot enter the same river twice”
~ Heraclitus
http://en.wikiquote.org/wiki/Heraclitus
StreamsReal Time Stream Processing
When you attach “late” to a Publisher,you may miss initial elements – it’s a river of data.
http://en.wikiquote.org/wiki/Heraclitus
Reactive Streams: Goals
1. Back-pressured Asynchronous Stream processing
2. Standard implemented by many libraries
Reactive Streams - Who?
http://reactive-streams.org
Kaazing Corp.rxJava @ Netflix,
reactor @ Pivotal (SpringSource),vert.x @ Red Hat,
Twitter,akka-streams @ Typesafe,
spray @ Spray.io,Oracle,
java (?) – Doug Lea - SUNY Oswego …
Reactive Streams - Inter-op
http://reactive-streams.org
We want to make different implementations co-operate with each other.
Reactive Streams - Inter-op
http://reactive-streams.org
The different implementations “talk to each other”using the Reactive Streams protocol.
Reactive Streams - Inter-op
http://reactive-streams.org
The Reactive Streams SPI is NOT meant to be user-api.You should use one of the implementing libraries.
package com.rolandkuhn.rsinterop
import ratpack.rx.RxRatpackimport ratpack.test.embed.EmbeddedAppimport ratpack.handling.Handlerimport ratpack.handling.Contextimport rx.Observableimport scala.collection.JavaConverters._import akka.stream.scaladsl.Flowimport akka.stream.scaladsl.Sourceimport rx.RxReactiveStreamsimport akka.stream.scaladsl.Sinkimport akka.actor.ActorSystemimport akka.stream.FlowMaterializerimport ratpack.http.ResponseChunksimport java.util.function.Consumerimport ratpack.test.http.TestHttpClientimport reactor.rx.Streams
object ScalaMain extends App { val system = ActorSystem("InteropTest") implicit val mat = FlowMaterializer()(system) RxRatpack.initialize()
Reactive Streams - Inter-op
EmbeddedApp.fromHandler(new Handler { override def handle(ctx: Context): Unit = { // RxJava Observable val intObs = Observable.from((1 to 10).asJava)
// Reactive Streams Publisher val intPub = RxReactiveStreams.toPublisher(intObs)
// Akka Streams Source val stringSource = Source(intPub).map(_.toString)
// Reactive Streams Publisher val stringPub = stringSource.runWith(Sink.fanoutPublisher(1, 1))
// Reactor Stream val linesStream = Streams.create(stringPub).map[String](new reactor.function.Function[String, String] { override def apply(in: String) = in + "\n" })
// and now render the HTTP response (RatPack) ctx.render(ResponseChunks.stringChunks(linesStream)) }
}).test(new Consumer[TestHttpClient] { override def accept(client: TestHttpClient): Unit = { val text = client.getText() println(text) system.shutdown() } })}
Reactive Streams - Inter-op
Back-pressure? Push + NACK model (a)
Kernel does this!Routers do this!
(TCP)
Use bounded buffer, drop messages + require re-sending
Just push – not safe when Slow Subscriber
Just pull – too slow when Fast Subscriber
Back-pressure? RS: Dynamic Push/Pull
Solution:Dynamic adjustment
Back-pressure? RS: Dynamic Push/Pull
Just push – not safe when Slow Subscriber
Just pull – too slow when Fast Subscriber
Back-pressure? RS: Dynamic Push/PullSlow Subscriber sees it’s buffer can take 3 elements. Publisher will never blow up it’s buffer.
Back-pressure? RS: Dynamic Push/PullFast Publisher will send at-most 3 elements. This is pull-based-backpressure.
Back-pressure? RS: Dynamic Push/Pull
Fast Subscriber can issue more Request(n), before more data arrives!
Back-pressure? RS: Dynamic Push/PullFast Subscriber can issue more Request(n), before more data arrives.
Publisher can accumulate demand.
Back-pressure? RS: Accumulate demandTotal demand of elements is safe to publish. Subscriber’s buffer will not overflow.
Back-pressure? RS: Requesting “a lot”Fast Subscriber can issue arbitrary large requests, including “gimme all you got” (Long.MaxValue)
Reactive Streams SPI
public interface Publisher<T> { public void subscribe(Subscriber<? super T> s);}
Reactive Streams SPI
public interface Publisher<T> { public void subscribe(Subscriber<? super T> s);}
gives a public interface Subscription { public void request(long n); public void cancel();}
A
Reactive Streams SPI
public interface Subscriber<T> { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete();}
public interface Publisher<T> { public void subscribe(Subscriber<? super T> s);}
gives a
to a
public interface Subscription { public void request(long n); public void cancel();}
Akka
Akka has multiple modules:
akka-actor: actors (concurrency abstraction)akka-remote: remote actorsakka-cluster: clusteringakka-persistence: CQRS / Event Sourcingakka-camel: integrationakka-streams: stream processing…
AkkaAkka is a high-performance concurrency library for Scala and Java.
At it’s core it focuses on the Actor Model:
An Actor can only: • Send and receive messages• Create Actors• Change it’s behaviour
AkkaAkka is a high-performance concurrency library for Scala and Java.
At it’s core it focuses on the Actor Model:
class Player extends Actor {
def receive = { case NextTurn => sender() ! decideOnMove() }
def decideOnMove(): Move = ???}
Akka
Akka
Akka has multiple modules:
akka-actor: actors (concurrency abstraction)akka-camel: integrationakka-remote: remote actorsakka-cluster: clusteringakka-persistence: CQRS / Event Sourcingakka-streams: stream processing…
Akka Streams – design decisions
Superior:• reusability• expansibility• performance• bound buffer space• Java and Scala APIs
Akka Streams – Linear Flow
Flow[Double].map(_.toInt). [...]
No Source attached yet.“Pipe ready to work with Doubles”.
Akka Streams – Linear Flow
implicit val sys = ActorSystem("tokyo-sys")
An ActorSystem is the world in which Actors live in.AkkaStreams uses Actors, so it needs ActorSystem.
Akka Streams – Linear Flow
implicit val sys = ActorSystem("tokyo-sys")implicit val mat = FlowMaterializer()
Contains logic on HOW to materialise the stream.
Akka Streams – Linear Flow
implicit val sys = ActorSystem("tokyo-sys")implicit val mat = FlowMaterializer()
A materialiser chooses HOW to materialise a Stream.
The Flow’s AST is fully “lifted”.The Materialiser can choose to materialise the Flow in any way it sees fit.
Our implementation uses Actors.But you could easily plug in an SparkMaterializer!
Akka Streams – Linear Flow
implicit val sys = ActorSystem("tokyo-sys")implicit val mat = FlowMaterializer()
You can configure it’s buffer sizes etc.
Akka Streams – Linear Flow
implicit val sys = ActorSystem("tokyo-sys")implicit val mat = FlowMaterializer()
val foreachSink = Sink.foreach[Int](println)val mf = Source(1 to 3).runWith(foreachSink)
Akka Streams – Linear Flow
implicit val sys = ActorSystem("tokyo-sys")implicit val mat = FlowMaterializer()
val foreachSink = Sink.foreach[Int](println)val mf = Source(1 to 3).runWith(foreachSink)(mat)
Uses the implicit FlowMaterializer
Akka Streams – Linear Flow
implicit val sys = ActorSystem("tokyo-sys")implicit val mat = FlowMaterializer()
// sugar for runWithSource(1 to 3).foreach(println)
Akka Streams – Linear Flow
val mf = Flow[Int]. map(_ * 2). runWith(Sink.foreach(println)) // is missing a Source, // can NOT run == won’t compile!
Akka Streams – Linear Flow
val f = Flow[Int]. map(_ * 2).
runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
Akka Streams – Linear Flow
val f = Flow[Int]. map(_ * 2).
runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
Akka Streams – Linear Flow
val f = Flow[Int]. map(_ * 2).
runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
Akka Streams – Linear Flow
val f = Flow[Int]. map(_ * 2).
runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
f.connect(Source(1 to 10)).run()
Akka Streams – Linear Flow
val f = Flow[Int]. map(_ * 2).
runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
f.connect(Source(1 to 10)).run()
With a Source attached… it can run()
Akka Streams – Linear Flow
Flow[Int]. map(_.toString). runWith(Source(1 to 10), Sink.ignore)
Connects Source and Sink, then runs
Akka Streams – Flows are reusable
f.withSource(IterableSource(1 to 10)).run() f.withSource(IterableSource(1 to 100)).run() f.withSource(IterableSource(1 to 1000)).run()
Akka Streams <-> Actors – Advanced
val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”))
Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber)
Akka Streams <-> Actors – Advanced
Each “group” is a stream too! It’s a “Stream of Streams”.
val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”))
Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber)
Akka Streams <-> Actors – Advanced
groupBy(_.last).
GroupBy groups “11” to group “1”, “12” to group “2” etc.
Akka Streams <-> Actors – Advanced
groupBy(_.last).
It offers (groupKey, subStreamSource) to Subscriber
Source
Akka Streams <-> Actors – Advanced
groupBy(_.last).
It can then start children, to handle the sub-flows!
Source
Akka Streams <-> Actors – Advanced
val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”))
Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber)
The Actor, will consume SubStream offers.
Akka Streams – GraphFlow
// first define some pipeline piecesval f1 = Flow[Input].map(_.toIntermediate)val f2 = Flow[Intermediate].map(_.enrich)val f3 = Flow[Enriched].filter(_.isImportant)val f4 = Flow[Intermediate].mapFuture(_.enrichAsync)
// then add input and output placeholdersval in = SubscriberSource[Input]val out = PublisherSink[Enriched]
Akka Streams – GraphFlow
FlowGraph { implicit b => import FlowGraphImplicits._ val bcast = Broadcast[Intermediate] val merge = Merge[Enriched]
in ~> f1 ~> bcast ~> f2 ~> merge bcast ~> f4 ~> merge ~> f3 ~> out }
Akka Streams – GraphFlow
FlowGraph { implicit b => import FlowGraphImplicits._ val bcast = Broadcast[Intermediate] val merge = Merge[Enriched]
in ~> f1 ~> bcast ~> f2 ~> merge bcast ~> f4 ~> merge ~> f3 ~> out }
There’s more to explore!
Topics we did explore today:
• asynchronous non-blocking back-pressure • complex graph processing pipelines • streams powered TCP server / client • a sneak peek into custom elements
There’s more to explore!
Topics we didn’t explore today:
• explicit buffering, and overflow strategies • integrating with Akka Actors • time-based operators (takeWhile, dropWhile, timer transforms) • plenty additional combinators and junctions• implementing custom processing stages and junctions
There’s more to explore!
Future plans:
• API stabilisation and documentation (1.0 soon)• Improve testability & TestKit• Performance tuning of Streams & HTTP• Provide more Sinks / Sources and operations
• Visualising flow graphs• great experiment by Tim Harper
https://github.com/timcharper/reactive-viz • Distributing computation graphs (?)
Links• http://akka.io • http://reactive-streams.org
• https://groups.google.com/group/akka-user
• Tim Harper’s awesome complex pipeline example + visualisation https://github.com/timcharper/reactive-viz
• 1.0-M2 Documentation (not complete) http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala.html
• Complete JavaDSL for all operations https://github.com/akka/akka/pulls?q=is%3Apr+javadsl
©Typesafe 2015 – All Rights Reserved
Reactive Application
Development!
SBT in Action! Akka in Action!
Play for Scala! Play for Java! Atomic Scala!
Check out eBooks at Typesafe!!
Visit http://typesafe.com/resources/e-books to view all!!
Check out eBooks at Typesafe
Visit http://typesafe.com/resources/e-books to view all