Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

transcript

akka streams,

Exploring Reactive Integrations with

Alpakka and Kafka

Konrad `ktoso` Malawski

Konrad `ktoso` MalawskiAkka Team, Reactive Streams TCK, Persistence, HTTP, Remoting / Cluster

Make building powerful concurrent & distributed applications simple.Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM

Actors – simple & high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community

Complete Java & Scala APIs for all features

What’s in the toolkit?

“Stream”has many meanings

akka streamsAsynchronous back pressured stream processing

Source Sink

(possible) asynchronous

boundaries

Source Sink

10 msg/s 1 msg/s

OutOfMemoryError!!

Source Sink

10 msg/s 1 msg/s

hand me 3 morehand me 3 more

1 msg/s Flow

akka streamsNot only linear streams

Source

SinkFlow

SourceSink

FlowFlow

And the many meanings it carries.

Reactive

The many meanings of Reactive

reactivemanifesto.org

The many meanings of Reactive

“Not-quite-Reactive-System” The reason we started researching into transparent to users flow control.

Reactive StreamsReactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols

http://www.reactive-streams.org

Reactive StreamsA buiding-block of Reactive Systems, not the “entire story”.

Reactive StreamsReactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols

http://www.reactive-streams.org

Part of JDK 9java.util.concurrent.Flow

http://openjdk.java.net/projects/jdk9/

JEP-266 – soon…!public final class Flow { private Flow() {} // uninstantiable

@FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); }

public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); }

public static interface Subscription { public void request(long n); public void cancel(); }

public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> { }}

Reactive Streams

RS Library A RS library B

async boundary

Reactive Streams

RS Library A RS library B

async boundary

Make building powerful concurrent & distributed applications simple.

The APIAkka Streams

Complete and awesome Java and Scala APIs (Just like everything in Akka)

Akka Streams in 20 seconds:

Source<Integer, NotUsed> source = null; Flow<Integer, String, NotUsed> flow = Flow.<Integer>create().map((Integer n) -> n.toString()); Sink<String, CompletionStage<Done>> sink = Sink.foreach(str -> System.out.println(str)); RunnableGraph<NotUsed> runnable = source.via(flow).to(sink);runnable.run(materializer);

CompletionStage<String> firstString = Source.single(1) .map(n -> n.toString()) .runWith(Sink.head(), materializer);

Source.single(1).map(i -> i.toString).runWith(Sink.head())

// types: _Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>>

Source.single(1).map(i -> i.toString).runWith(Sink.head())

// types: _Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>>

Materialization

Gears from GeeCON.org,(it’s an awesome conf)

What is “materialization” really?

Introspection, and more coming…

Highly sophisticated stream introspection capabilities.

AlpakkaA community for Streams connectors

http://blog.akka.io/integrations/2016/08/23/intro-alpakka

Alpakka – a community for Stream connectors

Threading & Concurrency in Akka Streams Explained (part I)

Mastering GraphStages (part I, Introduction)

Akka Streams Integration, codename Alpakka

A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II)

Writing Akka Streams Connectors for existing APIs

Flow control at the boundary of Akka Streams and a data provider

Akka Streams Kafka 0.11

http://developer.lightbend.com/docs/alpakka/current/

Akka Streams & HTTP

streams& HTTP

A core feature not obvious to the untrained eye…!

Akka Streams / HTTP

Quiz time! TCP is a ______ protocol?

A core feature not obvious to the untrained eye…!

Akka Streams / HTTP

Quiz time! TCP is a STREAMING protocol!

Streaming in Akka HTTP

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”

http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

HttpServer as a: Flow[HttpRequest, HttpResponse]

HTTP Entity as a: Source[ByteString, _]

Websocket connection as a: Flow[ws.Message, ws.Message]

Resource efficiency and dynamic fan-out

It’s turtles buffers all the way down!

xkcd.com

Streaming from Akka HTTP

Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system);

final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));

final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) );

final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer);

http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080");

Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system);

final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));

final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) );

final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer);

http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080");

Streaming from Akka HTTP (Scala)object Example extends App with SprayJsonSupport with DefaultJsonProtocol { import akka.http.scaladsl.server.Directives._ implicit val system = ActorSystem() implicit val mat = ActorMaterializer()

implicit val jsonRenderingMode = EntityStreamingSupport.json() implicit val TweetFormat = jsonFormat1(Tweet)

def tweetsStreamRoutes = path("tweets") { complete { Source.repeat(Tweet("")) } } Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080) System.out.println("Running at http://localhost:8080");}

Next steps for Akka

Completely new Akka Remoting (goal: 700.000+ msg/s (!)), (it is built using Akka Streams, Aeron).

More integrations for Akka Streams stages, project Alpakka.

Reactive Kafka polishing with SoftwareMill, Krzysiek Ciesielski

Akka Typed progressing again, likely towards 3.0.

Akka HTTP 2.0 Proof of Concept in progress.

Collaboration with Reactive Sockets

Ready to adopt on prod?

Totally, go for it.

Akka <3 contributionsEasy to contribute tickets:

https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22

Akka Stream Contrib https://github.com/akka/akka-stream-contrib

Mailing list: https://groups.google.com/group/akka-user

Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka

More resources: