Cloud Native Streaming and Event-Driven Microservices

transcript

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Cloud Native Event-Driven and Streaming MicroservicesBy Marius Bogoevici, Staff Engineer, Pivotal

@mariusbogoevici

Safe Harbor Statement

• The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation.

CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES

• Processing large quantities of data in real-time • Use cases:

• predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’

• Specific requirements • High throughput, low latency • Grouping, ordering, windowing

• Transition in messaging middleware design: • Traditional (JMS-based, RabbitMQ) • To log-style: Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs

• Resource management • memory, CPU, instance count

• Scaling up/down • Health monitoring and failover • Routing and load balancing

• Applications and deployment strategies designed to target of all above

ApacheYARNApacheMesos Kubernetes

• Typically known benefits and drawbacks of microservices:

• bounded context

• development agility

• added complexity

• Failure isolation

• Process-specific resource tuning:

• memory

• CPU

• instance count

• network

• Emphasize messaging as communication over HTTP

• Decoupling

• Discoverability

• Availability

• Eventual consistency across heterogenous resources

• alternative to distributed transactions

• Strong similarities between:

• data streaming - ‘big data’: ingestion, analytics

• event streaming - async interaction, event sourcing, etc

Spring Cloud Stream: The Origin Story

Spring Cloud Stream 2015

Spring XD Spring Cloud Data Flow

Spring Cloud Task

Spring Cloud Stream

• Event-driven microservice framework

• Built on battle-tested components

• Spring Boot: full-stack standalone applications

• Spring Integration: Messaging, EIP patterns, connectors

• Opinionated primitives

• Pluggable middleware abstractions

Spring Cloud Stream in a nutshell

Application Core

Spring Boot Spring Integration Spring Messaging Reactive APIs

inputs

outputs

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 / 10

The Binder Abstraction

Application

Messaging Middleware Binder channel

can be input or output

adapts channel to target middleware

created by framework

configured via Spring Boot properties

Programming model

@EnableBinding + Binder Implementation

Apache Kafka

JMS Google PubSub

Production-ready:

Experimental

Programming model

@SpringBootApplication@EnableBinding(Processor.class) public class UppercaseProcessor { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT) public String process(String s) { return s.toUpperCase(); }}

Pipes and filters model

http averages topN

ingest data

average value over last 5s

hottest sensors over last 10 seconds

Spring Cloud Stream: Durable Publish Subscribe

raw-sensor-data

averages

top-n Calculator

Spring Cloud Stream: Durable Publish Subscribe

raw-sensor-data

averages

top-n Calculator

Failure detector

in Spring Cloud Stream, all destinations are pub-sub

Data and event streaming - conceptually similar

Spring Cloud Stream: Consumer groups

raw-sensor-data

averages

HDFShttp

pubsub is great for interconnecting logical apps

what if we add more instances ?

raw-sensor-data

averages

HDFShttp

we want these two to compete

averages

HDFSand these two !

Averages

HDDSHDFSHDFS

raw-sensor-datagroups are in a pub-sub relationship to each other

AveragesAverages

consumers are competing within a group

Spring Cloud Stream: Partitioning

Average calculator

raw-sensor-data

Average calculator

partition 1

partition 2

raw-sensor-data

Average calculator

partition 1

partition 2

raw-sensor-data

Simple topologies : (relatively) easy to deploy …

spring.cloud.stream.bindings.output.destination=httphdfs.1

spring.cloud.stream.bindings.input.destination=httphdfs.1 spring.cloud.stream.bindings.input.group=httphdfs

httphdfs.1

… but how about complex topologies ?

raw-sensor-data

averages

top-n Calculator

Failure detector

averagesaverages

HDFSHDFSHDFS

Spring Cloud Data Flow

• Orchestration:

• DSL for Stream topologies

• REST API

• Shell

• UI

• Portable Deployment SPI

• OOTB apps for common integration use-cases

In a single sentence …

dataflow:> stream create demo --definition “http | file”

Spring Cloud Data Flow - Stream DSL

Stream definition

Boot Apps built on top of Spring Ecosystem

httpfile = http | file

Spring Cloud Data Flow - deployment SPI

• SPI for deploying applications

• Local (for testing)

• Cloud Foundry

• YARN

• Kubernetes

• Mesos + Marathon

• Different resources supported

• Spring Boot Uberjars

• Docker Images

• Different locations

• File system, HDFS, HTTP, Maven, Docker Hub

Spring Cloud Data Flow deployment

Pla$orm

Spring Cloud Stream 1.1

• Under development, currently on master

• 1.1.0.M1 release August 2016

• 1.1.0.RELEASE - early Q4 2016

• Based on Spring Boot 1.4

• Release train model

• binders in separate repositories

• New features

• Kafka 0.9 support (also intending 0.10)

• Reactive programming support

• Schema Registry support

Programming model - imperative/procedural

@SpringBootApplication@EnableBinding(Processor.class) public class UppercaseProcessor { @StreamListener(“input”) @SendTo(“output”) public String process(String s) { return s.toUpperCase(); }}

Reactive/functional Programming via Reactor

@SpringBootApplication@EnableBinding(Processor.class) public class UppercaseProcessor { @StreamListener @Output(“output”) public Flux<String> process(@Input(“input”) Flux<String> s) { return f.map(s -> s.toUppercase()); }}

Handling multiple returns (or none at all)

@SpringBootApplication@EnableBinding(Processor.class) public class UppercaseProcessor { @StreamListener @Output(“output”) public Flux<String> process(@Input(“input”) Flux<String> s) { return f.filter(s->startsWith(“a”)).map(s -> s.toUppercase()); }}

Stateful operators: grouping, windowing

SpringBootApplication@EnableBinding(Processor.class) public class WordCountApplication { @StreamListener @Output("output") public Flux<?> countWords(@Input("input") Flux<String> words) { return words.window(ofSeconds(5), ofSeconds(1)) .flatMap(window -> window.groupBy(word -> word) .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1) .map(count -> new WordCount(group.key(), count)))); } }

Content-type transformations (since 1.0)

• Users can specify the intended content type of an output value, e.g. “application/json”

• Outgoing messages are converted to that content type

• A message header value carries the content type

• The content type can be used by inputs to convert to expected types (e.g. `@StreamListener` method arguments

• Extensible model, just requires the registration of AbstractMessageConverter beans

Schema management and Avro support

• Newly introduced Avro MessageConverters

• Static schema-based resources (classpath, filesystem, etc)

• Schema Registry Client-based

• Schema registry server

• Not Spring Cloud Stream specific

• Allows storing different formats and versions

• Schema evolution support via Schema Registry Client converter

Heterogenous data sources/consumers

sensor v1

sensor v2

average-temperature v1

sensor data

average-pressure v2

Schema evolution in Spring Cloud Stream 1.1

Schema Registry Server - embeddable in apps

@SpringBootApplication @EnableSchemaRegistryServer public class SchemaRegistryServerApplication { public static void main(String[] args) { SpringApplication .run(SchemaRegistryServerApplication.class,args); } }

Pluggable schema registry client

@EnableBinding(Sink.class) @EnableAutoConfiguration@EnableSchemaRegistryClientpublic static class VoteCounterApplication { @StreamListener(Sink.INPUT) public void listen(Vote vote) { voteService.count(vote); } }

Future plans

• Additional binders

• JMS

• Google Pub Sub

• Kinesis

• Fully reactive pipelines

• KStream support

• First class support for stateful processing

Fully reactive pipelines (planned)

• Use natively reactive middleware clients for reactive support

• e.g. Reactive Kafka client

public interface ReactiveProcessor {

@Input(Processor.INPUT) Flux<?> output(); @Output(Processor.OUTPUT) Flux<?> output(); }

@EnableBinding(ReactiveProcessor.class)

Kafka Stream Support (planned)

@StreamListener@Output(“output”)public KStream<Integer,String> addByKey(@Input(“input”) KStream<Integer,String> input) { return input.map((i,s) -> new KeyValue<>(i, s.toUpperCase())) .reduceByKey((String value1, String value2) -> value1 + value2, TimeWindows.of("windowName", 1000)) .toStream() }

Related talks

Reactive Kafka

Rajini Sivaram

Thursday, August 4, 9:00 AM - 10:05 AM

Orchestrate All the Things! with Spring Cloud Data Flow

Eric Bottard, Ilayaperumal Gopinathan

Thursday, August 4 11:10 AM - 12:15 PM

Learn More. Stay Connected.

Tell your friends!

@springcentral spring.io/blog

@pivotal pivotal.io/blog

@pivotalcf http://engineering.pivotal.io

Cloud Native Streaming and Event-Driven Microservices

Technology