Alexander Kolb - Flinkspector – Taming the squirrel

FlinkspectorTaming the Squirrel

Alexander Kolb otto group BI

@lofifnc

Alexander Kolb Otto Group BI

@lofifnc

Alexander Kolb - 2016 - otto group

Agenda

• Introduction

• Testing Apache Flink Applications

• Flinkspector • Concept • Input • Expectations • Execution

• Conclusion

• Outlook


Introduction


• Multichannel Retail

• Financial Services

• Services

• 30 countries

• 123 companies


Problem

• Transparency

• Traceability

• Reproducibility


Solutions

• Visualisation

• Datalineage graph / Message tracing

• Unit Testing


Testing Apache Flink Applications


Apache Flink Applicationfinal StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream<String> text = env.socketTextStream("localhost", 9999); DataStream<Tuple2<String, Integer>> words = text.flatMap(new WordCount.LineSplitter()); DataStream<Tuple2<String, Integer>> aWords = words.filter(new StartsWithAFilter()); DataStream<Tuple2<String, Integer>> counts = aWords.keyBy(0).timeWindow(Time.of(2, TimeUnit.MINUTES)).sum(1); counts.print(); env.execute("Wordcount Example");


User Defined Functions

public final class StartsWithAFilter implements FilterFunction<Tuple2<String,Integer>> { @Override public boolean filter(Tuple2<String, Integer> t) throws Exception { return t.f0.startsWith("a"); } }


User Defined Functions

@Testpublic void testAFilter() throws Exception { StartsWithAFilter filter = new StartsWithAFilter(); Assert.assertEquals(true, filter.filter(Tuple2.of("aTest",1))); Assert.assertEquals(false, filter.filter(Tuple2.of("bTest",1))); }


Stream Transformations

public static DataStream<Tuple2<String,Integer>>countAWords(DataStream<Tuple2<String,Integer>> aWords) { return aWords .keyBy(0) .timeWindow(Time.of(2, TimeUnit.MINUTES)) .sum(1); }


Stream Transformations@Testpublic void aWordCountTest() { StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(); env.setParallelism(1); DataStream<Tuple2<String, Integer>> words = env.fromCollection(Arrays.asList( Tuple2.of("a1", 1), Tuple2.of("a2", 1), Tuple2.of("a1", 1) )); DataStream<Tuple2<String, Integer>> results = SocketTextStreamWordCount.countAWords(words); Iterator<Tuple2<String, Integer>> output = DataStreamUtils.collect(results); Assert.assertEquals(output.next(), Tuple2.of("a1", 1)); Assert.assertEquals(output.next(), Tuple2.of("a2", 2)); Assert.assertEquals(output.hasNext(), false); }


Flinkspector


Concept

1. Specify input.

2. Aquire data stream from the input specification.

3. Define expectations for the resulting data stream.

4. Apply the expectations to the produced data stream.


TestStreamEnvironment

Runtime

Subscriber

Test Source Input

Filter

KeyedWindowReducer

Concept

Verifier Test Sink Publisher

Trigger(default)

paired


Expectations


Input

EventTimeInput<Tuple2<String,Integer>> input = EventTimeInputBuilder.startWith(Tuple2.of("a1", 1)) .emit(Tuple2.of("a2", 1), after(1, minutes)) .emit(Tuple2.of("a3", 1), before(1, seconds), times(2)) .emit(Tuple3.of(“a4”, 1), intoWindow(4, seconds)) .repeatAll(after(1, minutes), times(3));


List Based

ExpectedRecords.create(asList(1,2,3,4));

ExpectedRecords.create(asList(1,2,3,4)) .refine().only();

ExpectedRecords.create(asList(1,2,3,4)) .refine().sameFrequency();

(4,1,2,3,3,5)

(4,1,2,3,3,5)

(4,1,2,3,3,5)


List Based

(4,1,2,3,3,5)

(4,1,2,3,3,5)

(4,1,2,3,3,5)





List Based

(4,1,2,3,3,5)

(4,1,2,3,3,5)

(4,1,2,3,3,5)





List Based

(4,1,2,3,3,5)

(4,1,2,3,3)

(4,1,2,3,3,5)





List Based

(4,1,2,3,3,5)

(4,1,2,3,3)

(4,1,2,3,3,5)





List Based

(4,1,2,3,3,5)

(4,1,2,3,3)

(4,1,2,3,5)





List Based

ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(notStrict).all();

ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(strict).from(1);

List(4,1,2,3,3,5)

List(5,2,3,4,1)


List Based

List(4,1,2,3,3,5)

List(5,2,3,4,1)




List Based

List(1,2,3,3,5,4)

List(5,2,3,4,1)




List Based

List(1,2,3,3,5,4)

List(5,2,3,4,1)




List Based

List(1,2,3,4,1)

MatcherBuilder

only

sameFrequency

order

ExpectedRecords.create(asList(1,2,3,4)) .refine() .only() .sameFrequency() .inOrder(strict).from(1);


List Based


List(1,2,3,4,1)

MatcherBuilder

only

sameFrequency

order


List Based

List(1,2,3,4,1)

MatcherBuilder

only

sameFrequency

order



List Based

List(1,2,3,4,1)

MatcherBuilder

only

sameFrequency

order



List Based

List(2,3,4,1)

MatcherBuilder

only

sameFrequency

order



List Based

List(2,3,4,1)

MatcherBuilder

only

sameFrequency

order



Assertion Based

new MatchTuples<Tuple2<String,Integer>>("word","count") .assertThat("word", startsWith("a")) .assertThat("count", greaterThan(3)) .onEachRecord();


Assertion Based

new MatchTuples<Tuple2<String,Integer>>("word","count") .assertThat("word", startsWith("a")) .assertThat("count", greaterThan(3)) .oneOfThem() .onEachRecord();


Assertion Based

.assertThat(A,…)

.assertThat(B,…)

.assertThat(C,…)

.eachOfThem()

.onAnyRecord()

A B C


Assertion Based

A B C

.assertThat(A,…)

.assertThat(B,…)

.assertThat(C,…)

.eachOfThem()

.onAnyRecord()


Assertion Based

.assertThat(A,…)

.assertThat(B,…)

.eachOfThem()

.onAnyRecord()

A B C


Assertion Based

.assertThat(A,…)

.assertThat(B,…)

.assertThat(C,…)

.eachOfThem()

.onAnyRecord()

A B C


Assertion Based

.assertThat(A,…)

.assertThat(A,…)

.assertThat(A,…)

.atLeastNOfThem(2)

.onEach()

A B C


Assertion Based

.assertThat(A,…)

.assertThat(B,…)

.assertThat(C,…)

.atExactlyNOfThem(2)

.onEach()

A B C


Assertion Basedpublic class Each<T> extends UntilCombineMatcher<T> { public Each(Iterable<Matcher<? super T>> matchers) { super(matchers); } @Override public String prefix() { return "each of"; } @Override public boolean validWhen(int matches, int possibleMatches) { return matches == possibleMatches; } @Factory public static <T> Each<T> each(Iterable<Matcher<? super T>> matchers) { return new Each<T>(matchers); } }



Runtime

Subscriber

Test Source Input

Filter

KeyedWindowReducer

Execution

Verifier Test Sink Publisher

Trigger(default)

paired


Execution

Test SourceInput

Filter

KeyedWindowReducer

Test Sink Publisher

Test Source Input

Filter

KeyedWindowReducer

Test Sink Publisher

1

1

1

1

2

2

2

2

Loca

l Clu

ster


Runtime

Subscriber

Verifier Trigger(default)


Execution

Loca

l Clu

ster

TS I1

F

KWR

TSi P1

1

1

TS I22

F2

KWR1

1

2

TSi P2

2


Runtime

Subscriber


Timeout


ExecutionTestStreamEnvironment

Runtime

Subscriber


Loca

l Clu

ster

TS I1

F

KWR

TSi P1

1

1

TS I22

F2

KWR1

1

2

TSi P2

2

Timeout



Runtime

Subscriber


Loca

l Clu

ster

TS I1

F

KWR

TSi P1

1

1

TS I22

F2

KWR1

1

2

TSi P2

2

Timeout


Loca

l Clu

ster

TSI1

F

KWR

TSiP1

1

1

TSI22

F2

KWR1

1

2

TSiP2

2


Runtime

Subscriber


Timeout

CLOSE 1

CLOSE 2


TSI1

F

KWR

TSiP1

1

1

TSI22

F2

KWR1

1

2

TSiP2

2


Runtime

Subscriber


Loca

l Clu

ster



Runtime

Subscriber

Verifier Trigger

Loca

l Clu

ster

TS I1

F

KWR

TSi P1

1

1

TS I22

F2

KWR1

1

2

TSi P2

2

TimeoutTimeout



Runtime

Subscriber

Verifier Trigger

Loca

l Clu

ster

TS I1

F

KWR

TSi P1

1

1

TS I22

F2

KWR1

1

2

TSi P2

2



Runtime

Subscriber


Loca

l Clu

ster

TS I1

F

KWR

TSi P1

1

1

TS I22

F2

KWR1

1

2

TSi P2

2

TimeoutTimeout



Runtime

Subscriber


Loca

l Clu

ster

TS I1

F

KWR

TSi P1

1

1

TS I22

F2

KWR1

1

2

TSi P2

2

TimeoutTimeout


Conclusion


Conclusion

• Small and concise test cases.

• Extensible dedicated runtime.

• Integration into existing test platforms.

• ProcessingTimeWindows.

• Input specification too expansive.


Outlook


Outlook

• Scala (Test) support.

• Symbolic testing integration.

• System tests on the actual cluster.


https://github.com/ottogroup/flink-spector

https://github.com/ottogroup/flink-spector


ottogroup.comWE ARE HIRING!

Date post:	21-Jan-2018
Category:	Data & Analytics
Upload:	flink-forward
View:	230 times
Download:	0 times

Alexander Kolb - Flinkspector – Taming the squirrel

Data & Analytics