Zero-Copy Event-Driven Servers with Netty

Daniel Bimschas Institute of Telematics, University of Lübeck

http://www.itm.uni-luebeck.de

1

Content •  Basics

–  Zero-Copy Techniques –  Event-Driven Architectures

•  Introduction to Netty –  Buffers –  Codecs –  Pipelines & Handlers

•  Netty Examples –  Discard –  HTTP Server

2

Zero-Copy Techniques

3

Traditional Approach •  copying and context switches between kernel

and user space à poor performance!

4

App

Kernel Context

Application Context

Application Buffer

Read Buffer Socket Buffer NIC Buffer Fast

Slow

Slide Source: Distributed Systems Course 2011/2012 by Dennis Pfisterer, Institute of Telematics, University of Lübeck, Germany

Zero-Copy Approach •  Kernel handles the copy process via Direct Memory Access (DMA)

–  No CPU load –  Lower load on bus system –  No copying between kernelspace and userspace

5

App

Kernel Context

Application Context

Application Buffer

Read Buffer Socket Buffer NIC Buffer Fast

Perfect! Task

Slide Source: Distributed Systems Course 2011/2012 by Dennis Pfisterer, Institute of Telematics, University of Lübeck, Germany

Simple Benchmark: Copy vs. Zero-Copy D

urat

ion

[ms]

Data [Mbyte] 6 Slide Source: Distributed Systems Course 2011/2012 by Dennis Pfisterer, Institute of Telematics, University of Lübeck, Germany

Zero-Copy Between Communication Layers •  Often copying is not necessary

–  If data is not modi"ed a slice can be passed forward without copying to a different buffer

7

Link Layer

Internet

Transport

Application

Ethernet IP TCP HTTP XML

Ethernet IP TCP HTTP XML

Ethernet IP TCP HTTP XML

Ethernet IP TCP HTTP XML

Ethernet IP TCP HTTP XML

Slide Source: Distributed Systems Course 2011/2012 by Dennis Pfisterer, Institute of Telematics, University of Lübeck, Germany

Zero-Copy Between Communication Layers •  Sometimes slices of multiple packages can be

combined to extract e.g., a payload that is split over multiple packages

•  Newly “created” buffer points to original buffers à No copying necessary

8

TCP HTTP (Part 1) TCP HTTP (Part 2)

HTTP (Part 1) HTTP (Part 2)

Received Buffers

Virtual Buffer

Event-Driven Architecture in Networking

9

Request Processing in Multi-Thread Servers t1: Thread S1: ServerSocket

d1: Decoder

s1: Servlet

db1: DataBase

t2: Thread

waitForData()

bytes = read()

<<create>>(socket)

run()

<<create>>

decode(bytes)

<<create>>

processRequest(req) query(...)

results response write(response)

socket = accept()

Waits most of time without doing

actual work! socket = accept()

waitForData()

bytes = read() req = decode(bytes)

= thread idle

s2: Socket

10

Request Processing in Multi-Thread Servers •  Usually one thread per request

–  Thread idle most of the time (e.g. waiting for I/O) –  Thread even more idle when network slow – Number of simultaneous clients mostly limited by

maximum number of threads

•  Thread construction is expensive – Higher latency when constructing on request – Can be improved using pools of Threads

(see Java‘s ExecutorService & Executors classes)

11

Request Processing in Event-Driven Servers io1: NioWorker s1: Socket s2: Socket

dataAvailable()

bytes = read()

e1: ExecutorThread

handleEvent(s1, bytes)

Disclaimer: this slide may contain errors and is far away from real implementation code but should do good for illustrative purposes

d1: Decoder <<create>>

bytes = read() handleEvent(s2, bytes)

dataAvailable()

d2: Decoder <<create>>

decode(bytes)

decode(bytes) bytes = read() handleEvent(s1, bytes)

dataAvailable()

req = decode(bytes)

resp = processRequest(bytes)

write(resp)

bytes = read() handleEvent(s2, bytes)

dataAvailable()

resp = processRequest(bytes)

req = decode(bytes)

write(resp)

= request 1

= request 2

12

Request Processing in Event-Driven Servers •  Calls to I/O functions of OS are non-blocking •  Heavy usage of zero-copy strategies •  Everything is an event

–  Data available for reading –  Writing data –  Connection established / shut down

•  Different requests share threads •  Work is split into small tasks

–  Tasks are solved by worker threads from a pool –  Thread number and control decoupled from individual

connections / simultaneous requests •  Number of simultaneous clients can be very high

–  Netty: 50.000 on commodity hardware!

13

Introduction to Netty

14

Introduction to Netty

•  „The Netty project is an effort to provide an asynchronous event-driven network application framework for rapid development of maintainable high-performance protocol servers & clients.“

•  Good reasons to use Netty: •  Simpli"es development •  Amazing performance •  Amazing documentation (Tutorials, JavaDocs), clean concepts •  Lots of documenting examples •  Unit testability for protocols •  Heavily used at e.g., twitter for performance critical applications

Source: http://netty.io

15

Netty – Feature Overview

16

Introduction to Netty - Buffers •  Netty uses a zero-copy strategy for efficiency •  Primitive byte[] are wrapped in a ChannelBuffer •  Simple read/write operations, e.g.:

–  writeByte() –  writeLong() –  readByte() –  readLong() –  …

•  Hides complexities such as byte order •  Uses low overhead index pointers for realization:

17

Introduction to Netty - Buffers •  Combine & slice ChannelBuffers without copying

any payload data by e.g., –  ChannelBuffer.slice(int index, int length) –  ChannelBuffers.wrappedBuffer(ChannelBuffer... Buffers)

•  Pseudo-Code Example:

TCP HTTP (Part 1) TCP HTTP (Part 2)

HTTP (Part 1) HTTP (Part 2)

Received Buffers

Virtual Buffer

requestPart1 = buffer1.slice(OFFSET_PAYLOAD, buffer1.readableBytes() – OFFSET_PAYLOAD);

requestPart2 = buffer2.slice(OFFSET_PAYLOAD, buffer2.readableBytes() – OFFSET_PAYLOAD);

request = ChannelBuffers.wrappedBuffer(requestPart1, requestPart2);

18

Netty – Feature Overview

19

Introduction to Netty - Codes •  Many protocol encoders/decoders included

–  Base64 –  Zlib –  Framing/Deframing –  HTTP –  WebSockets –  Google Protocol Buffers –  Real-Time Streaming Protocol (RTSP) –  Java Object Serialization –  String –  (SSL/TLS)

20

Introduction to Netty - Codecs •  Abstract base classes for easy implementation

–  OneToOneEncoder –  OneToOneDecoder

•  Converts one Object (e.g. a ChannelBuffer) into another (e.g. a HttpServletRequest)

–  ReplayingDecoder •  The bytes necessary to decode an Object (e.g. a

HttpServletRequest) may be split over multiple data events •  Manual implementation forces to check and accumulate data

all the time •  ReplayingDecoder allows you to implement decoding

methods just like all required bytes were already received

21

Netty – Putting it all together

22

Introduction to Netty – Pipelines & Handlers

•  Every socket is attached to a ChannelPipeline

•  It contains a stack of handlers –  Protocol

Encoders / Decoders –  Security Layers

(SSL/TLS, Authentication) – Application Logic –  ...

23

Introduction to Netty – Pipelines & Handlers •  One ChannelPipeline per

Connection •  Handlers can handle

–  Downstream events –  Upstream events –  Or both

•  Everything is an event –  Data available for reading –  Writing data –  Connection established /

shut down –  …

24

Netty – ChannelPipeline Example: HTTP(S) Client

Client Application

Channel

write(httpRequest) read(httpResponse)

HttpRequestDecoder HttpRequestEncoder

httpRequest httpResponse

ChannelP

ipeline StringEncoder StringDecoder

SSLDecoder SSLEncoder

String

ChannelBuffer

String

ChannelBuffer

OS Socket object

ChannelBuffer ChannelBuffer

Disclaimer: this slide is imprecise, may contain errors and there’s no one-to-one implementation. It shows a logic conceptual view of the Netty pipeline.

•  Applications based on Netty are built as a stack

•  Application Logic sites on top of the channel

•  Everything else (decoding, securing, ...) is done inside the pipeline

25

Netty Examples

DISCARD Server

26

Netty Examples – DISCARD Server http://tools.ietf.org/html/rfc863

Source: Netty project source code @ http://github.com/netty/netty 27

Netty Examples – DISCARD Server Bootstrap

Source: Netty project source code @ http://github.com/netty/netty 28

Netty Examples – DISCARD Server

Source: Netty project source code @ http://github.com/netty/netty

29

Netty Examples – Echo Server

Source: Netty project source code @ http://github.com/netty/netty 30

Netty Examples – Static HTTP File Server

Source: Netty project source code @ http://github.com/netty/netty 31

Netty Examples – Static HTTP File Server

Source: Netty project source code @ http://github.com/netty/netty 32

Netty Examples – Static HTTP File Server

Source: Netty project source code @ http://github.com/netty/netty ...

33

References •  Netty

–  Project: http://netty.io –  Tutorial: http://netty.io/docs/ –  JavaDoc: http://netty.io/docs/3.2.6.Final/api/ –  Sources: http://netty.io/docs/3.2.6.Final/xref/ – Development: https://github.com/netty/netty

34