How we scaled push messaging for millions of Netflix devices · Push Registry Zuul Push Servers...

transcript

How we scaled push messaging for millions of Netflix devices

Susheel AroskarCloud Gateway

Why do we need push?

How I spend my time in Netflix application...

● What is push?

● What is push?● How you can build it

● What is push?● How you can build it● How you can operate it

● What is push?● How you can build it● How you can operate it● What can you do with it

Susheel Aroskar

Senior Software EngineerCloud Gateway

saroskar@netflix.com

github.com/raksoras @susheelaroskar

PERSISTUNTILSOMETHINGHAPPENS

Zuul Push Architecture

Zuul Push Servers

WebSockets / SSE

Push Registry

Zuul Push Servers

Register User

WebSockets / SSE

Push Registry

Zuul Push Servers

Register User

WebSockets / SSE

Push Library

Push Registry

Zuul Push Servers

Register User

WebSockets / SSE

Push Library

Push Message Queue

Push Registry

Zuul Push Servers

Register User

WebSockets / SSE

Message Processor

Push Library

Push Message Queue

Push Registry

Zuul Push Servers

Register User

WebSockets / SSE

Message Processor

Push Library

Push Message Queue

Push Registry

Zuul Push Servers

Register User

WebSockets / SSE

Message Processor

Push Library

Push Message Queue

Push Registry

Zuul Push Servers

Register User

Lookup server

WebSockets / SSE

Message Processor

Push Library

Push Message Queue

Push Registry

Zuul Push Servers

Register User

Lookup server

Deliver message

WebSockets / SSE

Handling millions of persistent connections

Zuul Push server

C10K challenge

Socket Socket

Thread per Connection

Thread-1 Thread-2

WriteWrite

Socket Socket

Thread per Connection

Thread-1 Thread-2

WriteWrite

Async I/O

Socket

read callback

write callback

Socket

Single Threadread

callbackwrite

callback

SOCKET

ChannelInboundHandler

ChannelOutboundHandler

Channel Pipeline

Head Tail

protected void addPushHandlers(ChannelPipeline pl) {

pl.addLast(new HttpServerCodec());

pl.addLast(new HttpObjectAggregator());

pl.addLast(getPushAuthHandler());

pl.addLast(new WebSocketServerCompressionHandler());

pl.addLast(new WebSocketServerProtocolHandler());

pl.addLast(getPushRegistrationHandler());

Authenticate by Cookies, JWT or any other custom scheme

Plug in your custom authentication policy

Tracking clients’ connectionMetadata in real-time

Push Registry

public class MyRegistration extends PushRegistrationHandler {

@Override

protected void registerClient(

ChannelHandlerContext ctx,

PushUserAuth auth,

PushConnection conn,

PushConnectionRegistry registry) {

super.registerClient(ctx, authEvent, conn, registry);

ctx.executor().submit(() -> storeInRedis(auth));

Push registry features checklist

● Low read latency

● Low read latency● Record expiry

● Low read latency● Record expiry● Sharding

● Low read latency● Record expiry● Sharding● Replication

What we use

https://github.com/Netflix/dynomite

Redis + Auto-sharding+ Read/Write quorum+ Cross-region replication

Dynomite

Message Processing

Queue, RouteDeliver

We use Kafka message queues to decouple message senders from receivers

Fire and Forget

Cross-region Replication

Different queues for different priorities

We run multiple message processor instances in parallel to scale our message processing throughput.

Operating Zuul Push Different than REST of them

Persistent connections make Zuul Push server stateful

Long lived stable connections

Long lived stable connections○ Great for client efficiency

Long lived stable connections○ Great for client efficiency○ Terrible for quick deploy/rollback

If you love your clients set them free...

Tear down connections periodically

Randomize each connection’s lifetime

Effect of randomizing connection lifetime on reconnect peaks

Ask client to close its connection.

Most connections are idle!

How to optimize push server

BIG Server, tons of connections

ulimit -n 262144

net.ipv4.tcp_rmem="4096 87380

16777216"

net.ipv4.tcp_wmem="4096 87380

16777216"

Goldilocks strategy

Optimize for cost, NOT instance count

How to auto-scale?

RPS? CPU??

How to auto-scale?

RPS? CPU??

Open Connections

Amazon Elastic Load Balancers cannot proxy WebSockets.

Solution - Run ELB as a TCP load balancer

7 Application

6 Presentation

5 Session

4 Transport

3 Network

2 Data link

1 Physical

Ethernet

OSI 7 network layers (conceptual)

HTTP over TCP/IP

Layer 7 HTTP (WebSocket Upgrade Request)

Layer 4 TCP

Managing push cluster - a quick recap

● Recycle connections after tens of minutes

● Recycle connections after tens of minutes● Randomize each connection’s lifetime

● Recycle connections after tens of minutes● Randomize connection’s lifetime● More number of smaller servers >> few BIG servers

● Recycle connections after tens of minutes● Randomize connection’s lifetime● More number of smaller servers >> few BIG servers● Auto-scale on number of open connections per box

● Recycle connections after tens of minutes● Randomize connection’s lifetime● More number of smaller servers >> few BIG servers● Auto-scale on number of open connections per box● WebSocket aware vs TCP load balancer

If you build it,They will push

On-demand diagnostics

Remote recovery

User messaging

WHAT WILL YOU

USE IT FOR?

Call to action

https://github.com/Netflix/zuul

In conclusion, push can make you

In conclusion, push can make you rich (in functionality),

In conclusion, push can make you rich (in functionality), thin (by getting rid of polling)

In conclusion, push can make you rich (in functionality), thin (by getting rid of polling) and happy!

Thank you.

Questions? Susheel Aroskar

Senior Software EngineerCloud Gateway

saroskar@netflix.com

github.com/raksoras@susheelaroskar

Rich, exciting Apps

More efficient systems

Easy to customize

Easy to operate

Zuul Push

Battle tested

How we scaled push messaging for millions of Netflix devices · Push Registry Zuul Push Servers...

Documents