Why reinvent the wheel at Criteo?

Cedrick MONTOUT

2016 June 8th

A story about C# @ Criteo

Why do we reinvent the wheel ?

2 | Copyright © 2016 Criteo

• Criteo• Global leader on retargeting

• Scalability• One of many groups in the Criteo R&D department

• WebScale• We write code to help scale up the real-time Criteo Platform

• https://www.linkedin.com/in/kerdrek• https://github.com/kerdrek

• @kerdrek

Who am I ?

https://www.linkedin.com/in/kerdrek



https://github.com/kerdrek

https://github.com/kerdrek

Client Side Load Balancing

PLACEHOLDER IMAGE

Click icon to add picture


The situation then … 1/5

Footer:

App A Pool App B Pool

HA Proxy

Service Pool



Footer:


HA Proxy

Service Pool

Random DC Service PoolTraffic 185k QPS

Input size ~5K bytes

Output size ~3,5K bytes

Network traffic ~12Gbits/s



Footer:


HA Proxy

Service Pool





Compression 30% gain



Footer:


HA Proxy

Service Pool






Bonding 2 ports

Combine two physical data links into one logical link, by connecting 2 ports of the switch to 2 network interfaces of the HAProxy



Footer:


HA Proxy

Service Pool






Bonding 2 ports

Combine two physical data links into one logical link, by connecting 2 ports of the switch to 2 network interfaces of the HAProxy

4 pairs


The new wheel : Client Side Load Balancing

• Bypass HA Proxy• Implemented inside Twitter/Finagle• Reuse existing health Check • Re-implement monitoring

PLACEOLDER IMAGE

App Pool A App Pool B

Service Pool

DevHost

PLACEHOLDER IMAGE


The situation then …

Footer:

• Harder and harder to release• No available memory for new feature• Fragmented feature in production• Tightly coupled with the HTTP stack


The new wheel: Component

• One Input • One Output• On Process

PLACEHOLDER IMAGE


The new wheel: A Host with Services

• A collection of services

PLACEHOLDER IMAGE


The new wheel: DevHost

• The DevHost• Several Components• Several Services• Asynchronous Process• Transport agnostic• Not front facing

PLACEHOLDER IMAGE


The new wheel: DevHost

• Still use in production• ~45% of the windows production machine• Next iteration will use .net core (WiP)

PLACEHOLDER IMAGE

Web (IIS)300020%

De-vHost250017%

Other5003%

CentOS

900060%

Monitoring @ Task level

PLACEHOLDER IMAGE


• Synchronous processing everywhere• Timeout on several pipeline (loosing money)• No clear diagnostic on execution path


Footer:


The new Wheel : Asynchronous Token Framework

• TPL was not a solution at that time• Asynchronous Completion Token• Delegate based• Execution is time boxed• Task underneath• Timing for every thing• Metrics available on the machine• Metrics available aggregated on Graphite

PLACEHOLDER IMAGE

PLACEHOLDER IMAGE

Apache Kafka Driver in C#



Footer:

• Syslog• Text based• Fire and Forget• Single messages• No built-in resiliency• No API for consuming



Footer:

• Apache Kafka• Binary• Acknowledged message• Batched messages• Partitioning and replication• Consuming support

• Syslog• Text based• Fire and Forget• Single messages• No built-in resiliency• No API for consuming


Apache Kafka where is your C# driver ?

We looked at several drivers


Goldilocks Conundrum all over again


First driver never used in production





Second driver was impossible to unit test





Second driver was impossible to unit test

Third driver was not recently maintained


The new Wheel: kafka-sharp

• Yet another C# driver • Highly tuneable• Written with perf and scale in mind• Battle tested in production• Available here: https://github.com/criteo/kafka-sharp

https://github.com/criteo/kafka-sharp

https://github.com/criteo/kafka-sharp


The wheel lists

• Distributed load balancer between clients.• Lightweight hosting server.• Low level asynchronous execution framework. • Yet another C# driver for Apache Kafka.


PLACEHOLDER IMAGE

Click to add text

Date post:	09-Feb-2017
Category:	Engineering
Upload:	criteolabs
View:	171 times
Download:	1 times

Why reinvent the wheel at Criteo?

Engineering