Mutil-Tenant Data Cloud with YARN & Helix

transcript

Multi-Tenant Data Cloud with YARN & Helix

LinkedIn - Data infra : Helix, Espresso

@kishore_b_gYahoo - Ads infra : S4

Kishore Gopalakrishna

1Thursday, June 5, 14

What is YARNNext Generation Compute Platform

MapReduce

Hadoop 1.0

MapReduce

Hadoop 2.0

Others(Batch, Interactive, Online,

Streaming)

YARN(cluster resource management)

What is YARNNext Generation Compute Platform

MapReduce

Hadoop 1.0

MapReduce

Hadoop 2.0

Others(Batch, Interactive, Online,

Streaming)

Enables

HDFS/Common Area

YARNYARN Architecture

ClientResource Manager

Node Manager Node Manager

submit job

node statusnode statuscontainer request

App Package

Application Master Container

So, let’s build something

Example System

Generate Data

Redis Server 3

HDFS 3

- Generate data in Hadoop - Use it for serving

Example System

Generate Data

Server 3

HDFS 3

Example SystemRequirements

Big Data :-)

Partitioned, replicated

Fault tolerant, Scalable

Efficient resource utilization

Generate Data

Server 3

HDFS 3

ApplicationMaster

Example System

Request Containers Assign work

Handle FailureHandle

workload Changes

RequirementsBig Data :-)

Partitioned, replicated

Fault tolerant, Scalable

Efficient resource utilization

Generate Data

Server 3

HDFS 3

Allocation + Assignment

Server 1 Server 2Server 3

Partition Assignment - affinity, even distribution

Replica Placement - on different physical machines

Container Allocation - data affinity, rack aware placement

M/Rp1 p2 p3 p4 p5 p6

Server 3p3 p4

Server 3p5 p6

Multiple servers to serve the partitioned data

M/R job generates partitioned data

Failure HandlingServer 1 Server 2Server 1

Acquire new container close to data if possible

Assign failed partitions to new container

On Failure - Even load distribution, while waiting for new container

Server 23 Server 3

p5 p4 p1 p6 p3 p2

p1 p2 p3 p4 p5 p6

Server 23 Server 3

p5 p4 p1 p6 p3 p2

p1 p2 p3 p4 p5 p6

Server 23 Server 3 Server 4

p5 p4 p1 p6 p3 p2

p1 p2 p3 p4 p5 p6

Workload ChangesServer 1 Server 2Server 3

Workload change - Acquire/Release containers

Container change - Re-distribute work

Monitor - CPU, Memory, Latency, Tps

Server 3p3 p4

Server 3p5 p6

Server 3p3 p4

Server 3p5 p6

Server 3p4 p6

Server 3p3 p4

Server 3p5 p6

Server 3p4 p6

Service Discovery

Dynamically updated on changes Discover everything, what is running where

Server 3p3 p4

Server 3p5 p6

Service Discovery

Dynamically updated on changes Discover everything, what is running where

Server 3p3 p4

Server 3p5 p6

Client Client

Service Discovery

Building YARN Application

Writing AM is Hard and Error ProneHandling Faults, Workload Changes is non-trivial and often overlooked

Request container

How many containers

Assign work

Place partitions &

replicas

Affinity

Workload changes

acquire/release

containers

Minimize movement

Faults Handling

Detect non trivial failures

new v/s reuse

containers

Service Discovery

Monitoring

Building YARN Application

Writing AM is Hard and Error ProneHandling Faults, Workload Changes is non-trivial and often overlooked

Request container

How many containers

Assign work

Place partitions &

replicas

Affinity

Workload changes

acquire/release

containers

Minimize movement

Faults Handling

Detect non trivial failures

new v/s reuse

containers

Service Discovery

Monitoring

Is there something that can make this easy?

Apache Helix

What is Helix?

Built at LinkedIn, 2+ years in production

Generic cluster management framework

Contributed to Apache, now a TLP: helix.apache.org

Decoupling cluster management from core functionality

Helix at LinkedIn

OracleOracleOracleDB

Change Capture

ChangeConsumers

Index Search Index

User Writes

Data Replicator

In Production

Analytics

Helix at LinkedInIn Production

Over 1000 instances covering over 30000 partitions

Over 1000 instances for change capture consumers

As many as 500 instances in a single Helix cluster

(all numbers are per-datacenter)

Others Using Helix

Helix concepts

Resource (Database, Index, Topic, Task)

Helix concepts

Partitionsp1 p2 p3 p4 p5 p6

Helix concepts

PartitionsReplicas

p1 p2 p3 p4 p5 p6

Helix concepts

PartitionsReplicas

p1 p2 p3 p4 p5 p6

Container Process

Helix concepts

PartitionsReplicas

p1 p2 p3 p4 p5 p6

Container Process

Assignment ?

State Model and ConstraintsHelix Concepts

bootstrap

State Constraints

Transition Constraints

Partition

Resource

Cluster

Serve: 3bootstrap: 0 Max T1 transitions in

parallel

- Max T2 transitions in parallel

No more than 10 replicas

Max T3 transitions in parallel

bootstrap

State Constraints

Transition Constraints

Partition

Resource

Cluster

Serve: 3bootstrap: 0 Max T1 transitions in

parallel

No more than 10 replicas

Max T3 transitions in parallel

StateCount= Replication factor:3

ParticipantParticipantParticipant

Helix Architecture

P1stop

bootstrapserver

Controller

Client Client Target Provider

Provisioner

Rebalancer

assign work via callback

spectator spectator

Service Discovery

metrics

Helix ControllerHigh-Level Overview

Resource Config

Constraints

Objectives

Controller

TargetProvider

Provisioner

Rebalancer

Number of Containers

Task-> Container Mapping

YARN RM

Helix ControllerTarget Provider

Determine how many containers are required along with the spec

Fixed CPU Memory Bin Packing

monitoring system provides usage informationDefault implementations, Bin Packing can be used to customize further

TargetProvider

Resources p1,p2 .. pn

Existing containers c1,c2 .. cn

Health of tasks, containers cpu, memory, health

Allocation constraints

Affinity,rack locality

Fixed: 10 containersCPU headroom:30%Memory Usage: 70%

time: 5h

Number of container

release listacquire list

Container speccpu: x

memory: ylocation: L

Helix ControllerProvisioner

Given the container spec, interact with YARN RM to acquire/release, NM to start/stop containers

Interacts with YARN RM and subscribes to notifications

Helix ControllerRebalancer

Based on the current nodes in the cluster and constraints, find an assignment of task to node

Auto Semi-Auto Static

Rebalancer

Tasks t1,t2 .. tn

Existing containers c1,c2 .. cn

Allocation constraints &

objectives

Affinity,rack locality,

Even distribution of tasks,

Minimize movement while expanding

Assignment C1: t1,t2C2: t3,t4

User defined

Based on the FSM, compute & fire the transitions to Participants

Example System: Helix-Based Solution

Solution

Configure App

Configure Target Provider

Configure Provisioner

Configure RebalancerGenerate Data

Server 3

HDFS 3

Configure AppConfigure App

App Name Partitioned Data Server

App Master Package

/path/to/GenericHelixAppMaster.tar

App package /path/to/RedisServerLauncher.tar

App Config DataDirectory: hdfs:/path/to/data

Configure target providerConfigure target provider

TargetProvider RedisTargetProvider

Goal Target TPS: 1 million

Min container 1

Max containers 25

Configure ProvisionerConfigure Provisioner

YARN RM host:port

Configure RebalancerConfigure RebalancerPartitions 6Replica 2

Max partitions per container 4

Rebalancer.Mode AUTO

Placement Data Affinity

FailureHandling Even distribution

Scaling Minimize Movement

app_config_spec.yaml

Example System: Helix-Based Solution

yarn_app_launcher.sh app_config_spec.yaml

Launch Application

Helix + YARN

Server 1 Server 2

Helix + YARN

YARN Resource Manager

Client

submit job

Server 1 Server 2

Application Master

Helix + YARN

Client

submit job

Launch AM

Server 1 Server 2

Application Master

Helix + YARN

Helix Controller

Target Provider

Provisioner

RebalancerClient

submit job

Launch AM

Server 1 Server 2

Application Master

Helix + YARN

Helix Controller

Target Provider

Provisioner

RebalancerClient

submit job

Launch AM

request cntrs

Server 1 Server 2

Node ManagerNode Manager

Application Master

Helix + YARN

Helix Controller

Node Manager

Target Provider

Provisioner

RebalancerClient

submit job

Launch AM

request cntrs

launch containers

Server 1 Server 2participant 3 participant 3 participant 3

Node ManagerNode Manager

Application Master

Helix + YARN

Helix Controller

Node Manager

Target Provider

Provisioner

Rebalancer

assign work

Client

submit job

Launch AM

request cntrs

launch containers

Server 1 Server 2participant 3p1 p2

participant 3p3 p4

participant 3p5 p6

Auto Scaling

Non linear scaling from 0 to 1M TPS and back

Failure Handling: Random Faults

Recovering from faults at 1M Tps (5%, 10%, 20% failures/min)

Summary

HELIX(container + task management)

Others(Batch, Interactive, Online, Streaming)

Fault tolerance, Expansion handled transparently

Generic Application Master

Efficient resource utilization by task model

Questions?

Website

Twitter

helix.apache.org, #apachehelix

@apachehelix, @kishore_b_g

user@helix.apache.org

Kanak Biscuitwala, Zhen Zhang?We love helping & being helped

Mutil-Tenant Data Cloud with YARN & Helix

Engineering