8/8/2019 Fair Share Pres

1/31

The Hadoop Fair Scheduler

Matei Zaharia

Cloudera / Facebook / UC Berkeley

UC Berkeley

8/8/2019 Fair Share Pres

2/31

Outline

Motivation / Hadoop usage at Facebook

Fair scheduler basics

Configuring the fair scheduler Future plans

8/8/2019 Fair Share Pres

3/31

Motivation

Provide short response times to small jobs in a

shared Hadoop cluster

Improve utilization over private clusters / HOD

8/8/2019 Fair Share Pres

4/31

Hadoop Usage at Facebook

Data warehouse running Hive

600 machines, 4800 cores, 2.4 PB disk

3200 jobs per day 50+ engineers have used Hadoop

8/8/2019 Fair Share Pres

5/31

Facebook Data Pipeline

Web Servers Scribe Servers

NetworkStorage

Hadoop Cluster

Oracle RAC

MySQL

Analysts

8/8/2019 Fair Share Pres

6/31

8/8/2019 Fair Share Pres

7/31

Outline

Motivation / Hadoop usage at Facebook

Fair scheduler basics

Configuring the fair scheduler Future plans

8/8/2019 Fair Share Pres

8/31

Fair Scheduler Basics

Group jobs into pools

Assign each pool a guaranteed minimum share

(split up among its jobs) Split excess capacity evenly between jobs

8/8/2019 Fair Share Pres

9/31

Pools

Determined from a configurable job property

Default (before 0.20): mapred.queue.name

At Facebook: user.name (one pool per user) Unmarked jobs go into a default pool

Pools have properties:

Minimum map slots Minimum reduce slots

Limit on # of running jobs

8/8/2019 Fair Share Pres

10/31

Scheduling Algorithm

Divide each pools min share among its jobs

Divide excess capacity among all jobs*

When a slot needs to be assigned: If there is any job below its min share, schedule it

Else schedule the job that weve been most unfair

to (based on deficit)

* Fair schedulers from Hadoop 0.20 on will share equally between pools, not jobs;

patch available at https://issues.apache.org/jira/browse/HADOOP-4789

8/8/2019 Fair Share Pres

11/31

Scheduler Dashboard

8/8/2019 Fair Share Pres

12/31

Scheduler Dashboard

8/8/2019 Fair Share Pres

13/31

Scheduler Dashboard

Reassign pool

8/8/2019 Fair Share Pres

14/31

Scheduler Dashboard

Change priority

8/8/2019 Fair Share Pres

15/31

Scheduler Dashboard

FIFO mode (for testing)

8/8/2019 Fair Share Pres

16/31

Additional Features

Job weights for unequal sharing:

Based on priority (each level is 2x more)

Based on size (mapred.fairscheduler.sizebasedweight

)

Limits on # of running jobs:

Per user Per pool

8/8/2019 Fair Share Pres

17/31

Outline

Motivation / Hadoop usage at Facebook

Fair scheduler basics

Configuring the fair scheduler

Future plans

8/8/2019 Fair Share Pres

18/31

Installing the Scheduler

Compile it:

ant package

Place it on the classpath:

cp build/contrib/fairscheduler/*.jar lib

Alternatively, add the JAR to HADOOP_CLASSPATH

in conf/hadoop-env.sh

8/8/2019 Fair Share Pres

19/31

Configuration Files

Hadoop config (conf/hadoop-site.xml)

Contains scheduler options, pointer to pools file

Pools file (pools.xml)

Contains min share allocations and limits on pools

R

eloaded every 15 seconds to allow reconfiguringpools at runtime

8/8/2019 Fair Share Pres

20/31

Minimal hadoop-site.xml

mapred.jobtracker.taskScheduler

org.apache.hadoop.mapred.FairScheduler

mapred.fairscheduler.allocation.file

/path/to/pools.xml

8/8/2019 Fair Share Pres

21/31

Minimal pools.xml

8/8/2019 Fair Share Pres

22/31

Configuring a Pool

10

5

Any pools not configured in pools.xml will

have minMaps=0 and minReduces=0

8/8/2019 Fair Share Pres

23/31

8/8/2019 Fair Share Pres

24/31

Default Jobs Limit for Users

10

5

3

1

10

8/8/2019 Fair Share Pres

25/31

Other hadoop-site.xml Properties

mapred.fairscheduler.assignmultiple:

Assign a map and reduce on each heartbeat;

improves ramp-up speed and throughput;recommendation: set to true

8/8/2019 Fair Share Pres

26/31

Other hadoop-site.xml Properties

mapred.fairscheduler.poolnameproperty:

Which jobconf property to use to determine

what pool a job is in- Default: mapred.queue.name (queue name)

- Another useful option: user.name

- Can also make up your own, e.g. project

8/8/2019 Fair Share Pres

27/31

Other hadoop-site.xml Properties

mapred.fairscheduler.weightadjuster:

Allows modifying job weights through a plugin

class; one useful example is provided a newjob booster to let short jobs finish faster:

Please see README for details

mapred.fairscheduler.weightadjuster

org.apache.hadoop.mapred.NewJobWeightBooster

8/8/2019 Fair Share Pres

28/31

Outline

Motivation / Hadoop usage at Facebook

Fair scheduler basics

Configuring the fair scheduler Future plans

8/8/2019 Fair Share Pres

29/31

Future Plans

Share equally between pools, not jobs

(Hadoop 0.20 release, HADOOP-4789)

Preemption if a job is starved of its min or fair

share for some timeout (HADOOP-4665)

Locality wait optimization (HADOOP-4667)

8/8/2019 Fair Share Pres

30/31

Future Plans

Simpler scheduling model (HADOOP-4803)

FIFO pools (HADOOP-4803, HADOOP-5186)

Delayed job initialization (HADOOP-5186)

Scalability and operational improvements

8/8/2019 Fair Share Pres

31/31

Thanks!

The Fair Scheduler is available in Hadoop 0.19;

docs in src/contrib/fairscheduler/README

Hadoop 0.17 and 0.18 versions athttp://issues.apache.org/jira/browse/HADOOP-3746

matei@cloudera.com