Post on 15-Dec-2015
transcript
Towards Provision of Quality of Service
Guarantees in Job Scheduling
Mohammad Islam Pavan Balaji
P. Sadayappan D. K. Panda
Computer Science and Engineering
The Ohio State University
Publicly Usable Supercomputer Centers and Compute Clusters
– Becoming increasingly common (OSC, SDSC, etc)
– Allow people to run jobs and charge the user based on the job requirements
– Job requirements include Number of CPUs, CPU time, memory, etc.
– Jobs are submitted by the user with resource requirements
Scheduled to execute at a later time
Dedicated job scheduler schedules the submitted jobs (Ex: PBS, Maui, Silver)
Jobs provide an estimate of the run-time together with the job
Independent Parallel Job Scheduling Model
– Dynamically arriving Independent Parallel Jobs
– Resource mapping: Submitted Jobs to Resources
– Viewed as a two dimensional chart: Processors vs. Time
Job Schedulers Today
Two Dimensional Scheduling Grid
J1
J2
Time
Pro
cess
ors J3
Current Time
J5 J6J4
Job Queue
Running Jobs
Job Scheduling Overview
Submit jobs have several associated parameters
– Resources Required
– Arrival Time (time with the job is submitted by the user)
– Run-time estimate of the job (users need to provide a run-time estimate)
Queuing Order
– FCFS (First Come First Served)
– SJF (Shortest Job First)
Reservations are made for some of the jobs
– Conservative backfilling model
– EASY backfilling model
Backfilling Models
J1
J2
Time
Pro
cess
ors J3
Current Time
J5 J6J4
Job Queue
Running JobsJ4
J5Reservation
Violation
J5
J6
Conservative gives reservations to all jobs; EASY gives reservations to just one job
Guarantees in Service
Number of Techniques studied over the years
– Backfilling (Ex: Conservative, EASY, No Guarantee)
– Priority based scheduling
Differentiated service to different classes of jobs
Soft Real-time or Best Effort guarantees to the completion time
Hard Real-Time or “Deadline-based” scheduling
– Allow Users to specify the deadline they desire
– Cost model based on Resources Used AND Deadline Specified
– Requires a deadline-based scheduling algorithm
QoS for Job Scheduling
Two Components in providing QoS
– Job Scheduling Component
Admission Control
– Can we meet the specified deadline?
Once admitted, cannot miss the specified deadline
Studied as a part of our previous work [islam03:qops]
– Cost Model Component
Based on Resources Used AND Deadline Specified
More urgent jobs are charged more
Role of user characteristics in the charging model
– Tolerance of users to missed deadlines
We deal with this component of QoS in this paper
[islam03:qops] “QoPS: A QoS based scheme for Parallel Job Scheduling”, M. Islam, P. Balaji, P. Sadayappan and D. K. Panda. Published in JSSPP ’03 and LNCS ‘04.
Overview
Introduction and Motivation
Cost Model for Supercomputer Centers
Understanding User-Tolerance
Dealing with User-Tolerance
Artificial Slack
Kill-and-Restart
Experimental Results
Conclusions and Future Work
Cost Model in Supercomputer Centers
Current Cost Model based on Resources used
– Cost independent of the response time of the job
– User cannot request for a quicker service
Some schedulers allow differentiated service (Ex: NERSC)
– Use high priority, normal and low priority queues
– High Priority queue charges more; Low Priority queue charges less
– No guarantees provided to the user about the response time
– Idea of charging the user based on the service/priority: Still Relevant !
Two components in the Cost Model
– Resource Charge
– QoS Charge
Cost Model Components
Resource Charge
– Depends on the CPU, Memory, Disk Space, etc.
– We only consider the CPU resource; extendable to other resources
QoS Charge
– Depends on the urgency of the job
– Depends on the difficulty in scheduling the job; based on two components
Current load in the system
Slowdown of the category to which the job belongs (Ex: Short-Wide)
– Slowdown depicts the factor by which the job is delayed compared to its runtime
Resource Charge = Processors x Run-Time
QoS Charge = Category Slowdown / Requested Slowdown
Overview
Introduction and Motivation
Cost Model for Supercomputer Centers
Understanding User-Tolerance
Dealing with User-Tolerance
Artificial Slack
Kill-and-Restart
Experimental Results
Conclusions and Future Work
Understanding User Tolerance
If the requested deadline cannot be met?
– Some users might submit another job or to another supercomputer center
– Some users might re-submit the job with another deadline
What new deadline? Hit-and-Trial?
– We modified QoPS to provide some feedback about the best possible deadline
We model user-tolerance based on a Tolerance Factor (TF)
– Factor of extension in the requested deadline, the user can accept
– If the (Earliest Possible Deadline) < (TF x Requested Deadline)
The user is assumed to accept the new deadline
– Else
The user is assumed to reject the new deadline
Feedback based QoPS Algorithm
This is for the case when the user requested deadline cannot be met
– Say this deadline is D1
When a new job arrives:
– Try to place the new job without disturbing any other job
– Calculate the completion time of the job
– This is a definitely feasible deadline (D2)
Use a binary search for N iterations to find a better deadline
– For each iteration, run QoPS with the new deadline
– If feasible, try a tighter deadline; else try a looser deadline
Impact of User ToleranceResource Charge
5
5.5
6
6.5
7
7.5
8
1 1.2 1.5 2 2.5 3 3.5 4
Tolerance Factor (TF)
Re
sou
rce
Ch
arg
e (
x 1
0^8
)
load 1.3 load 1.6
QoS Charge
20000
40000
60000
80000
100000
120000
140000
160000
1 1.2 1.5 2 2.5 3 3.5 4
Tolerance Factor (TF)Q
oS
Ch
arg
e
load 1.3 load 1.6
• Increase in Resource Charge: Intuitive
• Decrease in QoS Charge; as the users get more tolerant, the center loses QoS money !• This is because of accepting cheaper jobs which would have otherwise been rejected
• Overall profit of the Supercomputer Center depends on the ratio of the two charges
Overview
Introduction and Motivation
Cost Model for Supercomputer Centers
Understanding User-Tolerance
Dealing with User-Tolerance
Artificial Slack
Kill-and-Restart
Experimental Results
Conclusions and Future Work
Dealing with User-Tolerance
User-Tolerance is not an always win situation!
– Depends on the QoS charge to Resource charge ratio
If QoS charge were 0, user-tolerance is always beneficial
If Resource charge were 0, user-tolerance is never beneficial
– We need some way to deal with this behavior of user-tolerance
Allows us to maximize profit depending on the QoS and Resource charges
We propose two approaches to counter user-tolerance
– Providing Artificial Slack
– Kill and Restart Mechanism
Artificial Slack
If a requested deadline cannot be met
– Provide the best possible deadline to the job
Maximizes the revenue for the job
Creates a very tight schedule; difficulty in accepting later arriving jobs
– Provide the job with an artificial slack
Loses out on the revenue for the job
The overall schedule is loose enough to accept later arriving jobs
– Offered Deadline extended
Arrival Time + (Earliest Deadline – Arrival Time) x Slack Factor
Kill and Restart
Some supercomputer centers support a kill-and-restart mechanism
A running job can be killed and restarted: No permanent modifications
Some centers such as the OSC support such mechanisms
Our approach utilizing the kill-and-restart mechanism
– If a jobs requested deadline cannot be met
Kill a running job and try scheduling this job
Verify that both the new job and the killed job can be scheduled
If they can be scheduled, accept the schedule
– It is to be noted that this approach might result in wastage of resources
Overview
Introduction and Motivation
Cost Model for Supercomputer Centers
Understanding User-Tolerance
Dealing with User-Tolerance
Artificial Slack
Kill-and-Restart
Experimental Results
Conclusions and Future Work
Impact of Slack Factor (SF)Resource Charge (Load = 1.3)
5
5.2
5.4
5.6
5.8
6
6.2
1 1.5 2 2.5 3 4
Slack Facfor (SF)
Re
sou
rce
Ch
arg
e (
x 1
0^8
)
TF = 1.0 TF = 2.0
TF = 3.0 TF = 4.0
QoS Charge (Load = 1.3)
37000
38000
39000
40000
41000
42000
43000
44000
45000
1 1.5 2 2.5 3 4
Slack Factor (SF)Q
oS
Ch
arg
e
TF = 1 TF = 2 TF = 3 TF = 4
• Counter Impact as User Tolerance !
• Drop of Resource charge; more smaller jobs accepted, force larger jobs to be rejected
• Increase in QoS charge; additional slack forces significantly delayed jobs to be rejected
Impact of Kill-and-Restart (TF = 4.0)Resource Charge
5
5.5
6
6.5
7
7.5
8
1 1.5 2 2.5 3 4
Slack Factor (SF)
Re
sou
rce
Ch
arg
e (
x 1
0^8
)
NoKill KillRestart
QoS Charge
60000
70000
80000
90000
100000
110000
120000
130000
140000
150000
1 1.5 2 2.5 3 4
Slack Factor (SF)Q
oS
Ch
arg
e
NoKill KillRestart
• Kill-and-Restart loses out on resource charge due to wastage of resources
• Gains on the QoS charge due to its ability to accept later arriving expensive jobs
Impact of Non-deadline Jobs (20% deadline)Resource Charge
7.3
7.4
7.5
7.6
7.7
7.8
7.9
8
1 1.5 2 2.5 3 4
Slack Factor (SF)
Re
sou
rce
Cha
rge
(x
10
^8)
NoKill KillRestart
QoS Charge
15000
17000
19000
21000
23000
25000
27000
29000
31000
33000
35000
1 1.5 2 2.5 3 4
Slack Factor (SF)Q
oS
Ch
arg
e
NoKill KillRestart
• Benefits of Kill-and-Restart are even higher
• Benefits seen in both cost components
Overview
Introduction and Motivation
Cost Model for Supercomputer Centers
Understanding User-Tolerance
Dealing with User-Tolerance
Artificial Slack
Kill-and-Restart
Experimental Results
Conclusions and Future Work
Concluding Remarks
Considerable Research on the topic of Parallel Job Scheduling
The issue of provision of QoS has received little attention
In this paper, we extend our previous QoS based scheme, QoPS
– A feedback mechanism to provide the best possible deadline
– Study the impact of user-tolerance on overall revenues
– Propose schemes to minimize the negative impacts of user-tolerance
Demonstrated capabilities and issues based on simulation studies
Future Work
Incorporate the QoPS scheme into Maui/Moab at the OSC
Study QoS aspects for multi-site schedulers
Clusters with heterogeneous systems
Extend the scheme to multiple resources
Study QoS aspects for network flow scheduling
The Basic QoPS Algorithm
When a job (JN+1) arrives
– If J1, J2, …, JN are already present and scheduled in that order
– Place the job (JN+1) at the start of all jobs
Try scheduling the jobs in that order
– If all jobs are able to meet their deadlines, Great ! Admit it !
– If some job fails, we have two options:
– Option1:
Consider the failed job as a critical job
Push the failed job to the start of the schedule and retry
‘k’ number of such re-orderings of existing jobs are allowed
If (number of re-orderings > k) switch to option 2
– Option2:
Back off exponentially in the position at which you try placing job (JN+1) and retry
J3 J2J2 J1
Working of the QoPS Algorithm
J12 J11 J7J8 J6 J5J10 J9 J4 J1
J13
J13
J13
J1
J2
J3
J3J13
J13
J1
J2
J3
J1 J13 J2J3
J13
J1
J4
J3
J2
J6 J5 J4J1J13 J2J3
Max. Violations Allowed = 2
Current Violations = 0Current Violations = 1Current Violations = 2Current Violations = 0
Simulation Approach
CTC/SDSC Trace
Load Variation
Deadline Calculator
Deadline-based Trace
QoPSSimulation
MSBSimulation
MRTSimulation
EASYSimulation
Duplication/Expansion
Trace Generation
Many job logs available, but no associated deadlines
Synthetic Deadline Generation
– Generate a schedule for the job trace using EASY
– For any job J, if the Turnaround time in this schedule is T
– Deadline for J = Arrival Time + max (runtime, (1-SF) x T)
– SF is the “Stringency factor” (0 < SF < 1)
0 would give the least stringent deadlines and 1 the most stringent
Some jobs might not come with deadlines
– Very lax deadlines to prevent starvation
– If ‘T’ is the current expected Turnaround time,
Deadline = Arrival Time + max (24hrs, R x T)
– R is the “Relaxation Factor” of the schedule