This paper is included in the Proceedings of the 2020 USENIX Annual Technical Conference. July 15–17, 2020 978-1-939133-14-4 Open access to the Proceedings of the 2020 USENIX Annual Technical Conference is sponsored by USENIX. Fewer Cores, More Hertz: Leveraging High-Frequency Cores in the OS Scheduler for Improved Application Performance Redha Gouicem and Damien Carver, Sorbonne University, LIP6, Inria; Jean-Pierre Lozi, Oracle Labs; Julien Sopena, Sorbonne University, LIP6, Inria; Baptiste Lepers and Willy Zwaenepoel, University of Sydney; Nicolas Palix, Université Grenoble Alpes; Julia Lawall and Gilles Muller, Inria, Sorbonne University, LIP6 https://www.usenix.org/conference/atc20/presentation/gouicern
Transcript
This paper is included in the Proceedings of the 2020 USENIX Annual Technical Conference.
July 15–17, 2020978-1-939133-14-4
Open access to the Proceedings of the 2020 USENIX Annual Technical Conference
is sponsored by USENIX.
Fewer Cores, More Hertz: Leveraging High-Frequency Cores in the OS Scheduler
Baptiste Lepers, Willy ZwaenepoelUniversity of Sydney
Nicolas PalixUniversité Grenoble Alpes
Julia Lawall, Gilles MullerInria, Sorbonne University, LIP6
AbstractIn modern server CPUs, individual cores can run at differentfrequencies, which allows for fine-grained control of the per-formance/energy tradeoff. Adjusting the frequency, however,incurs a high latency. We find that this can lead to a problemof frequency inversion, whereby the Linux scheduler placesa newly active thread on an idle core that takes dozens to hun-dreds of milliseconds to reach a high frequency, just before an-other core already running at a high frequency becomes idle.
One source of challenges in managing core frequencies isthe Frequency Transition Latency (FTL). Indeed, transitioninga core from a low to a high frequency, or conversely, has anFTL of dozens to hundreds of milliseconds. FTL leads to aproblem of frequency inversion in scenarios that are typicalof the use of the standard POSIX fork() and wait() systemcalls on process creation, or of synchronization betweenlightweight threads in a producer-consumer application. Theproblem occurs as follows. First, a task Twaker running on coreCwaker creates or unblocks a task Twoken. If the CompletelyFair Scheduler (CFS), i.e., the default scheduler in Linux,finds an idle core CCFS, it will place Twoken on it. Shortlythereafter, Twaker terminates or blocks, because e.g., it wasa parent process that forked a child process and waitedjust afterwards, or because it was a thread that was doneproducing data and woke up a consumer thread as its lastaction before going to sleep. Now Cwaker is idle and yetexecuting at a high frequency because it was running Twaker
until recently, and CCFS, on which Twoken is running, is likelyto be executing at a low frequency because it was previouslyidle. Consequently, the frequencies at which Cwaker and CCFS
operate are inverted as compared to the load on the cores. Thisfrequency inversion will not be resolved until Cwaker reachesa low frequency and CCFS reaches a high frequency, i.e., forthe duration of the FTL. Current hardware and softwareDFS policies, including the schedutil policy [9] that wasrecently added to CFS cannot prevent frequency inversionas their only decisions consist in updating core frequencies,thus paying the FTL each time. Frequency inversion reducesperformance and may increase energy usage.
Xeon-based machine with 80 cores (160 hardware threads).Our case study finds repeated frequency inversions when pro-cesses are created through the fork() and wait() systemcalls, and our profiling traces make it clear that frequencyinversion leads to tasks running on low frequency cores for asignificant part of their execution.
USENIX Association 2020 USENIX Annual Technical Conference 435
Figure 1: Execution trace when building the Linux kernel version 5.4 using 320 jobs.
Based on the results of the case study, we propose to ad-dress frequency inversion at the scheduler level. Our key ob-servation is that the scheduler can avoid frequency inversionby taking core frequencies into account when placing a taskon a core. For this, we propose and analyze two strategies.Our first strategy Slocal is for the scheduler to simply placeTwoken on Cwaker, as frequency inversion involves a core Cwaker
that is likely at a high frequency, and may soon be idle. Thisstrategy improves the kernel build performance. It runs therisk, however, that Twaker does not promptly terminate or block,causing a long wait before Twoken is scheduled. Accordingly,our second strategy Smove additionally arms a high-resolutiontimer when it places Twoken on Cwaker, and if the timer expiresbefore Twoken is scheduled, then Twoken is migrated to CCFS,i.e., the core CFS originally chose for it. Furthermore, evenslightly delaying Twoken by placing it on Cwaker is not worth-while when CCFS is above the minimum frequency. Thus,Smove first checks whether the frequency of CCFS is above theminimum, and if so places Twoken on CCFS directly.
The contributions of this paper are the following.
• The identification of the frequency inversion phe-nomenon, which leads to some idle cores running ata high frequency while some busy cores run at a lowfrequency for a significant amount of time.
• A case study, building the Linux kernel on an 80-coreserver, with independent per-core frequencies.
• Two strategies, Slocal and Smove, to prevent frequency in-version in CFS. Implementing these policies only re-quired minor code changes: 3 (resp. 124) lines were mod-ified in the Linux kernel to implement Slocal (resp. Smove).
With the powersave governor on the server machine, wefind that both Slocal and Smove perform well overall: out ofthe 60 applications used in the evaluation, Slocal and Smove
improve the performance of 27 and 23 applications by morethan 5% respectively, and worsen the performance of only 3applications by more than 5%. In the best case, Slocal and Smove
improve application performance by 58% and 56% respec-tively with no energy overhead. However, Slocal performs verypoorly with two of the applications, even worsening perfor-mance by 80% in the worst case, which may not be acceptablefor a general-purpose scheduler. Smove performs much betterin the worst case: the increase in application execution timeis only 8% and mitigated by a 9% improvement in terms ofenergy usage. Evaluation results with schedutil show thatthis governor does not address the frequency inversion issue,and exhibits several more cases in which Slocal performs verypoorly—while Smove again has much better worst-case perfor-mance. The evaluation on the desktop machine shows similartrends, albeit on a smaller scale. Again, Smove performs betterthan Slocal on edge cases.
SpeedStep and Turbo Boost technologies, our CPUs can in-dividually vary the frequency of each core between 1.2 and3.0 GHz. The frequency of the two hardware threads of a coreis the same. In the rest of the paper, for simplicity, we use theterm “core” for hardware threads.
Figure 1 shows the frequency of each core of the machinewhile the kernel build workload is running. This plot wasproduced with two tools that we have developed, SchedLogand SchedDisplay [10]. SchedLog collects the executiontrace of an application with very low overhead. SchedDisplayproduces a graphical view of such a trace. We have usedSchedDisplay to generate all execution traces presented in thispaper. SchedLog records the frequency information shown in
436 2020 USENIX Annual Technical Conference USENIX Association
Figure 1 at each tick event (4ms in CFS). Consequently, theabsence of a colored line in such traces means that ticks havebeen disabled by CFS on that core. CFS disables ticks oninactive cores to allow them to switch to a low-power state.
In Figure 1, we notice different phases in the execution. Fora short period around 2 seconds, for a longer period between4.5 and 18 seconds, and for a short period around 28 seconds,the kernel build has highly parallel phases that use all of thecores at a high frequency. The second of these three phasescorresponds to the bulk of the compilation. In these threephases, the CPUs seem to be exploited to their maximum.Furthermore, between 22 and 31 seconds, there is a longphase of mostly sequential code with very few active cores, ofwhich there is always one running at a high frequency. In thisphase, the bottleneck is the CPU’s single-core performance.
Between 0 and 4.5 seconds, and between 18 and 22 secondshowever, there are phases during which all cores are used butthey run at the CPU’s lowest frequency (1.2 GHz). Uponcloser inspection, these phases are actually mainly sequential:zooming in reveals that while all cores are used across theduration of the phase, only one or two cores are used at anygiven time. This raises two questions: why are so many coresused for a nearly sequential execution, and why are thosecores running at such a low frequency.
We focus on the first couple of seconds where core uti-lization seems to be suboptimal. Zooming around 1 second,we first look at runqueue sizes and scheduling events, as il-lustrated in Figure 2a. We are in the presence of a patternthat is typical of mostly-sequential shell scripts: processes arecreated through the fork() and exec() system calls, and gen-erally execute one after the other. These processes can easilybe recognized on Figure 2a as they start with WAKEUP_NEWand EXEC scheduler events. After the process that runs onCore 56 blocks around the 0.96 s mark, three such short-lived processes execute one after the other on Cores 132, 140,and 65. After that, two longer-running ones run on Core 69around the 0.98 s mark, and on Core 152 between the 0.98 sand 1.00 s mark. This pattern goes on for the entire durationof the execution shown in Figure 2a, with tasks created oneafter the other on Cores 148, 125, 49, 52, 129, 156, 60 andfinally 145.
Looking at the core frequencies in the same part of theexecution, as illustrated by Figure 2b, gives us a hint as towhy cores are running slowly in this phase: there seems to bea significant delay between the time when a task starts run-ning on a core, and the time when the core frequency startsincreasing. For instance, between 1.00 s and 1.02 s, the taskon Core 49 runs at a low frequency, and only when it is overat around 1.04 s does the frequency of the core rise to itsmaximum—before starting to decrease again almost instantlyas the hardware notices that no task is running anymore on thatcore. The same issue can be observed shortly before 1.00 s onCore 152, and around 0.98 s on Core 69. In this last example,the core’s frequency was even on a downward slope when the
(a) Scheduler events.
(b) Core frequencies.
Figure 2: Zoom over a sparse region of Figure 1.
task started, and the frequency keeps going down even afterthe task ended before finally increasing again around 1.00 s.It appears that in the considered phase of the execution, theFTL is much higher than the duration of the tasks. Since tasksthat follow each other tend to be scheduled on different cores,they are likely to always run at a low frequency as most coresare idle most of the time in this phase of the execution.
To confirm our intuition about the FTL, we develop afine-grained tool [1] to monitor the frequency of a singlecore around the execution of an isolated busy loop, using thepowersave governor. As shown in Figure 3, the task runs for0.20 s, as illustrated by the start and end vertical lines in thefigure. It takes an FTL of 29 ms for the core to go from itsminimum frequency of 1.25 GHz to its maximum frequencyof 3.00 GHz in order to accommodate the task. When thetask ends, it takes approximately 10 ms for the core to goback to its initial frequency, but the duration of the FTL iscompounded by the fact that the frequency tends to bounceback several times for around 98 ms before stabilizing at thecore’s lowest frequency. These measurements are consistentwith our interpretation of Figure 2b: a FTL of several dozensof milliseconds is significantly longer than the executionof the tasks that are visible in the figure, as the longest taskruns for around 20 ms between the 1.00 s and 1.02 s marks.Note that the duration of the FTL is mainly due to the timefor the hardware to detect a load change and then decideto change the frequency. Previous work [22] shows that theactual latency for the core to change its frequency is only
USENIX Association 2020 USENIX Annual Technical Conference 437
current base min maxFrequency:
start endWorkload:
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40Time (s)
observing is the following. Computations in the (near) sequen-tial phases of the build of Linux are launched sequentiallyas processes through the fork() and wait() system calls,and the execution of these computations is shorter than theFTL. Consequently, cores speed up after they have performeda computation, even though at that point, computation hasmoved to newly forked processes, which are likely to run oncores that were not recently used if the machine is not verybusy. Indeed, CFS often selects different cores for tasks towake up on, and if most cores are idle, it is likely that theselected cores were not used recently, and therefore run at alow frequency. The tasks that initiated the fork() performwait() operations shortly afterwards, which means that thefrequency increase they initiated is mostly wasted. We are inthe presence of recurring frequency inversion, which is causedby a very common scenario: launching a series of sequentialprocesses, as is commonly done in a shell script.
Sequential creation of processes through the fork() andwait() system calls is not the only cause of recurring fre-quency inversion. This phenomenon can also occur withlightweight threads that unblock and block each other, as iscommon in producer-consumer applications. Indeed, the CFScode that selects a core for a new task to wake up on is alsoused to select a core for already existing tasks that wake up.Note that CFS does not use different code paths dependingon the type of task, namely, a process or a thread.
3 Strategies to Prevent Frequency Inversion
Since frequency inversion is the result of scheduling deci-sions, we believe it must be addressed at the scheduler level.In our experience, every change to the scheduler may have un-predictable consequences on some workloads, and the more
complex the change, the less predictable the consequences.Therefore, proposing extensive or complex changes to thescheduler, or a complete rewrite, would make it unclear whereperformance gains come from. Striving for minimal, simplechanges allows for an apples-to-apples comparison with CFS.
We propose two strategies to solve the frequency inversionproblem. The first one is a simple strategy that offers goodperformance but suffers from large performance degradationsin some scheduling scenarios. The second solution aims tohave the same benefits as the first solution while minimizingworst cases at the expense of some simplicity.
3.1 Placing Threads LocallyThe first strategy that we propose to prevent frequency in-version is Slocal: when a thread is created or unblocked, it isplaced on the same core as the process that created or un-blocked it. In the context of creating a single process throughthe fork() and wait() system calls, this strategy implies thatthe created process is more likely to run on a high-frequencycore, as the frequency of the core may already be high due tothe activity of the parent. Furthermore, the duration in whichthere are two processes running on the same core will belimited, if the parent process calls wait() shortly afterwards.In the context of a producer-consumer application, when aproducer thread wakes up a consumer thread, this strategyagain implies that the consumer thread is more likely to runon a high-frequency core, and the duration in which there aretwo processes running on the same core will again be limited,if the last action of the producer is to wake up the consumerbefore blocking or terminating.
However, there are cases in which Slocal might hurt perfor-mance: if the task that created or woke another task does notblock or terminate quickly afterwards, the created or wokentask will wait for the CPU resource for a certain period oftime. This issue is mitigated by the periodic load balancerof the Linux scheduler that will migrate one of the tasks toanother less loaded core. However, waiting for the next loadbalancing event might be quite long. In CFS, periodic loadbalancing is performed hierarchically, with different periods:cores in the same cache domain are more frequently balancedthan cores on different NUMA nodes. These periods can varyfrom 4 to hundreds of milliseconds on large machines.
Slocal significantly changes the behavior of CFS by fully re-placing its thread placement strategy. Additionally, the afore-mentioned shortcomings make it a high risk solution for cer-tain workloads. Both issues make this solution unsatisfactorygiven the prerequisites that we previously set.
3.2 Deferring Thread MigrationsIn order to fix core oversubscription without waiting for pe-riodic load balancing, we propose a second strategy, Smove.With vanilla CFS, when a thread is created or woken, CFS
438 2020 USENIX Annual Technical Conference USENIX Association
Table 1: Configurations of our experimental machines.
decides on which core it should run. Smove defers the use ofthis chosen core to allow waking threads to take advantage ofa core that is more likely to run at a high frequency.
Let Twoken be the newly created or waking task, Cwaker thecore where task Twaker that created or woke Twoken is runningand CCFS the destination core chosen by CFS. The normalbehavior of the scheduler is to directly enqueue task Twoken
into CCFS’s runqueue. We propose to delay this migrationto allow Twoken to be more likely to use a high-frequencycore if CCFS is running at a low frequency. First, if CCFS isrunning at a frequency higher than the CPU’s minimum one,we enqueue Twoken in CCFS’s runqueue. Otherwise, we arm ahigh-resolution timer interrupt that will perform the migrationin D µs and we enqueue Twoken into Cwaker’s runqueue.Thetimer is cancelled if Twoken is scheduled on Cwaker.
The rationale behind Smove is that we want to avoid wakinglow frequency cores if the task can be performed quicklywhen placed locally on a core that is likely to run at ahigh frequency. Indeed, Twaker is running at the time of theplacement, meaning that Cwaker is likely to run at a highfrequency. The delay D can be changed at run time by writingto a parameter file in the sysfs pseudo file system. We havechosen a default value of 50 µs, which is close to the delaybetween a fork and a wait system call during our Linux kernelbuild experiments. We have found that varying the valueof this parameter between 25 µs and 1 ms has insignificantimpact on the benchmarks used in Section 4.
independent frequencies for each core1 We have implementedSlocal and Smove in the latest LTS kernel, Linux 5.4, released inNovember 2019 [3], and compare our strategies to Linux 5.4.
We first consider the execution under powersave. Figure 4ashows the improvement in terms of performance and energyconsumption of Slocal and Smove as compared to CFS. We con-sider that improvements or deteriorations that do not exceed5% to be on par with CFS.
1This is different from turbo frequencies: many desktop and laptop CPUshave per-core DFS in order to support turbo frequencies, but in practice, allcores not using the turbo range run at the same frequency.
USENIX Association 2020 USENIX Annual Technical Conference 439
Slocal Smove
−80 −80−60 −60−40 −40−20 −20
0 020 2040 40
Perfo
rman
ce (%
)
hack
benc
h-10
000
apac
he-0
nas_
bt.B
-160
c-ra
y-0
kbui
ld-a
ll-80
nas_
ua.B
-160
deep
spee
ch-0
redi
s-1
mkl
-dnn
-7-1
perl-
benc
hmar
k-1
node
-oct
ane-
1sc
imar
k2-2
apac
he-s
iege
-5go
-ben
chm
ark-
3sc
hben
ch-6
-7na
s_sp
.B-1
60go
-ben
chm
ark-
2na
s_cg
.C-1
60ol
tp-m
ysql
-80
build
-llvm
-0na
s_lu
.B-1
60sc
imar
k2-1
mkl
-dnn
-7-2
scim
ark2
-4sc
imar
k2-3
scim
ark2
-6gi
t-0ph
pben
ch-0
scim
ark2
-5ao
benc
h-0
go-b
ench
mar
k-4
apac
he-s
iege
-2op
enss
l-0na
s_ep
.C-1
60na
s_m
g.D-
160
go-b
ench
mar
k-1
apac
he-s
iege
-4na
s_cg
.C-8
0kb
uild
-all-
160
kbui
ld-a
ll-32
0ru
st-p
rime-
0ol
tp-m
ysql
-160
oltp
-mys
ql-3
20na
s_ep
.C-8
0ap
ache
-sie
ge-3
build
-linu
x-ke
rnel
-0na
s_lu
.B-8
0na
s_ua
.B-8
0co
mpr
ess-
7zip
-0na
s_ft.
C-16
0na
s_bt
.B-8
0na
s_m
g.D-
80kb
uild
-sch
ed-3
20kb
uild
-sch
ed-1
60kb
uild
-sch
ed-8
0ap
ache
-sie
ge-1
nas_
sp.B
-80
llvm
cmak
ena
s_ft.
C-80
perl-
benc
hmar
k-2
−40 −40
−20 −20
0 0
20 20
40 40
Ener
gy im
prov
emen
t (%
)
3.80
s85
18.5
7 rq
/s4.
69 s
14.5
5 s
36.8
4 s
6.38
s19
3.03
s1.
04e+
6 rq
/s6.
60 m
s1.
71e-
1 s
2.93
e+4
poin
ts25
7.35
Mflo
ps1.
95e+
4 tr/
s1.
78e+
6 ns
/op
1.08
e+5
usec
5.50
s1.
58e+
4 ns
/op
7.35
s1.
58e+
4 tr/
s12
6.89
s5.
63 s
510.
72 M
flops
1566
.50
ms
103.
60 M
flops
991.
23 M
flops
684.
51 M
flops
8.54
s4.
86e+
5 po
ints
517.
47 M
flops
45.2
8 s
8.1e
+5 n
s/op
1.57
e+4
tr/s
1.48
e+4
signs
/s2.
53 s
84.3
0 s
3.e+
10 n
s/op
1.88
e+4
tr/s
12.6
7 s
31.0
8 s
31.1
3 s
4.39
s1.
03e+
4 tr/
s81
29.1
0 tr/
s3.
59 s
1.74
e+4
tr/s
34.5
0 s
6.62
s5.
67 s
2.2e
+5 m
ips
7.80
s7.
65 s
86.0
9 s
6.45
s6.
44 s
6.47
s21
51.7
1 tr/
s6.
31 s
26.9
2 s
10.2
0 s
4.67
e-3
s
1285
.10
J4.
01e+
4 J
2828
.49
J1.
51e+
4 J
1.46
e+4
J33
00.5
5 J
4.73
e+4
J99
16.9
7 J
8506
.28
J2.
76e+
4 J
1.58
e+4
J1.
53e+
4 J
6.30
e+4
J1.
5e+4
J2.
35e+
4 J
3043
.91
J1.
11e+
4 J
4388
.85
J46
93.3
7 J
7.18
e+4
J30
15.9
1 J
1.52
e+4
J1.
24e+
4 J
1.52
e+4
J1.
53e+
4 J
1.5e
+4 J
9666
.20
J1.
23e+
4 J
1.51
e+4
J1.
83e+
4 J
1.42
e+4
J1.
09e+
4 J
1.87
e+4
J12
14.3
5 J
4.74
e+4
J1.
46e+
4 J
3.47
e+4
J63
30.5
4 J
1.31
e+4
J1.
32e+
4 J
8631
.80
J47
80.1
8 J
4844
.18
J15
35.5
4 J
2.03
e+4
J2.
66e+
4 J
3195
.04
J26
34.4
1 J
5.05
e+4
J45
11.3
9 J
3568
.76
J4.
45e+
4 J
1715
.62
J17
13.9
0 J
1719
.92
J1.
03e+
4 J
3077
.06
J67
22.5
6 J
4938
.77
J24
6.97
J
(a) Comparison with CFS using the powersave governor.
−80 −80−60 −60−40 −40−20 −20
0 020 2040 40
Perfo
rman
ce (%
)
hack
benc
h-10
000
apac
he-0
nas_
bt.B
-160
c-ra
y-0
kbui
ld-a
ll-80
nas_
ua.B
-160
deep
spee
ch-0
redi
s-1
mkl
-dnn
-7-1
perl-
benc
hmar
k-1
node
-oct
ane-
1sc
imar
k2-2
apac
he-s
iege
-5go
-ben
chm
ark-
3sc
hben
ch-6
-7na
s_sp
.B-1
60go
-ben
chm
ark-
2na
s_cg
.C-1
60ol
tp-m
ysql
-80
build
-llvm
-0na
s_lu
.B-1
60sc
imar
k2-1
mkl
-dnn
-7-2
scim
ark2
-4sc
imar
k2-3
scim
ark2
-6gi
t-0ph
pben
ch-0
scim
ark2
-5ao
benc
h-0
go-b
ench
mar
k-4
apac
he-s
iege
-2op
enss
l-0na
s_ep
.C-1
60na
s_m
g.D-
160
go-b
ench
mar
k-1
apac
he-s
iege
-4na
s_cg
.C-8
0kb
uild
-all-
160
kbui
ld-a
ll-32
0ru
st-p
rime-
0ol
tp-m
ysql
-160
oltp
-mys
ql-3
20na
s_ep
.C-8
0ap
ache
-sie
ge-3
build
-linu
x-ke
rnel
-0na
s_lu
.B-8
0na
s_ua
.B-8
0co
mpr
ess-
7zip
-0na
s_ft.
C-16
0na
s_bt
.B-8
0na
s_m
g.D-
80kb
uild
-sch
ed-3
20kb
uild
-sch
ed-1
60kb
uild
-sch
ed-8
0ap
ache
-sie
ge-1
nas_
sp.B
-80
llvm
cmak
ena
s_ft.
C-80
perl-
benc
hmar
k-2
−40 −40
−20 −20
0 0
20 20
40 40
Ener
gy im
prov
emen
t (%
)
3.73
s88
15.3
1 rq
/s4.
91 s
14.6
1 s
36.9
2 s
6.40
s22
0.24
s9.
6e+5
rq/s
6.71
ms
1.72
e-1
s2.
86e+
4 po
ints
252.
60 M
flops
1.97
e+4
tr/s
1.88
e+6
ns/o
p1.
08e+
5 us
ec6.
20 s
1.63
e+4
ns/o
p7.
25 s
1.37
e+4
tr/s
126.
45 s
5.81
s50
9.83
Mflo
ps15
66.7
6 m
s10
3.17
Mflo
ps99
1.57
Mflo
ps68
5.91
Mflo
ps8.
71 s
4.85
e+5
poin
ts51
8.96
Mflo
ps45
.24
s8.
51e+
5 ns
/op
1.8e
+4 tr
/s1.
47e+
4 sig
ns/s
2.53
s83
.87
s3.
27e+
10 n
s/op
1.91
e+4
tr/s
12.7
4 s
31.2
6 s
31.2
4 s
4.41
s1.
06e+
4 tr/
s82
05.5
0 tr/
s3.
51 s
1.78
e+4
tr/s
34.8
6 s
6.72
s5.
80 s
2.19
e+5
mip
s7.
62 s
8.29
s87
.14
s6.
71 s
6.73
s6.
71 s
1835
.59
tr/s
6.77
s32
.71
s10
.39
s4.
79e-
3 s
1246
.04
J3.
79e+
4 J
2897
.59
J1.
39e+
4 J
1.47
e+4
J33
19.4
0 J
5.29
e+4
J87
35.1
9 J
7250
.04
J2.
65e+
4 J
1.46
e+4
J1.
41e+
4 J
6.15
e+4
J1.
33e+
4 J
2.22
e+4
J33
52.0
2 J
1.03
e+4
J43
18.9
6 J
4144
.48
J7.
19e+
4 J
3081
.44
J1.
39e+
4 J
1.12
e+4
J1.
41e+
4 J
1.41
e+4
J1.
41e+
4 J
8507
.35
J1.
11e+
4 J
1.41
e+4
J1.
70e+
4 J
1.22
e+4
J95
66.5
0 J
1.74
e+4
J11
96.2
3 J
4.74
e+4
J1.
41e+
4 J
3.36
e+4
J63
23.3
1 J
1.32
e+4
J1.
32e+
4 J
7384
.72
J44
31.5
9 J
4504
.90
J14
99.2
8 J
1.96
e+4
J2.
58e+
4 J
3204
.89
J26
65.5
2 J
4.92
e+4
J44
33.7
2 J
3749
.42
J4.
50e+
4 J
1776
.64
J17
80.8
0 J
1775
.00
J93
23.8
7 J
3251
.07
J81
24.3
5 J
5015
.55
J24
7.28
J
(b) Comparison with CFS using the schedutil governor.
Figure 4: Performance and energy consumption improvement w.r.t. Linux 5.4 on the server machine (higher is better).
440 2020 USENIX Annual Technical Conference USENIX Association
−20 −20
−10 −10
0 0
10 10
20 20
Perfo
rman
ce (%
)
hack
benc
h-10
000
apac
he-0
nas_
bt.B
-160
c-ra
y-0
kbui
ld-a
ll-80
nas_
ua.B
-160
deep
spee
ch-0
redi
s-1
mkl
-dnn
-7-1
perl-
benc
hmar
k-1
node
-oct
ane-
1sc
imar
k2-2
apac
he-s
iege
-5go
-ben
chm
ark-
3sc
hben
ch-6
-7na
s_sp
.B-1
60go
-ben
chm
ark-
2na
s_cg
.C-1
60ol
tp-m
ysql
-80
build
-llvm
-0na
s_lu
.B-1
60sc
imar
k2-1
mkl
-dnn
-7-2
scim
ark2
-4sc
imar
k2-3
scim
ark2
-6gi
t-0ph
pben
ch-0
scim
ark2
-5ao
benc
h-0
go-b
ench
mar
k-4
apac
he-s
iege
-2op
enss
l-0na
s_ep
.C-1
60na
s_m
g.D-
160
go-b
ench
mar
k-1
apac
he-s
iege
-4na
s_cg
.C-8
0kb
uild
-all-
160
kbui
ld-a
ll-32
0ru
st-p
rime-
0ol
tp-m
ysql
-160
oltp
-mys
ql-3
20na
s_ep
.C-8
0ap
ache
-sie
ge-3
build
-linu
x-ke
rnel
-0na
s_lu
.B-8
0na
s_ua
.B-8
0co
mpr
ess-
7zip
-0na
s_ft.
C-16
0na
s_bt
.B-8
0na
s_m
g.D-
80kb
uild
-sch
ed-3
20kb
uild
-sch
ed-1
60kb
uild
-sch
ed-8
0ap
ache
-sie
ge-1
nas_
sp.B
-80
llvm
cmak
ena
s_ft.
C-80
perl-
benc
hmar
k-2
−20 −20
−10 −10
0 0
10 10
20 20
30 30
Ener
gy im
prov
emen
t (%
)
Figure 5: Performance of schedutil compared to powersave with CFS on the server machine.
Performance. Both Slocal and Smove perform well overallwith respectively 27 and 23 out of 60 applications outper-forming CFS. The best results for these policies are seen, asexpected, on benchmarks that extensively use the fork/waitpattern, and therefore exhibit a large number of frequency in-versions. In the best case, Slocal and Smove gain up to 58% and56% respectively on perl-benchmark-2, that measures thestartup time of the perl interpreter. This benchmark benefitsgreatly from avoiding frequency inversions since it mostlyconsists of fork/wait patterns. In terms of performancelosses, both strategies deteriorate the performance of only3 applications, but on very different scales. Slocal deterioratesmkl-dnn-7-1 by 80% and nas_lu.B-160 by 17% whileSmove has a worst case deterioration of 8.4% on hackbench.
Energy consumption. Overall, both Slocal and Smove im-prove energy usage. Out of our 60 applications, we improveenergy consumption by more than 5% for 16 and 14 applica-tions, respectively, compared to CFS. Most of the improve-ments are seen on benchmarks where performance is alsoimproved. In these cases, the energy savings are likely mostlydue to the shorter execution times of the applications. How-ever, we also see some improvements on applications wherethe performance is on par with that on CFS. This is due to thefact that we avoid waking up cores that are in low power states,therefore saving the energy necessary to power up and runthose cores. In terms of loss, Slocal consumes more energy thanCFS on only one application, nas_lu.B-160. This loss is ex-plained by the bad performance of Slocal on this application.This benchmark’s metric is its execution time, and increas-ing the execution time without correspondingly reducing thefrequency increases the energy consumption. Smove consumes
more energy than CFS on two applications: hackbench, be-cause of the performance loss, and deepspeech that has toohigh a standard deviation for its results to have significance.
Overall score. To compare the overall impact of our strate-gies, we compute the geometric mean of all runs, where eachrun is normalized to the mean result of CFS. Smove has a perfor-mance improvement of 6%, a reduction in energy usage of 3%and an improvement of 4% with both metrics combined. Slocal
has similar overall scores (always 5%), but its worst casessuggest that Smove is a better option for a general-purposescheduler. These small differences are expected becausemost of the applications we evaluate perform similarly withCFS and with our strategies. We also evaluate the statisticalsignificance of our results with a t-test. With p-values of atmost 3 ·10−20, we deem our results statistically significant.
4.2 Execution Using schedutil
Next, we consider execution under the schedutil governor.As a baseline, Figure 5 first shows the performance and energyimprovements of the schedutil governor compared to thepowersave governor with CFS. Overall, we observe that theschedutil governor deteriorates the performance of mostapplications while improving energy usage. This indicatesthat this new governor is more aggressive in terms of powersavings than the one implemented in hardware. We omit rawvalues since they are already available in Figures 4a and 4b.Figure 4b then shows the improvement in terms of perfor-mance and energy consumption of our strategies compared toCFS, when using the schedutil governor.
USENIX Association 2020 USENIX Annual Technical Conference 441
Slocal Smove
80 80
60 60
40 40
20 20
0 0
20 20
40 40
Perf
orm
ance
(%
)
hack
bench
-40
0
hack
bench
-20
00
redis
-1
hack
bench
-10
00
nas_
bt.
B-8
nas_
bt.
B-1
6
nas_
bt.
B-4
apach
e-s
iege-2
scim
ark
2-6
mkl
-dnn-7
-2
nas_
ep.C
-16
scim
ark
2-5
scim
ark
2-3
nas_
ep.C
-8
schbench
-6-7
scim
ark
2-1
c-ra
y-0
com
pre
ss-7
zip-0
node-o
ctane-1
apach
e-s
iege-3
nas_
ep.C
-4
nas_
cg.C
-4
nas_
cg.C
-16
nas_
cg.C
-8
nas_
sp.B
-16
nas_
lu.B
-4
apach
e-s
iege-4
nas_
ft.C
-4
go-b
ench
mark
-4
nas_
ua.B
-8
apach
e-s
iege-5
nas_
sp.B
-4
phpbench
-0
scim
ark
2-4
nas_
ua.B
-16
nas_
ft.C
-16
go-b
ench
mark
-3
nas_
sp.B
-8
nas_
lu.B
-8
rust
-pri
me-0
nas_
ua.B
-4
nas_
ft.C
-8
mkl
-dnn-7
-1
nas_
lu.B
-16
scim
ark
2-2
openss
l-0
perl
-bench
mark
-1
kbuild
-all-
16
apach
e-0
aobench
-0
kbuild
-all-
8
go-b
ench
mark
-2
deepsp
eech
-0
kbuild
-all-
4
git
-0
go-b
ench
mark
-1
kbuild
-sch
ed-8
kbuild
-sch
ed-1
6
kbuild
-sch
ed-4
perl
-bench
mark
-2
llvm
cmake
apach
e-s
iege-1
1.4
2e-1
s
7.2
2e-1
s
1.5
7e+
6 r
q/s
3.6
5e-1
s
54
.54
s
59
.07
s
50
.10
s
3.6
2e+
4 t
r/s
43
1.0
3 M
flops
1.7
1e+
4 m
s
18
.19
s
66
1.5
4 M
flops
11
10
.79
Mflops
17
.92
s
1.4
4e+
6 u
sec
49
5.2
9 M
flops
13
4.7
9 s
1.6
6e+
4 m
ips
3.9
1e+
4 p
oin
ts
2.9
9e+
4 t
r/s
33
.78
s
53
.74
s
54
.65
s
53
.39
s
61
.53
s
32
.01
s
2.8
7e+
4 t
r/s
51
.59
s
2.5
7e+
6 n
s/op
58
.87
s
2.7
9e+
4 t
r/s
58
.95
s
5.3
5e+
5 p
oin
ts
12
0.8
4 M
flops
61
.24
s
82
.95
s
9.8
9e+
6 n
s/op
61
.40
s
33
.02
s
82
.51
s
60
.08
s
81
.32
s
28
.45
ms
10
55
.54
s
15
5.1
3 M
flops
86
0.5
9 s
igns/
s
1.5
7e-1
s
26
2.3
8 s
2.2
4e+
4 r
q/s
36
.73
s
26
0.0
7 s
71
51
.70
ns/
op
11
6.1
1 s
35
0.3
6 s
6.0
8 s
2.8
e+
10
ns/
op
7.7
3 s
7.9
8 s
9.6
0 s
2.4
9e-3
s
20
.96
s
48
20
.62
tr/
s
Figure 6: Performance improvement w.r.t. Linux 5.4 on the desktop machine (higher is better).
Performance. Slocal and Smove outperform CFS on 22 and20 applications out of 60 respectively. The applications con-cerned are the same that were improved with the powersavegovernor. In terms of performance losses, however, Slocal ismore impacted by the schedutil governor than Smove, with7 applications performing worse than CFS versus only 2.
Energy consumption. The overall improvement in termsof energy usage of schedutil with CFS would suggest thatwe might see the same trend with Slocal and Smove. And indeed,the results are quite similar to what we observe with thepowersave governor.
Overall score. The geometric means with this governor arethe following for schedutil and Smove: 6% for performance,4% for energy and 5% with both metrics combined. Slocal hassimilar results (2%, 6% and 4% respectively), but the worstcases are still too detrimental for a general-purpose scheduler.These results are also statistically significant with p-values ofat most 3 ·10−20.
As shown in Figure 6, we observe the same general trend ason our server machine. Slocal and Smove behave similarly whenthere is improvement, and Smove behaves better on the fewbenchmarks with performance degradation. We measure atworst an 11% slowdown and at best a 52% speedup for Smove,with an aggregate performance improvement of 2%. Addi-tionally, Smove improves the performance of 7 applications by
more than 5% while only degrading the performance of 4applications at the same scale. The Slocal strategy gives thesame results regarding the number of improved and degradedapplications, but suffers worse edge cases. Its best perfor-mance improvement is 42% while its worst deterioration is25%, with an aggregate performance improvement of 1%. Weconclude that even if there is no major global improvement,Smove is still a good strategy to eliminate frequency inversionson machines with smaller core counts. Our performance re-sults are statistically significant, with p-values of 5 ·10−4 forSmove and 3 ·10−2 for Slocal.
4.4 In-Depth AnalysisWe now present a detailed analysis of specific benchmarksthat either performed particularly well or particularly poorlywith our solutions. In this section all traces were obtainedwith the powersave governor.
kbuild Figure 7 shows the execution of the build of theLinux kernel as presented in the case study, with CFS (top)and Smove (bottom). During the mostly sequential phases withmultiple cores running at a low frequency on CFS (0-2 s,2.5-4.5 s, 17-22 s), Smove uses fewer cores at a higher fre-quency. This is mainly due to the fork()/wait() pattern: asthe waker thread calls wait() shortly after the fork(), theSmove timer does not expire and the woken threads remain onthe local core running at a high frequency, thus avoiding fre-
442 2020 USENIX Annual Technical Conference USENIX Association
(a) CFS
(b) Smove
Figure 7: Execution trace when building the Linux kernelversion 5.4 using 320 jobs.
quency inversion. As a result, for example, the phase beforethe long parallel phase is executed in 4.4 seconds on CFS andin only 2.9 seconds with Smove.
To understand the impact of Smove better, Figure 8 showsthe kbuild-sched-320 benchmark, which builds only thescheduler subsystem of the Linux kernel. Here, the parallelphase is much shorter than with a complete build, as there arefewer files to compile, making the sequential phases of theexecution more visible. Again, we see that fewer cores areused, at a higher frequency.
mkl The mkl-dnn-7-1 benchmark is the worst-case sce-nario for Slocal: all threads keep blocking and unblocking andtherefore avoid periodic load balancing and continue return-ing to the same set of cores. Thus, threads that are sharinga core with another thread will tend to remain there withthe Slocal strategy. Figure 9 shows the number of threads onthe runqueue of each core with all three schedulers with thepowersave governor. A black line indicates that there is onethread in the runqueue, and a red line indicates that there ismore than one. CFS spreads the threads on all cores rapidly,and achieves a balanced machine with one thread per core inless than 0.2 seconds. On the other hand, Slocal tries to max-imize core reuse and oversubscribes 36 cores. This leads tonever using all cores, achieving at most 85% CPU utilizationwith multiple cores overloaded. This is a persistent viola-tion of the work-conservation property, as defined by Lozi etal. [21], i.e., no core is idle if a core has more than one thread
(a) CFS
(b) Smove
Figure 8: Execution trace when building the sched directoryof the Linux kernel version 5.4 using 320 jobs.
in its runqueue.Interestingly, in our experiment, the balancing operations
that spread threads are due to system or daemon threads (e.g.systemd) that wake up and block immediately, thus triggeringan idle balancing from the scheduler. On a machine withnothing running in the background, we could have stayed inan overloaded situation for a long period of time, as ticks aredeactivated on idle cores, removing opportunities for periodicbalancing. We can see the same pattern on nas-lu.B-160,another benchmark that does not work well with Slocal. Smove
solves the problem by migrating, after a configurable delay,the threads that overload cores to available idle cores.
hackbench The hackbench-10000 benchmark is theworst application performance-wise for the Smove strategy.This micro-benchmark is particularly stressful for the sched-uler, with 10,000 running threads. However, the patterns ex-hibited are interesting to better understand the shortcomingsof Smove and give insights on how to improve our strategies.
This benchmark has three phases: thread creation, com-munication and thread termination. Figure 10 shows the fre-quency of all cores during the execution of hackbench withCFS, Slocal and Smove. The first phase corresponds to the firsttwo seconds on all three schedulers. A main thread creates10,000 threads with the fork() system call, and all childthreads immediately wait on a barrier. With CFS, child threadsare placed on idle cores that become idle again when thethreads arrive at the barrier. This means that all cores remain
USENIX Association 2020 USENIX Annual Technical Conference 443
(a) CFS (b) Slocal (c) Smove
Figure 9: Number of threads per core during the execution of mkl-dnn-7-1.
(a) CFS
(b) Slocal
(c) Smove
Figure 10: Core frequency when executing hackbench.
mostly idle. This also leads to the main thread remaining onthe same core during this phase. However, Slocal and Smove
place the child threads locally, causing oversubscription ofthe main thread’s core and migrations by the load balancer.The main thread itself is thus sometimes migrated from coreto core. When all threads are created, the main thread re-leases the threads waiting on the barrier and waits for theirtermination, thus beginning the second phase. During thisphase, the child threads communicate by reading and writingin pipes. CFS tries to even out the load between all cores, butits heuristics give a huge penalty to migrations across NUMAnodes, so a single node runs at a high frequency (cores 0, 4,8, etc. share the same node on our machine) while the othershave little work to perform and run at lower frequencies. Thisphase finishes at 2.8 seconds. The remainder of the executionis the main thread reaping its children and terminating.
Slocal packs threads aggressively, leading to long runqueuesin the second phase, and therefore facilitating load balancingacross nodes because of the large induced overload. However,Slocal still does not use all cores, mainly avoiding running onhyperthreaded pairs of cores (cores n and n+ 80 are hyper-threaded on our machine). Slocal runs the second phase fasterthan CFS, terminating it at 2.5 seconds, because it uses halfof the cores at a high frequency all the time, and many of theother cores run at a medium frequency.
On the other hand, Smove performs poorly in the secondphase, completing it at 3.4 seconds. The behavior seems veryclose to that of CFS, with one core out of four running at ahigh frequency. However, Smove results in more idleness orlow frequency on the other cores. This is due to Smove placingthreads locally: many threads contend for the local core; someare able to use the resource while others are migrated whenthe timer interrupt is triggered. The delays cause idleness com-pared to CFS, and the migrations leave cores idle, loweringtheir frequency compared to Slocal. Additionally, when threadsare migrated because of timers expiring, they are all placed onthe same core, and oversubscribe it. For hackbench, choosing
444 2020 USENIX Annual Technical Conference USENIX Association
the middle ground is the worst strategy. We can also note thatload balancing is not able to mitigate this situation becauseof the high volatility of this workload. This problem was alsodemonstrated by Lozi et al. [21] on a database application.
This hackbench setup is an extreme situation that is un-likely to happen in real life, with a largely overloaded machine(10,000 threads) and a highly volatile application. This mi-crobenchmark is only interesting to study the behavior of ourstrategies. Still, overall, Smove gives better performance thanSlocal.
4.5 Scheduling Overhead of Smove
Smove is more complex than Slocal, and so we analyze its over-head as compared to CFS, as an upper bound for our strategies.We identify two possible sources of overhead: querying fre-quency and using timers.
First, we evaluate the cost of querying the core frequency.Querying the frequency of a core mostly consists in readingtwo hardware registers and performing some arithmetic oper-ations, as the current frequency is the division of these tworegisters times the base frequency of the CPU. Even thoughthis is a very small amount of computation compared to therest of the scheduler, we minimize it furthermore by queryingthis information at every tick instead of every time it is needed.In our benchmarks, we notice no difference in performancewith or without querying the frequency at every tick.
Second, we evaluate the cost of triggering a large numberof timers in the scheduler. To do so, we run schbench ontwo versions of Linux: the vanilla 5.4 kernel and a modifiedversion with timers armed under the same condition as Smove.Here, however, the timer handler does not migrate the threadas in Smove. We choose schbench because it performs thesame workload as hackbench but provides, as a performanceevaluation, the latencies of the messages sent through pipesinstead of the completion time. Table 2 shows the results ofthis benchmark. Overall, the 99.5th percentile of latenciesis the same for both versions of the kernel, except for 256threads where timers have a negative impact. We can alsoobserve that the number of timers triggered increases with thenumber of threads but drops after 256 threads. This behavioris expected: more threads means more wake-ups, but whenthe machine starts being overloaded, all cores run at highfrequencies, and the timers are less frequently armed. Thistipping point arrives around 256 threads because schbenchthreads constantly block, meaning that typically fewer than160 threads are runnable at a time.
5 Discussion
As previously stated, our proposed solutions Slocal and Smove
are purposefully simple. We now discuss other more complexsolutions to the frequency inversion problem.
Table 2: schbench latencies (99.5th percentile, in µsec) andnumber of timers triggered.
High frequency pool. A possible solution would be to keepa pool of cores running at a high frequency even though nothread is running on them. This would allow threads to beplaced on an idle core running at a high frequency instanta-neously. This pool could, however, waste energy and reducethe maximal frequency attainable by busy cores, which dimin-ishes when the number of active cores increases.
Tweaking the placement heuristic. We could add a newfrequency heuristic to the existing placement strategy. How-ever, the tradeoff between using a core running at a higherfrequency and e.g., cache locality is not clear, and may varygreatly according to the workload and the architecture.
Frequency model. The impact of the frequency of one coreon the performance of other cores is hardware-specific. If thescheduler were to take frequency-related decisions, it wouldalso need to account for the impact its decision would haveon the frequency of all cores. Such models are not currentlyavailable, and would be complicated to create.
USENIX Association 2020 USENIX Annual Technical Conference 445
shift of DFS logic from the software side to the hardwareside in recent years, the decision to develop the experimentalschedutil [9] governor in Linux was based on the idea thatsoftware still has a role to play in DFS, as it knows betterthe load being executed. Similarly, our strategies show thatthe software placing tasks on high-frequency cores can bemore efficient than waiting for the hardware to increase thefrequency of cores after task placement, due to the FTL.
Tracking inefficient scheduler behavior. Perf [15,16,32],which is provided with the Linux kernel, supports monitoringscheduler behavior through the perf sched command. Whileperf sched makes it possible to analyze the behavior ofthe scheduler on simple workloads with good accuracy, ithas significant overhead on the Linux kernel build and otherreal-world workloads. Lozi et al. [21] identify performancebugs in the Linux scheduler. To analyze them, they writea basic profiler that monitors, for each core, the number ofqueued threads and the load. Their basic profiler does notmonitor scheduling events. SchedLog and SchedDisplay [10],which we use in this paper, make it possible to record relevantinformation about all scheduler events with low overhead, andto efficiently navigate through the large amount of recordeddata with a powerful and scriptable graphical user interface.
Mollison et al [25] apply regression testing to schedulers.Their focus is limited to real-time schedulers, and they donot take DFS into account. More generally, there has been anongoing effort to test and understand the impact of the Linuxscheduler on performance. Since 2005, the LKP project [12]has focused on hunting performance regressions, and a myriadof tools that make it possible to identify performance bugs inkernels have been proposed by the community [7, 18, 26, 28].The focus of these tools, however, is to detect slowdownsinside the kernel code, and not slowdowns in application codethat were caused by decisions from the kernel. Consequently,they are unable to detect poor scheduling behavior.
Improving scheduler behavior. Most previous work fo-cuses on improving general-purpose OS scheduling with newpolicies that improve a specific performance metric, suchas reducing contention over shared resources [31, 35], opti-mizing the use of CPU caches [29, 30], improving NUMAlocality [8, 14] or minimizing idleness [20]. These paperssystematically disable DFS in their experiments. Merkel etal. [24] propose a scheduling algorithm that avoids resourcecontention by co-scheduling applications that use complemen-tary resources. They reduce contention by lowering the fre-quency of cores that execute inauspicious workloads. Zhanget al. [34] propose a scheduling policy for multi-core archi-tectures that facilitates DFS, although their main focus is re-ducing cache interference. They only consider per-chip DFS,as per-core DFS was not commonplace at the time.
Linux kernel developers have recently focused on DFS andturbo frequencies [13], as it was discovered that a short-lived
jitter process that runs on a previously idle core can make thatcore switch to turbo frequencies, which can in turn reduce thefrequencies used by other cores—even after the jitter processcompletes. To solve this issue, a patch [27] was proposed toexplicitly mark jitter tasks. The scheduler then tries to placethese marked tasks on cores that are active and expected toremain active. In contrast, the frequency inversion issue weidentified is not specifically caused by turbo frequencies: itcan occur with any DFS policy in which different cores mayrun at different frequencies.
Child runs first. CFS has a feature that may seem relatedto our solutions: sched_child_runs_first. At thread cre-ation, this feature assigns a lower vruntime to the child thread,giving it a higher priority than its parent. If CFS places thethread on the same core as its parent, the thread will preemptthe parent; otherwise, the thread will just run elsewhere. Thisfeature does not affect thread placement and thus cannot ad-dress the frequency inversion problem. Using this feature incombination with Smove would defeat Smove’s purpose by al-ways canceling the timer. The strategy would resemble Slocal,except that the child thread would always preempt its parent.
7 Conclusion
In this paper, we have identified the issue of frequency in-version in Linux, which occurs on multi-core CPUs withper-core DFS. Frequency inversion leads to running tasks onlow-frequency cores and may severely degrade performance.We have implemented two strategies to prevent the issue inthe Linux 5.4 CFS scheduler. Implementing these strategiesrequired few code changes: they can easily be ported to otherversions of the Linux kernel. On a diverse set of 60 applica-tions, we show that our better solution, Smove, often signifi-cantly improves performance. Additionally, for applicationsthat do not exhibit the frequency inversion problem, Smove
induces a penalty of 8% or less with 3 of the evaluated ap-plications. As independent core frequency scaling becomesa standard feature on latest generation processors, our workwill target a larger number of machines.
In future work, we want to improve thread placement in thescheduler by including the cores’ frequencies directly in theplacement algorithm. This improvement will need to accountfor various parameters such as architecture-specific DFS, si-multaneous multi-threading and maintaining cache locality.
Acknowledgments and Availability
This work is supported in part by Oracle donation CR 1930.We would also like to thank the anonymous reviewers and ourshepherd, Heiner Litz, for their feedback.
Slocal and Smove patches for Linux 5.4 are available at:https://gitlab.inria.fr/whisper-public/atc20.
446 2020 USENIX Annual Technical Conference USENIX Association
[5] D.H. Bailey, E. Barszcz, J. T. Barton, D. S. Brown-ing, R. L. Carter, L. Dagum, R.A Fatoohi, P. O. Fred-erickson, T. A Lasinski, R. S. Schreiber, H.D. Simon,V. Venkatakrishnan, and S.K. Weeratunga. The NASparallel benchmarks summary and preliminary results.In Supercomputing, pages 158–165, Seattle, WA, USA,1991.
[6] Ricardo Bianchini and Ram Rajamony. Power andenergy management for server systems. Computer,37(11):68–76, 2004.
[7] Silas Boyd-Wickizer, Austin T. Clements, YandongMao, Aleksey Pesterev, M. Frans Kaashoek, Robert Mor-ris, and Nickolai Zeldovich. An analysis of Linux scala-bility to many cores. In OSDI, pages 86–93, Vancouver,BC, Canada, 2010.
[8] Timothy Brecht. On the importance of parallel applica-tion placement in NUMA Multiprocessors. In USENIXSEDMS, San Diego, CA, USA, 1993.
[9] Neil Brown. Improvements in CPU frequency manage-ment. https://lwn.net/Articles/682391/.
[10] Damien Carver, Redha Gouicem, Jean-Pierre Lozi,Julien Sopena, Baptiste Lepers, Willy Zwaenepoel, Nico-las Palix, Julia Lawall, and Gilles Muller. Fork/waitand multicore frequency scaling: a generational clash.In PLOS, pages 53–59, Huntsville, ON, Canada, 2019.ACM.
[11] Jeffrey S Chase, Darrell C Anderson, Prachi N Thakar,Amin M Vahdat, and Ronald P Doyle. Managing energyand server resources in hosting centers. ACM SIGOPSoperating systems review, 35(5):103–116, 2001.
[12] Tim Chen, Leonid I Ananiev, and Alexander V Tikhonov.Keeping kernel performance from regressions. In LinuxSymposium, pages 93–102, Ottawa, ON, Canada, 2007.
[13] Jonathan Corbet. TurboSched: the return of small-taskpacking. Linux Weekly News, July 1, 2019. https://lwn.net/Articles/792471/.
[14] Mohammad Dashti, Alexandra Fedorova, Justin Fun-ston, Fabien Gaud, Renaud Lachaize, Baptiste Lepers,Vivien Quema, and Mark Roth. Traffic management: aholistic approach to memory placement on NUMA sys-tems. In ASPLOS, pages 381–394, Houston, TX, USA,2013.
[15] Arnaldo Carvalho de Melo. Performance counters onLinux. In Linux Plumbers Conference, Portland, OR,USA, 2009.
[16] Arnaldo Carvalho de Melo. The new Linux ‘perf’ tools.In Slides from Linux Kongress, Nuremberg, Germany,2010.
[17] EN Mootaz Elnozahy, Michael Kistler, and Ramakrish-nan Rajamony. Energy-efficient server clusters. InPACS, pages 179–197, Cambridge, MA, USA, 2002.Springer.
[18] Ashif S. Harji, Peter A. Buhr, and Tim Brecht. Ourtroubles with Linux and why you should care. In APSys,pages 1–5, Shanghai, China, 2011.
[20] Baptiste Lepers, Redha Gouicem, Damien Carver, Jean-Pierre Lozi, Nicolas Palix, Maria-Virginia Aponte, WillyZwaenepoel, Julien Sopena, Julia Lawall, and GillesMuller. Provable multicore schedulers with ipanema:application to work conservation. In EuroSys, pages3:1–3:16, Heraklion, Greece, 2020. ACM.
[21] Jean-Pierre Lozi, Baptiste Lepers, Justin Funston, Fa-bien Gaud, Vivien Quéma, and Alexandra Fedorova.The Linux scheduler: a decade of wasted cores. In Eu-roSys, pages 1–16, London, UK, 2016.
[22] Abdelhafid Mazouz, Alexandre Laurent, Benoît Pradelle,and William Jalby. Evaluation of CPU frequency transi-tion latency. Comput. Sci. Res. Dev., 29(3-4):187–195,2014.
[23] Phoronix Media. Phoronix test suite – Linux testing &benchmarking platform, automated testing, open-sourcebenchmarking. http://www.phoronix-test-suite.com/.
[24] Andreas Merkel, Jan Stoess, and Frank Bellosa.Resource-conscious scheduling for energy efficiencyon multicore processors. In EuroSys, pages 153–166,Paris, France, 2010. ACM.
USENIX Association 2020 USENIX Annual Technical Conference 447
[25] Malcolm S Mollison, Björn Brandenburg, and James HAnderson. Towards unit testing real-time schedulers inLITMUSRT. In OSPERT, Stuttgart, Germany, 2009.
[26] Sharon E. Perl and William E. Weihl. Performanceassertion checking. In SOSP, pages 134–145, Asheville,NC, USA, 1993.
[27] Parth Shah. TurboSched: A scheduler for sustainingturbo frequencies for longer durations, June 25, 2019.https://lkml.org/lkml/2019/6/25/25.
[28] Kai Shen, Ming Zhong, and Chuanpeng Li. I/O systemperformance debugging using model-driven anomalycharacterization. In FAST, pages 309–322, San Fran-cisco, CA, USA, 2005.
[29] David Tam, Reza Azimi, and Michael Stumm. Threadclustering: sharing-aware scheduling on SMP-CMP-SMT multiprocessors. In EuroSys, pages 47–58, Lisbon,Portugal, 2007.
[30] Lingjia Tang, J. Mars, Xiao Zhang, R. Hagmann,R. Hundt, and E. Tune. Optimizing Google’s warehousescale computers: The NUMA experience. In HPCA,pages 188–197, Shenzhen, China, 2013.
[31] Abhishek Verma, Luis Pedrosa, Madhukar Korupolu,David Oppenheimer, Eric Tune, and John Wilkes. Large-scale cluster management at Google with Borg. In Eu-roSys, pages 1–17, Bordeaux, France, 2015.
[32] Vincent M Weaver. Linux perf_event features and over-head. In FastPath, pages 80–87, Austin, TX, 2013.
[33] Mark Weiser, Brent B. Welch, Alan J. Demers, and ScottShenker. Scheduling for reduced CPU energy. In OSDI,pages 13–23, Monterey, CA, USA, 1994.
[34] Xiao Zhang, Sandhya Dwarkadas, and Rongrong Zhong.An evaluation of per-chip nonuniform frequency scalingon multicores. In USENIX ATC, Berkeley, CA, USA,2010.
[35] Sergey Zhuravlev, Juan Carlos Saez, Sergey Blagodurov,Alexandra Fedorova, and Manuel Prieto. Survey ofscheduling techniques for addressing shared resources inmulticore processors. ACM Computing Surveys (CSUR),45(1):4, 2012.
448 2020 USENIX Annual Technical Conference USENIX Association
