21
A Side-channel Attack on HotSpot Heap Management Xiaofeng Wu, Kun Suo, Yong Zhao, Jia Rao The University of Texas at Arlington July 9, 2018 HotCloud’18 1
ASide-channelAttackonHotSpotHeapManagement
Xiaofeng Wu, Kun Suo, Yong Zhao, Jia RaoThe University of Texas at Arlington
July9,2018HotCloud’181
Side-Channel Attack
• Attack based on information gained from the implementation of acomputer system
• Shared cache
• Timing
• Power consumption
• Acoustic measurement
2
Steal or infer secrets
Infer user activities tolaunch well-timed attack
Attack shared clock in multi-tenant systems tomanipulate users’ time measurement
GarbageCollection in HotSpot JVM
ParallelScavenge
3
eden s0 s1
young gen old gen: Minor GC : Major GC
.
...
VM thread GC threadsOther mutator threads VM thread
.
...
Other mutator threads
.
.
Stop The World
• Each individual GC shouldn’t take too long – large heap• Total time spent in GC shouldn’t be too much – small heap, too frequent GC
AdaptiveHeapSizinginPSGC
• Three objectives• Meet pausetime target• Meet throughputgoal• Minimize memoryfootprint
4
JVM automatically determines the heap sizein the range of the initial (-Xms) and themaximum (-Xmx) heap sizes
Pause time
Throughput
Footprint
Shrink heap
Expand heap
Shrink heap
Time is used as an indirect measure formemory efficiency
MinorandMajorGC
Major GCmutator Minor GCmutator mutator
T1
T3
T2
Major GC
T4
Minor GC cost = T2 / ( T1+T2 )
Major GC cost = T4 / ( T3+T4 )
Minor mutator time Minor GC time
Major mutator time Major GC time
Minor GC
Vulnerability
JVM throws an out-of-memory (OOM) error if fiveGCs fail to resolve the memory allocation failure
JVM infers heap efficiency based on measuredlengths of minor and major GCs, and adjustsheap size accordingly
5
step must be performed
step may be performed
Minor GCMem alloc failure Minor GC Check free space
in Old Gen
Major GC
Fail Not enough
Check free space in Old Gen
Major GC
Not enough
1st Alloc in Young Gen
Fail
1st Alloc in Young Gen
Fail
Major GCMajor GC2nd Alloc in Young Gen
2nd Alloc in Young Gen
3rd Alloc in Old Gen
Fail3rd Alloc in
Old GenMajor GCFail Major GC
4th Alloc in Young Gen
4th Alloc in Young Gen
5th Alloc in Old Gen
Fail5th Alloc in Old Gen
FailOut of Memory
Shared Clock
6
GC starts GC ends
JVM is not running
JVM is running
JVM is running
Measured GC time
1 2 32 3+ +1 = wall-clock time
3+1 = virtual time
gettimeofday()gettimeofday()
Time measurement can be inaccurate in thepresence of CPU multiplexing
Three Types of Attacks
• Cause OOM errors• Prevent JVM from expanding the heap in 5 GCs
• Cause excessive GC• Cause bloated heap
7
OOM Attack
• Attack pause time target• When there is a spike in memory demand and allocation failure, attack majorGC measurement
• Dilated major GC time cause
the heap to shrink, missing the
opportunity to avoid OOM errors
Pause time
Throughput
Footprint
Shrink heap
Expand heap
Shrink heap
8
Excessive GC Attack
• Similar to OOM attack but more general
• Oldgenerationhaveatendencytodropquickly,andthedecrementofheapsizeresultsinmoreGCs
9
mutatormutator mutator mutator
T1 T2
T3 T4’
mutator
T5’
T6
Minor GC Minor GC Minor GCMajor GC Major GC
Major GCmutatorMinor GCmutator Minor GCmutatorMinor GCmutator Major GCmutator
T1 T2
T3 T4
T5
T6
Target major GC,dilate its time
Memory Bloat Attack
10
Major GCmutatormutator mutator Major GCmutator
T1 T2’
T3 T4
mutator
T5
T6’
Minor GC Minor GC Minor GC
Major GCmutatorMinor GCmutator Minor GCmutatorMinor GCmutator Major GCmutator
T1 T2
T3 T4
T5
T6
Throughput =T1
T1 + T2<latexit sha1_base64="Rjym03mwzs1hdTQ10Uc/eE8jsGA=">AAACBnicbVDLSgMxFM3UV62vUZciBIsgCGWmCLoRim5cVpg+oB1KJs10QjPJkGSEMszKjb/ixoUibv0Gd/6NaTsLbT1w4XDOvSTnBAmjSjvOt1VaWV1b3yhvVra2d3b37P2DthKpxKSFBROyGyBFGOWkpalmpJtIguKAkU4wvp36nQciFRXc05OE+DEacRpSjLSRBvaxF0mRjqIk1fAa9kOJcOa5uZlzr54P7KpTc2aAy8QtSBUUaA7sr/5Q4DQmXGOGlOq5TqL9DElNMSN5pZ8qkiA8RiPSM5SjmCg/m8XI4alRhjAU0gzXcKb+vshQrNQkDsxmjHSkFr2p+J/XS3V45WeUm5CE4/lDYcqgFnDaCRxSSbBmE0MQltT8FeIImSq0aa5iSnAXIy+Tdr3mOjX3/qLauCnqKIMjcALOgAsuQQPcgSZoAQwewTN4BW/Wk/VivVsf89WSVdwcgj+wPn8A/xqYJw==</latexit><latexit sha1_base64="Rjym03mwzs1hdTQ10Uc/eE8jsGA=">AAACBnicbVDLSgMxFM3UV62vUZciBIsgCGWmCLoRim5cVpg+oB1KJs10QjPJkGSEMszKjb/ixoUibv0Gd/6NaTsLbT1w4XDOvSTnBAmjSjvOt1VaWV1b3yhvVra2d3b37P2DthKpxKSFBROyGyBFGOWkpalmpJtIguKAkU4wvp36nQciFRXc05OE+DEacRpSjLSRBvaxF0mRjqIk1fAa9kOJcOa5uZlzr54P7KpTc2aAy8QtSBUUaA7sr/5Q4DQmXGOGlOq5TqL9DElNMSN5pZ8qkiA8RiPSM5SjmCg/m8XI4alRhjAU0gzXcKb+vshQrNQkDsxmjHSkFr2p+J/XS3V45WeUm5CE4/lDYcqgFnDaCRxSSbBmE0MQltT8FeIImSq0aa5iSnAXIy+Tdr3mOjX3/qLauCnqKIMjcALOgAsuQQPcgSZoAQwewTN4BW/Wk/VivVsf89WSVdwcgj+wPn8A/xqYJw==</latexit><latexit sha1_base64="Rjym03mwzs1hdTQ10Uc/eE8jsGA=">AAACBnicbVDLSgMxFM3UV62vUZciBIsgCGWmCLoRim5cVpg+oB1KJs10QjPJkGSEMszKjb/ixoUibv0Gd/6NaTsLbT1w4XDOvSTnBAmjSjvOt1VaWV1b3yhvVra2d3b37P2DthKpxKSFBROyGyBFGOWkpalmpJtIguKAkU4wvp36nQciFRXc05OE+DEacRpSjLSRBvaxF0mRjqIk1fAa9kOJcOa5uZlzr54P7KpTc2aAy8QtSBUUaA7sr/5Q4DQmXGOGlOq5TqL9DElNMSN5pZ8qkiA8RiPSM5SjmCg/m8XI4alRhjAU0gzXcKb+vshQrNQkDsxmjHSkFr2p+J/XS3V45WeUm5CE4/lDYcqgFnDaCRxSSbBmE0MQltT8FeIImSq0aa5iSnAXIy+Tdr3mOjX3/qLauCnqKIMjcALOgAsuQQPcgSZoAQwewTN4BW/Wk/VivVsf89WSVdwcgj+wPn8A/xqYJw==</latexit><latexit sha1_base64="Rjym03mwzs1hdTQ10Uc/eE8jsGA=">AAACBnicbVDLSgMxFM3UV62vUZciBIsgCGWmCLoRim5cVpg+oB1KJs10QjPJkGSEMszKjb/ixoUibv0Gd/6NaTsLbT1w4XDOvSTnBAmjSjvOt1VaWV1b3yhvVra2d3b37P2DthKpxKSFBROyGyBFGOWkpalmpJtIguKAkU4wvp36nQciFRXc05OE+DEacRpSjLSRBvaxF0mRjqIk1fAa9kOJcOa5uZlzr54P7KpTc2aAy8QtSBUUaA7sr/5Q4DQmXGOGlOq5TqL9DElNMSN5pZ8qkiA8RiPSM5SjmCg/m8XI4alRhjAU0gzXcKb+vshQrNQkDsxmjHSkFr2p+J/XS3V45WeUm5CE4/lDYcqgFnDaCRxSSbBmE0MQltT8FeIImSq0aa5iSnAXIy+Tdr3mOjX3/qLauCnqKIMjcALOgAsuQQPcgSZoAQwewTN4BW/Wk/VivVsf89WSVdwcgj+wPn8A/xqYJw==</latexit>
Throughput0=
T1
T1 + T20<latexit sha1_base64="iZ8FzPFEAEZTvVCsMZjYVjlOpIc=">AAACDnicbVDLSsNAFJ3UV62vqEs3g6UoCCUpgm6EohuXFdIHtLFMppNm6GQSZiZCCfkCN/6KGxeKuHXtzr9x0mahrQcuHM65l5lzvJhRqSzr2yitrK6tb5Q3K1vbO7t75v5BR0aJwKSNIxaJnockYZSTtqKKkV4sCAo9Rrre5Cb3uw9ESBpxR01j4oZozKlPMVJaGpo1JxBRMg7iRN2nJxm8ggNfIJw6dqbnzGnkajY0q1bdmgEuE7sgVVCgNTS/BqMIJyHhCjMkZd+2YuWmSCiKGckqg0SSGOEJGpO+phyFRLrpLE4Ga1oZQT8SeriCM/X3RYpCKaehpzdDpAK56OXif14/Uf6lm1KuwxKO5w/5CYMqgnk3cEQFwYpNNUFYUP1XiAOk61C6wYouwV6MvEw6jbpt1e2782rzuqijDI7AMTgFNrgATXALWqANMHgEz+AVvBlPxovxbnzMV0tGcXMI/sD4/AEM/Ztx</latexit><latexit sha1_base64="iZ8FzPFEAEZTvVCsMZjYVjlOpIc=">AAACDnicbVDLSsNAFJ3UV62vqEs3g6UoCCUpgm6EohuXFdIHtLFMppNm6GQSZiZCCfkCN/6KGxeKuHXtzr9x0mahrQcuHM65l5lzvJhRqSzr2yitrK6tb5Q3K1vbO7t75v5BR0aJwKSNIxaJnockYZSTtqKKkV4sCAo9Rrre5Cb3uw9ESBpxR01j4oZozKlPMVJaGpo1JxBRMg7iRN2nJxm8ggNfIJw6dqbnzGnkajY0q1bdmgEuE7sgVVCgNTS/BqMIJyHhCjMkZd+2YuWmSCiKGckqg0SSGOEJGpO+phyFRLrpLE4Ga1oZQT8SeriCM/X3RYpCKaehpzdDpAK56OXif14/Uf6lm1KuwxKO5w/5CYMqgnk3cEQFwYpNNUFYUP1XiAOk61C6wYouwV6MvEw6jbpt1e2782rzuqijDI7AMTgFNrgATXALWqANMHgEz+AVvBlPxovxbnzMV0tGcXMI/sD4/AEM/Ztx</latexit><latexit sha1_base64="iZ8FzPFEAEZTvVCsMZjYVjlOpIc=">AAACDnicbVDLSsNAFJ3UV62vqEs3g6UoCCUpgm6EohuXFdIHtLFMppNm6GQSZiZCCfkCN/6KGxeKuHXtzr9x0mahrQcuHM65l5lzvJhRqSzr2yitrK6tb5Q3K1vbO7t75v5BR0aJwKSNIxaJnockYZSTtqKKkV4sCAo9Rrre5Cb3uw9ESBpxR01j4oZozKlPMVJaGpo1JxBRMg7iRN2nJxm8ggNfIJw6dqbnzGnkajY0q1bdmgEuE7sgVVCgNTS/BqMIJyHhCjMkZd+2YuWmSCiKGckqg0SSGOEJGpO+phyFRLrpLE4Ga1oZQT8SeriCM/X3RYpCKaehpzdDpAK56OXif14/Uf6lm1KuwxKO5w/5CYMqgnk3cEQFwYpNNUFYUP1XiAOk61C6wYouwV6MvEw6jbpt1e2782rzuqijDI7AMTgFNrgATXALWqANMHgEz+AVvBlPxovxbnzMV0tGcXMI/sD4/AEM/Ztx</latexit><latexit sha1_base64="iZ8FzPFEAEZTvVCsMZjYVjlOpIc=">AAACDnicbVDLSsNAFJ3UV62vqEs3g6UoCCUpgm6EohuXFdIHtLFMppNm6GQSZiZCCfkCN/6KGxeKuHXtzr9x0mahrQcuHM65l5lzvJhRqSzr2yitrK6tb5Q3K1vbO7t75v5BR0aJwKSNIxaJnockYZSTtqKKkV4sCAo9Rrre5Cb3uw9ESBpxR01j4oZozKlPMVJaGpo1JxBRMg7iRN2nJxm8ggNfIJw6dqbnzGnkajY0q1bdmgEuE7sgVVCgNTS/BqMIJyHhCjMkZd+2YuWmSCiKGckqg0SSGOEJGpO+phyFRLrpLE4Ga1oZQT8SeriCM/X3RYpCKaehpzdDpAK56OXif14/Uf6lm1KuwxKO5w/5CYMqgnk3cEQFwYpNNUFYUP1XiAOk61C6wYouwV6MvEw6jbpt1e2782rzuqijDI7AMTgFNrgATXALWqANMHgEz+AVvBlPxovxbnzMV0tGcXMI/sD4/AEM/Ztx</latexit>
Pause time
Throughput
Footprint
Shrink heap
Expand heap
Shrink heap X
Attack minor GC to prevent the heap fromshrinking even memory demand drops
Violate throughput target
Launch Attacks
• Proof-of-concept attacks• Modify JVM source code to manipulate GC time in the adaptive sizing algorithm
• Realistic attacks• Use eBPF to monitor libjvm.so to obtain GC thread ID and slowdown a specific type of GC
• Use cgroup to limit the CPU usage of GC threads and hence dilate GC time
• Results• Crash a Java-based micro-benchmark with OOM errors
• Cause ~65% more GC time in DaCapo
• Inflict up to ~400% memory bloat in SPECjvm2008
11
OOM Attack
• Attack major GC measurement• JAVA_OPTION=
• -XX:+UseAdaptiveSizePolicy• -XX:+UseParallelGC• -XX:+UseParallelOldGC• -XX:ParallelGCThreads=10• -Xms = 32m -Xmx = 2g
• Both proof-of-concept and realisticattacks crash the micro-benchmark
12
ArrayDeque
50 Bytes 20 MB
A micro-benchmark with a sudden spike inmemory demand
Pause time
Throughput
Footprint
Shrink heap
Expand heap
Shrink heap
Discussion
• Essence of the problem• Heap size should be determined by the characteristics of a Java program• But heap efficiency is measured by GC time, an indirect measure• External CPU contention can affect internal heap management
• Many programs designed for dedicated systems are vulnerable tosimilar attacks in multi-tenant systems
• CPU multiplexingà wall-clock time or virtual time?• VMs, containers, conventional processes• Linux jiffies and userspace gettimeofday track wall-clock time• Linux CFS uses steal_clock to track virtual time for thread scheduling
13See our [Suo-SoCC17] paper for another issue caused by time discontinuity
Is this a real problem?
• No• No evidence that manyapplications suffer frominaccurate time measurement.
• Even so, the effect is randomand universally distributedamong applications.
• Our attack is sophisticated andneeds to target a specific typeof GC, not easy.
14
• Yes• In theory, if not measuringabsolute latency, timemeasurement that is onlyrelevant to a particularprogram or to measure therelative progress of programthreads, should use virtualtime
• This could be the source oferroneous program behavior,unpredictability andinefficiency
Should we devise a completely isolated virtual timeinterface for individual programs/VMs/containers ?
Backup Slides …
16
A Realistic Attack• All experiments were conducted on a 64-core machine using
OpenJDK 1.8 and Linux 4.15.0. • The JVM was configured with 10 GC threads.• Benchmark
• Dacapo: h2• SPECjvm2008: mpegaudio
17
Pausetime-orientedAttack (excessive GC)
• A realistic attack using eBPF• Benchmark: h2 from Dacapo• Theinitial andmaximumheapsizes: 16MBand900MB• Themaximumpausetimeis set to 100ms
18
The attack shrinks the heap, causing 88%more GC time
Cont’d - Pause time-oriented• We choose h2 from Dacapo-9.12-MR1-bach as a case study
• execute a number of transactions• set the maximum pause time as 100 ms
19
The overhead induced by the pause time-oriented attack to the micro-benchmark.
Cont’d - Throughput-oriented
• -Xms32m -Xmx32g• Heap size is 1.61× larger
20
0 200 400 600 800
1000 1200 1400 1600
0 2 4 6 8 10 12 14
Size
(MB)
Time (s)
(a) Changes of heap size without attack
Used heap before GCUsed heap after GC
Heap size
0 200 400 600 800
1000 1200 1400 1600
0 2 4 6 8 10 12 14
Size
(MB)
Time (s)
(b) Changes of heap size under attack
ArrayDeque
50 Bytes
pause time
throughput
footprint
reduce generation
increase generation
reduce generation
Throughput-oriented Attack (memory bloat)
• A realistic attack using eBPF• mpegaudio from SPECjvm2008• Theinitial andmaximumheapsizes: 32MBand2.5GB
21
0 100 200 300 400 500 600 700 800 900
50 100 150 200 250 300 350 400
Size
(MB)
Time (s)
(a) Changes of heap size without attack
Used heap before GCUsed heap after GC
Heap size
0 100 200 300 400 500 600 700 800 900
50 100 150 200 250 300 350 400
Size
(MB)
Time (s)
(b) Changes of heap size under attack
w/o attack
under attackThe attack prevents the heap from shrinkingwhen memory demand drops, causing more
than 400% waste of memory
