1

1

Controlled Resource and Data

Sharing in Multi-Core Platforms*

Sandhya Dwarkadas

Department of Computer Science

University of Rochester

*Joint work with Arrvindh Shriraman, Hemayet Hossain, Xiao Zhang,

Hongzhou Zhao, Rongrong Zhong,

Michael L. Scott, Michael Huang, Kai Shen

2

The University of Rochester

• Small private research university

• 4400 undergraduates

• 2800 graduate students

• Set on the Genesee River in Western NY State, near the south shore of Lake Ontario

• 250km by road from Toronto; 590km from New York City

3 4

The Computer Science Dept.

• 15 tenure-track

faculty; 45 Ph.D.

students

• Specializing in AI,

theory, and parallel

and distributed

systems

• Among the best

small departments in

the US

2

5

Handheld DeviceJava

Internet

ClusterFortran/C

data

InterWeave Server

IW libraryIW library

IW library

cache

DesktopC/C++

cache

cache

The Hardware-Software Interface

TreadMarks

Cashmere-2L

InterWeave

RTM FlexTM

Concurrency: Coherence,

Synchronization, Consistency

Memory Systems

Peer-to-peer systems

Power-Aware Computing CAP

MCD

P P P

M

…

ARMCO DDCache LADLE

FCS Multi-Cores

Resource-Aware OS Scheduling

DT-CMT

Distributed Systems

Operating Systems

Willow

Sentry:

Protection Support

RPPT

6

The Implications of Technology

Scaling

• Many more transistors for compute power

• Energy constraints

• Large volumes of data

• High-speed communication

• Concurrency (parallel or distributed)

• Need support for

– Scalable sharing

– Reliability

– Protection and security

– Performance isolation Source: http://weblog.infoworld.com/tech-bottom-line/archives/IT-In-The-Clouds_hp.jpg

7

Multi-Core Challenges

• Ensuring performance isolation

• Providing protected and controlled sharing

across cores

• Scaling support for data sharing

8

Current Projects

• CoSyn: Communication and Synchronization Mechanisms for Emerging Multi-Core Processors

– Collaboration with Professors Michael Scott and Michael Huang

– Arrvindh Shriraman, Hemayet Hossain, Hongzhou Zhao

• Operating System-Level Resource Management in the Multi-Core Era

– Collaboration with Professor Kai Shen

– Xiao Zhang and Rongrong Zhong

See http://www.cs.rochester.edu/research/cosyn

and http://www.cs.rochester.edu/~sandhya

http://www.cs.rochester.edu/research/cosynhttp://www.cs.rochester.edu/~sandhya

3

9

Multi-Core Challenges

• Ensuring performance isolation

• Providing protected and controlled sharing

across cores

• Scaling support for data sharing

10

Performance Isolation

11

11

Resource Sharing is (and will be)

Ubiquitous!

• Floating point, integer, state, cache with multiple threads on a core

• Second-level cache with multiple cores on a chip

• Interconnect bandwidth on multiprocessors

Sun UltraSparc T1, … Intel’s 6-core (12-thread), …

AMD’s 12-core, …

12

Resource Sharing on Multicore Chip

• Memory bandwidth and last level cache are

commonly shared by sibling cores sitting on the

same chip

http://download.intel.com/pressroom/kits/45nm/Penryn Die Photo_300.jpg

4

13

13

Resource Management To Date

• Capitalistic - generation of more requests results

in more resource usage

– Performance: resource contention can result

in significantly reduced overall performance

– Fairness: equal time slice does not

necessarily guarantee equal progress

14

Poor Performance Due to

Uncontrolled Resource Contention

Experiments were conducted on a 3Ghz Intel Core 2 Duo processor with a shared 4MB L2 cache

Win-win situation

15

Fluctuating Performance Due to

Uncontrolled Resource Contention

Performance of art when co-running with different applications

on an Intel dual-core processor with a 4MB shared L2 cache

16

Fairness and Security Concerns

• Priority inversion

• Poor fairness among competing applications

• Information leakage at chip level

• Denial of service attack at chip level

5

17

Big Picture

A B D C

Select which applications run

together

X ………

…..

Control resource usage of co-running

applications

Resource-aware scheduling

[USENIX’10]

Page coloring

[Eurosys’09]

or Hardware throttling

[USENIX’09]

18

Existing Mechanism(I):

Software based Page Coloring

Thread A

Thread B

Shared Cache

Way-1 Way-n …………

Memory page A1

A2

A3

A4

A5

Thread A’s footprint • Classic technique to reduce cache misses,

now used by OS to manage cache

partitioning

• Partition cache at coarse

granularity

• No need for hardware

support

19

Drawbacks of Page Coloring • Expensive re-coloring cost

– Prohibitive in a dynamic environment where

frequent re-coloring may be necessary

• Complex memory management

– Introduces artificial memory pressure

Thread A

Thread B

Shared Cache

Way-1 Way-n …………

Memory page

A1

A2

A3

A4

A5

Thread A’s footprint

20

Toward Practical Page Coloring

• Hotness-based Page Coloring – Efficiently find a small group of hot pages

– Restrain page coloring or re-coloring to hot pages

– Pay less re-coloring overhead while achieving most of the cache partitioning benefit (separate competing applications’ most frequently accessed pages)

• Key challenge – Efficient way to track page hotness

6

21

Methods to Track Page Hotness

• Using page protection – Capture page accesses by triggering page faults

– Microseconds overhead per page fault

• Using access bits – A single bit stored in each Page Table Entry (PTE)

– Generally available on x86, automatically set by hardware upon page access

– Tens of cycles per page table entry check

– Recycle spare bits in PTE as hotness counter • Counter is aged to reflect recency and frequency

22

Sampling of Access Bits

• Decouple sampling frequency and window – Hotness sampling accuracy is determined by sampling time

window T

– Hotness sampling overhead is determined by sampling

frequency N

0

N 2N 3N 4N

T N+T 2N+T 3N+T 4N+T

Clear all access bits

Check all access bits

Time

Check all access bits

Clear all access bits

In our experiments, T = 2 milliseconds N = 100 or 10 milliseconds

23

Miss-Ratio-Curve Driven

Cache Partition Policy

Thread A’s Miss Ratio

Thread B’s Miss Ratio

Cache Allocation

Cache Allocation

0.5 0.7 0.3 0.2

0

0

Optimal partition point

Cache Size = ∑A,B Cache Allocation

4M

4M

System optimization metric =

24

Hot Page Coloring

• Budget control of page re-coloring overhead

– % of time slice, e.g. 5%

• Recolor from hottest until budget is reached

– Maintain a set of hotness bins during sampling

• bin[ i ][ j ] = # of pages in color i with normalized hotness in

range [ j, j+1]

– Given a budget K, K-th hottest page’s hotness value is

estimated in constant time by searching hotness bins

– Make sure hot pages are uniformly distributed among colors

7

25

Re-coloring Procedure

Budget = 3 pages

Cache share decrease

100

83

71

…

…

…

14

0

98 97 99

87

74

…

…

…

10

1

82

75

…

…

…

11

3

81

73

…

…

…

12

2

X

hotness counter value

Page sorted in

hotness ascending

order

Color Red Color Blue Color Green Color Gray 26

Performance Comparison

4 SPECcpu2k benchmarks (art,

equake, mcf, and twolf) are running

on 2 sibling cores (Intel core2duo)

that share a 4MB L2 cache.

27

Additional Benefit of

Hotness-based Page Coloring

• Page coloring introduces artificial memory pressure – App’s footprint is larger

than its entitled memory color pages, but system still has an abundance of memory pages

• Allow app to “steal”

other’s colors, but it preferentially copies cold pages to other’s memory colors

Thread A

Thread B

Cache Way-1 Way-n …………

Memory page A1

A2

A3

A4

A5

Thread A’s footprint

28

Big Picture

A B D C

Select which applications run

together

X ………

…..

Control resource usage of co-running

applications

Resource-aware scheduling

Page coloring or Hardware throttling

8

29

Hardware Execution Throttling • Instead of directly controlling resource allocation,

throttle the execution speed of application that overuses resource

• Available throttling knobs

– Duty-cycle modulation

– Frequency/voltage scaling

– Cache prefetchers

30

Comparing Hardware Execution

Throttling to Page Coloring

• Kernel code modification complexity – Code length: 40 lines in a single file, as a reference our

page coloring implementation takes 700+ lines of code

crossing 10+ files

• Runtime overhead of configuration – Less than 1 microseconds, as a reference re-coloring a

page takes 3 microseconds

31

Existing Mechanism(II):

Scheduling Quantum Adjustment

• Shorten the time slice of app that overuses cache

• May let core idle if there is no other active thread

available

Thread B

Thread A idle

Thread B

Thread A idle

Thread B

Thread A idle Core 0

Core 1

time

32

Drawback of Scheduling Quantum

Adjustment Coarse-grained control at scheduling quantum granularity may

result in fluctuating service delays for individual transactions

9

33

New Mechanism:

Hardware Execution Throttling [Usenix’09]

• Throttle the execution speed of app that overuses cache

– Duty cycle modulation

• CPU works only in duty cycles and stalls in non-duty cycles

• Different from Dynamic Voltage Frequency Scaling

– Per-core vs. per-processor control

– Thermal vs. power management

– Enable/disable cache prefetchers

• L1 prefetchers

– IP: keeps track of instruction pointer for load history

– DCU: when detecting multiple loads from the same line within a time limit,

prefetches the next line

• L2 prefetchers

– Adjacent line: Prefetches the adjacent line of required data

– Stream: looks at streams of data for regular patterns

34

Comparison of Hardware Execution

Throttling to other two mechanisms • Comparison to page coloring

– Little complexity to kernel • Code length: 40 lines in a single file, as a reference our page coloring implementation

takes 700+ lines of code crossing 10+ files

– Lightweight to configure • Read plus write register: duty-cycle 265 + 350 cycles, prefetcher 298 + 2065 cycles

• Less than 1 microseconds, as a reference re-coloring a page takes 3 microseconds

• Comparison to scheduling quantum adjustment

– More fine-grained controlling

Thread B

Core 0

Core 1

Thread A idle

Quantum adjustment Hardware execution

throttling

time

35

35

Fairness Comparison

• On average all three mechanisms are effective in improving fairness

• Case {swim, SPECweb} illustrates limitation of page coloring

• Unfairness factor: coefficient of variation (deviation-to-mean ratio, σ / μ) of co-running apps’ normalized performances (normalization base is the execution-time/throughput when the application monopolizes the whole chip)

36

36

Performance Comparison

• System efficiency: geometric mean of co-running apps’ normalized performances

• On average all three mechanisms achieve system efficiency comparable to default sharing

• Case where severe inter-

thread cache conflicts exist

favors segregation, e.g.

{swim, mcf}

• Case where well-interleaved

cache accesses exist favors

sharing, e.g. {mcf, mcf}

10

37

Policies for Hardware Throttling Based

Multicore Management

• User-defined service level agreements (SLAs) – Proportional progress among competing threads

• Unfairness metric: coefficient of variation of threads’ performance

– Quality of service guarantee for high-priority application(s)

• Key challenge

– Throttling configuration space grows exponentially as

the number of cores increases

– Quickly determining optimal or close to optimal

throttling configurations is challenging

38

Model-Driven Iterative Framework

• Customizable performance estimation model

• Reference configuration set and linear approximation

• Currently incorporates duty cycle modulation and

frequency/voltage scaling

• Iterative refinement

• Prediction accuracy gets improved over time as more

configurations are added into reference set

39

Iterative Refinement Patterns

40

Online Deployment:

Hill-Climbing Search Acceleration

• For a m-throttling-level n-core system, need to compute nm times to predict a “best” one

• Hill-climbing searches along the best child rather than all children

• Prunes the computation space to (m-1)n2

(X-

1,Y,Z,U)

(X,Y,Z,U)

(X,Y-

1,Z,U)

(X,Y,Z-

1,U)

(X,Y,Z,U-

1)

(X-1,Y-

1,Z,U) (X,Y-2,Z,U)

(X,Y-1,Z-

1,U)

(X,Y-1,Z,U-

1)

(X-1,Y-1,Z-

1,U) (X,Y-2,Z-1,U) (X,Y-1,Z-2,U)

(X,Y-1,Z-1,U-

1)

… … … …

11

41

Accuracy Evaluation

• Test platform

• A quad-core Nehalem processor with 8MB shared L3

cache

• Search space from full CPU speed (duty cycle level 8) to

half CPU speed (duty cycle level 4), so 369 configurations

for each test

• Benchmarks: SPECCPU2k • Set-1: {mesa, art, mcf, equake}

• Set-2: {swim, mgrid, mcf, equake}

• Set-3: {swim, art, equake, twolf}

• Set-4: {swim, applu, equake, twolf}

• Set-5: {swim, mgrid, art, equake}

42

Capability of Satisfying SLAs • Service Level Agreements (SLAs)

• Fairness-oriented: keep the unfairness below a threshold

• QoS-oriented: keep the QoS-core above a QoS

threshold

• 4 different unfairness/QoS thresholds for 5 sets

• Optimization goal: satisfy SLAs while optimizing

performance or power efficiency

# Passing tests Avg. num of

samples

Avg.

performance of

picked configs

that pass tests

Oracle 39/40 0 100%

Model 39/40 4.1 99.4%

Random 25/40 15 91.1% Recall the search space has 369 configurations

43

Accuracy of Performance Estimation

Error Rate = |Prediction – Actual| / Actual

44

Big Picture

A B D C

Select which applications run

together

X ………

…..

Control resource usage of co-running

applications

Resource-aware scheduling

Page coloring or Hardware throttling

12

45

Resource-aware Scheduling

• Scheduling decision could significantly affect performance

46

Similarity Grouping Scheduling

• Group applications with similar cache miss ratio on the same chip

– Separate high and low miss ratio apps on different chips

• Benefits – Mitigate cache thrashing effect

– Avoid over-saturating memory bandwidth

– Engage per-chip DVFS-based power savings

• A single voltage setting applies to all sibling cores on existing multicore chips

• High-miss-ratio chip runs at low frequency while low-miss-ratio chip runs at high frequency

47

Frequency-to-Performance Model

• Objective: explore power savings with bounded performance loss

• Assumptions – An application’s performance is linearly determined by

cache and memory access latencies

– Frequency scaling only affects on-chip accesses

– Miss ratio does not vary across frequencies

Normalized performance at frequency f = T(F) / T(f)

48

Model Accuracy

Error Rate = (Prediction - Actual) / Actual

13

49

Model-based Dynamic Frequency Setting

• Dynamically adjust CPU frequency based on

current running application’s behavior

– Collect cache miss ratio every 10 milliseconds

– Calculate an appropriate frequency setting based on

performance estimation model

• Guided by performance degradation threshold (e.g. 10%)

50

Model-based Dynamic Frequency Setting

51

Hardware Counter-based Power

Containers: An OS Resource

• Cross-core activity influence

• Online calibration with actual measurement

• Application-transparent online request context

tracking

52

Power Conditioning Using Power

Containers

14

53

Power Conditioning Achieved

Using Targeted Throttling

54

Ongoing Work

• Variation-directed information and management

– Using behavior fluctuation to trigger

monitoring

– Supporting fine-grain resource accounting

– Developing policies to reshape behavior for

high dependability and low jitter

– Request-level power attribution, modeling,

and management

55

See

http://www.cs.rochester.edu/research/cosyn

http://www.cs.rochester.edu/~sandhya

56

Arch/App: Shared Memory ++: DIMM [TRANSACT’06, ISCA’07, ISCA’08,ICS’09]

• Data Isolation (DI)

– Provide control over propagation of writes

– Buffer writes and allow group undo or propagation

Applications: Sand-boxing, transactional programming, speculation

• Memory Monitoring (MM)

– Monitor memory at summary or individual cache line level

Applications: Synchronization/event notification, reliability, security,

watchpoints/debugging

http://www.cs.rochester.edu/research/cosyn

15

57

Arch/App/OS:

Protection: Separation of Privileges

Reality: Today’s programs often consist of multiple modules

written by different programmers

Reliability and composability requires developing access and

interface conventions

58

Sentry: Light-Weight Auxiliary Memory

Access Control [ISCA’10]

• Access checks on an L1

miss

– Saves 90x energy

– Simplifies

implementation

• Metadata cache (M-cache)

accessed in parallel with

the L2 to speed up check

59

1 1 1 1

0 0 1 1

1 1 1 1

0 0 0 0

1 0

0 0 0

0

1

1

60

P0

P4

P11

The Indirection Problem

PACT 2008

A (M)

Home A

Load A

Data A

A(S)

1

DG A

Data A

2

A(S)

3 3 Load A

DG A

Data A

Data A

Longer distance means longer latency

16

61

Fine-Grain Data Sharing

Simultaneous access to the same data by more

than one core while the data still resides in

some L1 cache

Key Idea: Fine-grain sharing can be leveraged to localize

communication

62

Goal: Localize Shared Data Communication

P0

P4

P11

A (M)

Home A

Load A

Data A A(S)

A(S)

1 2

Load A Data A

Data availability at P0:

2 vs 10 physical hops

(whether P4 holds A in M or S)

63

Summary

• Harnessing 50B transistors requires a fresh look at conventional hardware-software boundaries and interfaces with support for

– Scalable coherence design

– Controlled data sharing via architectural support for

• Memory monitoring, isolation, and protection

– Controlled resource sharing via operating system-level policies for

• Performance isolation • We have examined coherence protocol additions to allow

– Fast event-based communication

– Fine-grain access control

– Programmable support for isolation

– Low-latency access for fine-grain data sharing

– Software to determine policy decisions in a flexible manner

A combined hardware/software approach to support for concurrency with improved performance and scalability