1

Cache Memory

2

Outline

• Cache mountain

• Matrix multiplication

• Suggested Reading: 6.6, 6.7

3

6.6 Putting it Together: The Impact of Caches on Program Performance

6.6.1 The Memory Mountain

4

The Memory Mountain P512

• Read throughput (read bandwidth)– The rate that a program reads data from the

memory system

• Memory mountain– A two-dimensional function of read bandwidth

versus temporal and spatial locality– Characterizes the capabilities of the memory

system for each computer

5

Memory mountain main routineFigure 6.41 P513

/* mountain.c - Generate the memory mountain. */

#define MINBYTES (1 << 10) /* Working set size ranges from 1 KB */

#define MAXBYTES (1 << 23) /* ... up to 8 MB */

#define MAXSTRIDE 16 /* Strides range from 1 to 16 */

#define MAXELEMS MAXBYTES/sizeof(int)

int data[MAXELEMS]; /* The array we'll be traversing */

6

Memory mountain main routine

int main()

{

int size; /* Working set size (in bytes) */

int stride; /* Stride (in array elements) */

double Mhz; /* Clock frequency */

init_data(data, MAXELEMS); /* Initialize each element in data to 1 */

Mhz = mhz(0); /* Estimate the clock frequency */

7

Memory mountain main routine

for (size = MAXBYTES; size >= MINBYTES; size >>= 1) {

for (stride = 1; stride <= MAXSTRIDE; stride++)

printf("%.1f\t", run(size, stride, Mhz));

printf("\n");

}

exit(0);

}

8

Memory mountain test functionFigure 6.40 P512

/* The test function */

void test (int elems, int stride) {

int i, result = 0;

volatile int sink;

for (i = 0; i < elems; i += stride)

result += data[i];

sink = result; /* So compiler doesn't optimize away the loop */

}

9

Memory mountain test function

/* Run test (elems, stride) and return read throughput (MB/s) */

double run (int size, int stride, double Mhz)

{

double cycles;

int elems = size / sizeof(int);

test (elems, stride); /* warm up the cache */

cycles = fcyc2(test, elems, stride, 0); /* call test (elems,stride) */

return (size / stride) / (cycles / Mhz); /* convert cycles to MB/s */

}

10

The Memory Mountain

• Data

– Size

• MAXBYTES(8M) bytes or MAXELEMS(2M) words

– Partially accessed

• Working set: from 8MB to 1KB

• Stride: from 1 to 16

11

The Memory MountainFigure 6.42 P514

s1

s3

s5

s7

s9

s11

s13

s15

8m 2

m 51

2k

12

8k 32

k 8k 2

k

0

200

400

600

800

1000

1200

rea

d t

hro

ug

hp

ut

(MB

/s)

stride (words) working set size (bytes)

Pentium III Xeon550 MHz16 KB on-chip L1 d-cache16 KB on-chip L1 i-cache512 KB off-chip unifiedL2 cache

Ridges ofTemporalLocality

L1

L2

mem

Slopes ofSpatialLocality

xe

12

Ridges of temporal locality

• Slice through the memory mountain with

stride=1

– illuminates read throughputs of different

caches and memory

Ridges: 山脊

13

Ridges of temporal localityFigure 6.43 P515

0

200

400

600

800

1000

12008

m

4m

2m

10

24

k

51

2k

25

6k

12

8k

64

k

32

k

16

k

8k

4k

2k

1k

working set size (bytes)

rea

d t

hro

ug

pu

t (M

B/s

)

L1 cacheregion

L2 cacheregion

main memoryregion

14

A slope of spatial locality

• Slice through memory mountain with

size=256KB

– shows cache block size.

15

A slope of spatial localityFigure 6.44 P516

0

100

200

300

400

500

600

700

800

s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14 s15 s16

stride (words)

rea

d t

hro

ug

hp

ut

(MB

/s)

one access per cache line

16

6.6 Putting it Together: The Impact of Caches on Program Performance

6.6.2 Rearranging Loops to Increase Spatial Locality

17

2221

1211

2221

1211

2221

1211

bb

bb

aa

aa

cc

cc

2222122122

2122112121

2212121112

2112111111

babac

babac

babac

babac

Matrix Multiplication P517

18

Matrix Multiplication ImplementationFigure 6.45 (a) P518

/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { c[i][j] = 0.0; for (k=0; k<n; k++) c[i][j] += a[i][k] * b[k][j]; }} O(n3)adds and multipliesEach n2 elements of A and B is read n times

/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { c[i][j] = 0.0; for (k=0; k<n; k++) c[i][j] += a[i][k] * b[k][j]; }} O(n3)adds and multipliesEach n2 elements of A and B is read n times

19

Matrix Multiplication P517

• Assumptions:– Each array is an nn array of double, with size 8

– There is a single cache with a 32-byte block size ( B=32 )

– The array size n is so large that a single matrix row does not fit in the L1 cache

– The compiler stores local variables in registers, and thus references to local variables inside loops do not require any load and store instructions.

20

/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}

/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}

Variable sumheld in register

Matrix MultiplicationFigure 6.45 (a) P518

21

/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}

/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}

A B C

(i,*)

(*,j)(i,j)

Inner loop:

Column-wise

Row-wise Fixed

• Misses per Inner Loop Iteration:A B C

0.25 1.0 0.0

Matrix multiplication (ijk)

Figure 6.46 P519

1) (AB)

22

/* jik */for (j=0; j<n; j++) { for (i=0; i<n; i++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum }}

/* jik */for (j=0; j<n; j++) { for (i=0; i<n; i++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum }}

A B C

(i,*)

(*,j)(i,j)

Inner loop:

Row-wise Column-wise

Fixed• Misses per Inner Loop Iteration:

A B C0.25 1.0 0.0

Matrix multiplication (jik)Figure 6.45 (b) P518

Figure 6.46 P519

1) (AB)

23

/* kij */for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}

A B C

(i,*)(i,k) (k,*)

Inner loop:

Row-wise Row-wiseFixed

• Misses per Inner Loop Iteration:

A B C0.0 0.25 0.25

Matrix multiplication (kij)Figure 6.45 (e) P518

Figure 6.46 P519

3) (BC)

24

/* ikj */for (i=0; i<n; i++) { for (k=0; k<n; k++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}

A B C

(i,*)(i,k) (k,*)

Inner loop:

Row-wise Row-wiseFixed

• Misses per Inner Loop Iteration:A B C

0.0 0.25 0.25

Matrix multiplication (ikj)Figure 6.45 (f) P518

Figure 6.46 P519

3) (BC)

25

/* jki */for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}

A B C

(*,j)(k,j)

Inner loop:

(*,k)

Column -wise

Column-wise

Fixed• Misses per Inner Loop

Iteration:A B C

1.0 0.0 1.0

Matrix multiplication (jki)Figure 6.45 (c) P518

Figure 6.46 P519

2) (AC)

26

/* kji */for (k=0; k<n; k++) { for (j=0; j<n; j++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}

A B C

(*,j)(k,j)

Inner loop:

(*,k)

FixedColumn-wise

Column-wise

• Misses per Inner Loop Iteration:

A B C1.0 0.0 1.0

Matrix multiplication (kji)Figure 6.45 (d) P518

Figure 6.46 P519

2) (AC)

27

Pentium matrix multiply performanceFigure 6.47 (d) P519

0

10

20

30

40

50

60

25 50 75 100 125 150 175 200 225 250 275 300 325 350 375 400

Array size (n)

Cyc

les

/ite

rati

on

kjijkikijikjjikijk

2) (AC)

3) (BC)

1) (AB)

2)

3)

1)

28

Pentium matrix multiply performance

• Notice that miss rates are helpful but not perfect predictors.

– Code scheduling matters, too.

29

for (i=0; i<n; i++) {

for (j=0; j<n; j++) {

sum = 0.0;

for (k=0; k<n; k++)

sum += a[i][k] * b[k][j];

c[i][j] = sum;

}

}

ijk (& jik): • 2 loads, 0 stores• misses/iter = 1.25

for (k=0; k<n; k++) {

for (i=0; i<n; i++) {

r = a[i][k];

for (j=0; j<n; j++)

c[i][j] += r * b[k][j];

}

}

for (j=0; j<n; j++) {

for (k=0; k<n; k++) {

r = b[k][j];

for (i=0; i<n; i++)

c[i][j] += a[i][k] * r;

}

}

kij (& ikj): • 2 loads, 1 store• misses/iter = 0.5

jki (& kji): • 2 loads, 1 store• misses/iter = 2.0

Summary of matrix multiplication

1) (AB) 3) (BC) 2) (AC)

30

6.6 Putting it Together: The Impact of Caches on Program Performance

6.6.3 Using Blocking to Increase Temporal Locality

31

Improving temporal locality by blocking P520

• Example: Blocked matrix multiplication– “block” (in this context) does not mean “cache

block”.

– Instead, it mean a sub-block within the matrix.

– Example: N = 8; sub-block size = 4

32

Improving temporal locality by blocking

C11 = A11B11 + A12B21 C12 = A11B12 + A12B22

C21 = A21B11 + A22B21 C22 = A21B12 + A22B22

A11 A12

A21 A22

B11 B12

B21 B22X =

C11 C12

C21 C22

Key idea: Sub-blocks (i.e., Axy) can be treated just like scalars.

33

for (jj=0; jj<n; jj+=bsize) { for (i=0; i<n; i++) for (j=jj; j < min(jj+bsize,n); j++) c[i][j] = 0.0; for (kk=0; kk<n; kk+=bsize) { for (i=0; i<n; i++) { for (j=jj; j < min(jj+bsize,n); j++) { sum = 0.0 for (k=kk; k < min(kk+bsize,n); k++) { sum += a[i][k] * b[k][j]; } c[i][j] += sum; } } }}

Blocked matrix multiply (bijk)Figure 6.48 P521

34

Blocked matrix multiply analysis

• Innermost loop pair multiplies a 1 X bsize

sliver of A by a bsize X bsize block of B and

accumulates into 1 X bsize sliver of C

– Loop over i steps through n row slivers of A & C,

using same B

Sliver: 长条

35

A B C

block reused n times in succession

row sliver accessedbsize times

Update successiveelements of sliver

i ikk

kk jjjj

for (i=0; i<n; i++) { for (j=jj; j < min(jj+bsize,n); j++) { sum = 0.0 for (k=kk; k < min(kk+bsize,n); k++) { sum += a[i][k] * b[k][j]; } c[i][j] += sum; }

InnermostLoop Pair

Blocked matrix multiply analysis

Figure 6.49 P522

36

Pentium blocked matrix multiply performanceFigure 6.50 P523

0

10

20

30

40

50

60

Array size (n)

Cy

cle

s/i

tera

tio

n

kji

jki

kij

ikj

jik

ijk

bijk (bsize = 25)

bikj (bsize = 25)

2)3)

1)

37

6.7 Putting it Together: Exploring Locality in Your Programs

38

Techniques P523

• Focus your attention on the inner loops

• Try to maximize the spatial locality in your

programs by reading data objects sequentially, in

the order they are stored in memory

• Try to maximize the temporal locality in your

programs by using a data object as often as

possible once it has been read from memory

• Miss rates, the number of memory accesses