ICS 211 1

System Memory System Memory – Memory – Memory HierarchyHierarchy

Lesson 9aLesson 9a

Acknowledgement: Notes adopted from those of Acknowledgement: Notes adopted from those of Prof. Okello OdongoProf. Okello Odongo

ICS 211 2

OutlineOutline• Hierarchical organization: goals• Justification• Design requirements• Cached memory system• Performance factors• Cache organization• Cache management policies• Memory hierarchy examples

ICS 211 3

Hierarchical Memory Hierarchical Memory Organization: Organization: GoalsGoals

• The performance related considerations for system memory

Speed: access time should be as low as possible to maximize performance in terms of processing throughput.

Size (capacity): it is desirable that memory be of as large a capacity as possible so as to maximize the quantity of information that may be stored.

Cost: economic feasibility places an upper limit on system memory cost.

ICS 211 4

Hierarchical Memory Organization: Hierarchical Memory Organization: GoalsGoals

• The speed & size vs. cost dilemma.

Memory technologies offering faster speeds are invariably more expensive.

Realization of large system memory with fast access time and at an acceptable cost is a challenge.

ICS 211 5

Solution: The hierarchical organization of

system memory.

Hierarchical Memory Organization: Hierarchical Memory Organization: GoalsGoals

ICS 211 6

Secondary Storage (Removable)

Secondary (direct) online

Main Memory

Cache

Registers

ICS 211 7

DISK (magnetic disk)

MEMORY (DRAM)

CACHE(SRAM)

REGISTERSFaster

Slower Larger

Smaller Expensive

Cheap

ICS 211 8

Hierarchical Memory Organization:Hierarchical Memory Organization: Memory HierarchyMemory Hierarchy

•System memory is constructed as a hierarchy of memory components with differing speeds.

ICS 211 9

Memory HierarchyMemory Hierarchy

• Top of the hierarchy

– fastest memory components to hold frequently required information.

– only very limited capacities are economically feasible

ICS 211 10

• Towards the bottom

– progressively slower memory components

– to hold less frequently required information

– larger capacities are possible.

Memory HierarchyMemory Hierarchy

ICS 211 11

Typical Memory HierarchyTypical Memory Hierarchy

• level0 Processor Registers

• level1 Cache memory

• level2 Main memory

• level3 Secondary storage

• [levelN offline storage?]

ICS 211 12

Memory Hierarchy OperationMemory Hierarchy Operation

To access an item:

First look for the item at the highest level

If the item not found at level i, look for it at level i+1.

ICS 211 13

Hierarchical Memory Organization:Hierarchical Memory Organization: JustificationJustification

• Consider a two level system memory, M

– Two types of memory: M1 and M2

– Access times: Ta1 and Ta2 with Ta1 < Ta2.

– Cost per bit: C1 > C2

– Capacities: S1 < S2

What is the mean access time of the memory?

ICS 211 14

Is the mean access time

Close to Ta2 ?

Close to Ta1 ?

Hierarchical Memory Organization:Hierarchical Memory Organization: JustificationJustification

ICS 211 15

The mean access time, Tamean will depend on the % of access requests satisfied at M1

The higher the % of access requests satisfied at M1, the closer the mean access time to Ta1

[ see Graph]

Hierarchical Memory Organization: Hierarchical Memory Organization: JustificationJustification

ICS 211 16

20 40 60 80 100

Ta1

Ta2

(Ta1+ Ta2)/2

Mean access time, Tamean

% of accesses going to faster memory

ICS 211 17

The hierarchical design can only be justifiable if it is indeed the case that a very high % of access requests will be satisfied at level1!

Hierarchical Memory Organization:Hierarchical Memory Organization: JustificationJustification

ICS 211 18

The Principal of Locality: Memory references tend to be

clustered in certain regions in time space and ordering.

Hierarchical Memory Organization:Hierarchical Memory Organization: Justification Justification

ICS 211 19

Items recently referenced are likely to be referenced again (temporal locality).

There is a tendency for a process to access items whose addresses are near one another (spatial locality).

Hierarchical Memory Organization:Hierarchical Memory Organization: JustificationJustification

ICS 211 20

• Therefore:

by keeping the current “clusters” in the higher memory (e.g. level 1), performance close to that of M1, as measured by Tamean, is realizable.

Hierarchical Memory Organization:Hierarchical Memory Organization: JustificationJustification

ICS 211 21

Hierarchical Memory Design Hierarchical Memory Design RequirementsRequirements

• The information stored in an n level memory hierarchy, M1 to Mn, should satisfy the three important properties of

Inclusion – subset of info

Coherence – consistency of copies at different levels

Locality of reference

ICS 211 22

Inclusion

• Level n contains a copy of all information items in the system.

• During processing, a subset of items at level i may be copied into level i-1 .

• So: IM1 IM2 IM3 Imn = I

Hierarchical Memory Design RequirementsHierarchical Memory Design Requirements

ICS 211 23

Coherency

• Accesses to different copies of same information item in different memory levels must be consistent.

Hierarchical Memory Design RequirementsHierarchical Memory Design Requirements

ICS 211 24

Locality of Reference:

• For any leveli, requests for information at that level should be successful with high probability.

Hierarchical Memory Design RequirementsHierarchical Memory Design Requirements

ICS 211 25

Locality of Reference:

The highest level in the hierarchy at which a copy of a given item is located should be determined by the probability of that item being required in the next (few) accesses.

Hierarchical Memory Design RequirementsHierarchical Memory Design Requirements

ICS 211 26

Memory hierarchy on Contemporary Memory hierarchy on Contemporary SystemsSystems

Cache memory a small high speed memory to cache

subsets of main memory contents is implemented by the machine

architecture.

Virtual memory a large external memory a subset of which

is cached by main memory it is implemented by the operating system

with support from the machine architecture.

ICS 211 27

Cached MemoryCached Memory• The cache (C) is a small but very fast memory

component intended to store a subset of the contents of main memory (MM).

The items in the cache at any time should be those frequently accessed by the processor or likely to be requested in the immediate future.

The objective is to minimize the number of references to the main memory, which is much slower.

ICS 211 28

• Organization of Cached Memory System– CPU – MMU – Cache and cache controller – MM and memory controller

» [See Diagram]

• Procedure:– CPU generates a virtual memory address – MMU translates to a physical memory address – If the target address is currently in the cache, then

this is considered a cache hit and the cache is the target of the access.

– Else a cache miss and the target has to be brought into the cache from main memory.

Cached MemoryCached Memory

ICS 211 29

Cached Memory System OrganizationCached Memory System Organization

CPU

MMU

Cache

Main Memory

VA PA PA

I or D I or D

ICS 211 30

A cache hit a copy of the target is in cachethe access request is satisfied by the

cache

A cache miss there is no copy of the target in the

cachea copy of the target has to be brought

in from memory

Cached MemoryCached Memory

ICS 211 31

Cached Memory: Performance Cached Memory: Performance FactorsFactors

Performance factors

Hit rate/ratio: Probability of finding target item in cache – number of cache hits divided by the number of accesses OR fraction of memory accesses found in the upper level

Search speed: time taken to determine whether the item is present in cache

Miss penalty: delay incurred when an item is not found in cache OR time to replace a block in the upper level with the corresponding block from the lower level, plus the time to deliver this block to the processor

Hit time: time between sending address and getting data from cache (access time) OR time to access upper level of memory

Miss rate/ratio: number of cache misses divided by the number of accesses

Miss latency: time between sending the address and data returning from the next level cache/memory

ICS 211 32

Performance factors

Obtain an expression for Tamean in terms of hit ratio and miss penalty:

What factors determine the hit ratio?

What factors determine the miss penalty?

Cached Memory: Performance Cached Memory: Performance FactorsFactors

Average Memory Access Time = Hit Time + Miss Ratio * Miss Penalty