Speculate Ambitiously To speculate ambitiously requires three capabilities: • The ability of the complier to find instructions that, with the possible use of register renaming, can be speculatively moved and not affect the program data flow. • The ability to ignore exceptions in speculated instructions, until we know that such exceptions should really occur. • The ability to speculatively interchange loads and stores, or stores and stores, which may have address conflicts. • The first of these is a complier capability, while the last two require hardware support .
Transcript
Speculate AmbitiouslyTo speculate ambitiously requires three capabilities:• The ability of the complier to find instructions that, with the
possible use of register renaming, can be speculatively moved and not affect the program data flow.
• The ability to ignore exceptions in speculated instructions, until we know that such exceptions should really occur.
• The ability to speculatively interchange loads and stores, or stores and stores, which may have address conflicts.
• The first of these is a complier capability, while the last two require hardware support .
Hardware Support for Preserving Exception Behavior
• To speculate ambitiously, we must be able to move any type
of instruction and still preserve its exception behavior.
• The key to being able to do this is to observe that the results
of a speculated sequence that is mispredicted will not be used
in the final computation , and such a speculated instruction
should not cause an exception.
Investigated for supporting more ambitious speculation
• There are four methods that have been investigated for supporting more ambitious speculation without introducing erroneous behavior:– The hardware and operating system cooperatively ignore
exceptions for speculative instructions. This approach preserves exception behavior for correct programs, but not for incorrect ones. This approach may viewed as unacceptable for some programs, but it has been used, under program control , as a “fast mode” in several processors.
– Speculative instructions that never raise exceptions are used, and checks are introduced to determine when an exception should occur.
Investigated for supporting more ambitious speculation
– A set of status bits, called poison bits, are attached to the
result registers written by speculated instructions when the
instructions cause exceptions. The poison bits cause a fault
when a normal instructions attempts to use the register.
– A mechanism is provided to indicate that an instruction is
speculative , and the hardware buffers the instruction result
until it is certain that the instruction is no longer speculative.
Schemes• To know the schemes, we need to distinguish between
exceptions that indicate a program error and would normally cause termination, such as a memory protection violation, and those that are handled and normally resumed, such as a page default.
• Exception that can be resumed can be accepted and processed for speculative instructions just as if they were normal instructions.
• If the speculative instruction should not have been executed, handling the unneeded exception may have some negative performance effects, but it cannot cause incorrect execution.
Schemes..• The cost of these exceptions may be high, however, and some processors
use hardware support to avoid taking such exceptions, just as processors with hardware speculation may take some exceptions in speculative mode, while avoiding others until an instruction is known not to be speculative.
• Exceptions that indicate a program error should not occur in correct programs, and the result of a program that gets such a exception is not well defined, except perhaps when the program is running in a debugging mode.
• If such exceptions arise in speculated instructions, we cannot take the exception until we know that the instruction is no longer speculative.
Schemes …• In the simplest method for preserving exceptions, the
hardware and the operating system handle all resumable exceptions when the exception occurs and simply return an undefined value for any exception that would cause termination.
• If the instruction generating the terminating exception was not speculative , then the program is in error.
• Note the instead of terminating the program, the program is allowed to continue, although it will almost certainly generate incorrect results.
• If the instruction generating the terminating exception is speculative, then the program may be correct and the speculative result will simply be unused.
Schemes …• Thus, returning an undefined value for the instruction cannot
be harmful.• This scheme can never cause a correct program to fail, no
matter how much speculation is done.• An incorrect program, which formerly might have received a
terminating exception, will get an incorrect result.• This a acceptable for some programs, assuming the complier
can also generate a normal version of the program, which does not speculate and can receive a terminating exception.
Schemes …• In such a scheme, it is not necessary to know that an
instruction is speculative. Indeed, it is helpful only when a program is in error and receives a terminating exception on a normal instruction; in such cases, if the instruction were not marked as speculative, the program could be terminated.
• A second approach to preserving exception behavior when speculating introduces speculative versions of instructions that do not generate terminating exceptions and instructions to check for such exceptions. This combines preserves the exception behavior exactly.
Schemes …• A third approach for preserving exception tracks exceptions as they occur
but postpones of the exception, although not in a completely precise fashion.
• The scheme is simple: a poison bit is added to every register, and another bit is added to every instruction to indicate whether a speculative instructions results in a terminating exception; all other exceptions are handled immediately.
• If a speculative instruction uses a register with a poison bit turned on, the destination register of the instruction simply has its poison bit turned on,
• If a normal instruction attempts to use a register source with its poison bit turned on, the instruction causes a fault .
• In this way, any program that would have generated an exception still generates one, albeit at the first instance where a result is used by an instruction that is not speculative.
Schemes …• Since poison bits exist only on register values and not memory values,
stores are never speculative and thus trap if either operand is “poison”.• One complication thus must be overcome is how the OS saves the user
registers on a context switch if the poison bit is set.• A special instruction is needed to save and reset the state of the poison
bits to the avoid this problem.• The fourth and final approach listed in earlier relies on a hardware
mechanism that operates like a reorder buffer.• In such an approach, instructions are marked by the complier as
speculative and include an indicator of how many branches the instruction was speculatively moved across and what branch action (taken/not taken) the complier assumed.
• The last piece of information basically tells the hardware the location of the code block where the speculated instruction originally was .
Schemes …• In practice, most of the benefit of speculative instruction is
marked by a sentinel, which tells the hardware that the earlier speculative instruction is no longer speculative and values may be committed.
• All instructions are placed in a reorder buffer when issued and are forced to commit in order, as in hardware speculation approach.(Notice , though, that no actual speculative branch prediction or dynamic scheduling occurs).
• The reorder buffer tracks when instructions are ready to commit and delays the “write-back” portion of any speculative instruction.
Schemes …• Speculative instructions are not allowed to commit until the
branches that have been speculatively moved over are also ready to commit, or ,alternatively until the corresponding sentinel is reached.
• At that point, we know whether the speculated instructions should have been executed or not.
• If it should have been executed and it generated a terminating exception, then we know that the program should be terminated.
• If the instruction should not have been executed then the can be ignored.
Example 1• Consider the code fragment from if-the-else statement of the
form:If (A==0) A=B; else A=a+4;Where A is at 0(R3) and B is at 0(R2). Assume
the then clause is almost always executed. Compile the code using complier-based speculation. Assume R14 is unused and available.
L1: DADDI R1,R1, #4 ; else clauseL2: SD R0,0(R3) ; store A
Example 3• Consider the code fragment and show how it would be
complied with speculative instructions and poison bits. Show where an exception for the speculative memory reference would be recognized. Assume R14 is unused and available.
L1: DADDI R1,R1, #4 ; else clauseL2: SD R0,0(R3) ; store A
Hardware support for memory Reference Speculation
• Moving loads across stores is usually done when the complier is certain the address do not conflict.
• A special instruction to check for address conflicts can be included in the architecture,
• The special instruction is left at the original location of the load instruction (and acts like a guardian), and the load is moved up across or more stores.
• When a speculated load is executed, the hardware saves the address of the accessed memory location.
• If a subsequent store the address of the location before the check instruction, then the speculation has failed.
Hardware support for memory Reference Speculation …
• If the location has not been touched, then the speculation is successful.
• Speculation failure can be handled in two ways.– If only the load instruction was speculated, then it suffices
to redo the load at the point of the check instruction (which could supply the target register in addition to the memory address).
– If additional instructions that depend on the load were also speculated , then a fix-up sequence that reexecutes all the speculation instructions starting with the load is needed.
– In this case, the check instruction specifies the address where the fix-up code is loaded.
Sections To be covered from Chapter 4
• Section 4.5• Software versus Hardware based scheduling
Memory Hierarchy
Memory Hierarchy & Cache Memory
Entry Quiz1. Primary cache or level 1 cache is
implemented in a separate chip
a. True b. False
Entry Quiz2. SRAM is implemented using
a. Flip-Flopb. Magnetic corec. Capacitord. Non-volatile Technology
Entry Quiz3. Main memory (200 ns) is slower compared
register (0.2 ns) by an order of
a. 3b. 4c. 1000d. 10000
Entry Quiz4. Virtual Memory is
a. Same as cachingb. Same as associative memoryc. Different from cachingd. Same as disk memory
Entry Quiz5. Cache Miss occurs when the
a. Required instruction is not found in the cacheb. Required data is not found in the cachec. Required instruction or data is not found in the
cached. Required instruction or data is not found in the
main memorye. For all of the above conditions
Module Objective• To understand
1. Memory requirements of different computers
2. Memory hierarchy and the motivation behind it 3. Moore’s Law 4. Principles of Locality 5. Cache Memory and its implementation6. Cache Performance7. Terms: Cache, Cache Miss, Cache Hit, Latency,
Bandwidth, SRAM, DRAM, by an order of, Direct Mapping, Associative Mapping, Set Mapping, Write Through, Write Allocated, Write Back, Dirty Bit, and Valid Bit
Memory Requirements• In general we would like to have
– Faster Memory (lower access time or latency)
– Larger (capacity and bandwidth) Memory– Simpler Memory
Memory Requirements - Server, Desktop, and Embedded devices
= IC X Read Access X Read Miss Rate X Read Penalty +
Instruction
IC X Write Access X Write Miss Rate X Write Penalty
Instruction
Example 1• Assume we have a computer where Clock cycles Per
Instruction (CPI) is 1.0 when all memory accesses are cache hits. The only data accesses are loads and stores and these total 50% of the instructions. If the miss penalty is 25 clock cycles and miss rate is 2%, how much faster would the computer be if all instructions were cache hits?
Example 1 …1. Execution time for the program in a
computer with 100 % hits
= (Useful CPU Cycles + Stalled CPU Cycles) x Cycle Time
= (IC X CPI + 0) X Cycle Time= IC X 1.0 X Cycle Time
Example 1 …2. Stall Cycle when there is one or more Cache
Misses: = IC X Memory Access X Miss Rate X Miss Penalty
Instruction
= IC X (1 + 0.5) X 0.02 X 25 Cycle Time = IC X 0.75 Cycle Time
3. Total Execution Time: = (IC X 1 + IC X 0.75) Clock Cycles = 1.75 IC Clock Cycles
Example 1 …
4. Ratios = 1.75 IC Clock Cycles / IC X 1.0 X Cycle Time
= 1.75
Execution time is 1.75 faster in a computer with no Misses.
Example 2
• Assume we have a computer where Clock cycles Per Instruction (CPI) is 1.0 when all memory accesses are cache hits. The only data accesses are loads and stores and these total 50% of the instructions. Miss penalty for read miss is 25 clock cycles and miss penalty for write miss is 50 clock cycles. Miss rate is 2% and out of this 80% is Read miss. How much faster would the computer be if all instructions were cache hits?
Cache Operation
Word Length(K Words)
...
Block
Block(K Words)
0123
2N - 1
Line Number
0
1
2
C= 16
Block
Block Length(K Words)
Tag
...
Cache
Main Memory
CPU
Elements of Cache Design• Cache Size• Block Size• Mapping Function• Replacement Algorithm• Write Policy• Write Miss• Number of caches• Split versus Unified/Mixed Cache
Mapping FunctionsTag
CacheMain Memory
Tag Line WordBlock 0
Block 1
Block n-1
Block n
Line 0
Line 1
Line 2
Line 3 (m)
3.Compare
1. Select
+4. Hit
Miss
To main memory
From CPUAddress
2. Copy
5. Load
Mapping Function• Direct
– Line value in address uniquely points to a line in cache. 1 tag Comparison
• Set Associative – Line value in address points to a set of lines in cache (typically
2/4/8, so 2/4/8 tag comparisons). This is known as 2/4/8 way Set Associative.
• Associative– Line value is always 0. This means Line points to all the lines
in cache (4 (m) tag Comparisons)– Uses Content Addressable Memory (CAM) for comparison.– Needs non-trivial replacement algorithm
Tag
Cache
Tag Line WordBlock 0
Block 1
Block 126
Block 127
Line 0
Line 1
Line 2
Line 3
3.Compare
1. Select
+4. Hit
Miss
To main memory
From CPUAddress
2. Copy
5. Load
0
1
23
Memory
508
509
510511
4
5
67
5 2 2 3
504
505
506507
SET 1
SET 0
Mapping Function Comparison
Cache Type Hit Ratio Search Speed
Direct Mapping Good Best
Fully Associative Best Moderate
N-Way Set AssociativeVery Good, Better as N Increases
Good, Worse as N Increases
Replacement Algorithm• Least Recently Used• First In First Out• Least Frequently Used• Random
Write Policy• Write-through
– Information written to cache and the memory
• Write back– Information written only to cache. Content of the
cache is written to the main memory only when this cache block is replaced or the program terminates.
• Dirty bit is used to indicate that a cache block needs “Write back”
Write Miss• Write-Allocate• No Write-Allocate
Number of Caches• Primary Cache (CPU)• Secondary Cache (SRAM)• L3 Cache (Cheaper SRAM)
Split versus Unified/Mixed Cache
Size (KB)Instruction Cache Data Cache Unified Cache
Exit Quiz1. In memory hierarchy top layer is occupied by
the
a. Fastest and the most expensive memoryb. Slowest and the most expensive memoryc. High band width and the fastest memoryd. Slowest and the least expensive memory
Exit Quiz2. The memory technology suitable for L2 cache
is
a. EPROMb. DRAMc. SRAMd. Flash
Exit Quiz3. Server’s memory requirements are different
from desktop’s memory requirementsa. Trueb. False
Exit Quiz4. Hit Rate is (1 – Miss Rate)
a. Trueb. False
5. Gordon Moore’s law states that the transistor density
a. Triples every yearb. Doubles every two yearsc. Doubles every 1.5 yearsd. Doubles every year
Improving Cache Performance• Two options
– By Reducing Miss Penalty– By Reducing Miss Rate
Reducing Cache Miss Penalty1. Multilevel caches2. Critical word first & early restart3. Priority for Read misses over Write4. Merging Write Buffers5. Victim Cache
Reducing Miss Penalty - Multilevel caches (1)
Cache1 CPU MainMemory Cache 1CPU Main
Memory
Faster but smaller Slower but larger
CPU MainMemory
Faster and Larger
Cache 1 Cache 2
2 ns, 16K 10 ns, 512K
? ns, 528 K
0.020.04
0.040.02
100 ns, 128M100 ns, 128M
100 ns, 128M
Reducing Miss Rate & Global Miss Rate (1)
CPU MainMemory
Faster and Larger
Cache 1 Cache 2
Average memory access timeeff = Hit TimeL1 + Miss RateL1 x Miss
PenaltyL1
Miss PenaltyL1 = Hit TimeL2 + Miss RateL2 x Miss PenaltyL2
Global Miss Rate = Miss RateL1 x Miss RateL2
0.050.02
2 ns, 16K 10 ns, 512K
100 ns, 128M2.6 ns, 528K
• Critical word first • Early restart • Lets discuss the following:
– How is Miss Penalty improved?– Under what conditions this is useful?– Is it worth going through all the trouble?
Reducing Miss Penalty – Critical Word First and Early Restart (2)
Reducing Miss Penalty – Priority To Read Miss (3)
• Read Miss takes priority• Write to Memory is put on hold• Lets discuss the following:
– Is there a problem?– How is Miss Penalty improved? – How this is done in Write-through?– How this is done in Write-back?– Under what conditions this is useful?– Is it worth going through all the trouble?
Reducing Miss Penalty – Merging Write Buffer (4)
• Write-through is sent to a write buffer• Buffer & Write possible options:
1. Buffer is empty – write address and data to buffer2. Buffer is not full – write or merge data3. Buffer is full – Stall until buffer is written to memory. – Note: Block write to memory is more efficient then multiple writes.
• Lets Discuss the following:– How many different ways do we optimize Miss Penalty in this
scheme?– Explain the Fig 5.12 in the book– When there is no block write (like I/O device), no merging
Reducing Miss Penalty – Fully Associative Victim Cache (5)
• Key word is Recycle• Another implicit key word is complex
(hardware)• Aimed at handling conflict misses• 1 to 6 victim caches are ideal
Reducing Misses• Misses due to 3Cs: Compulsory, Capacity, and
