1

COMP 206:COMP 206:Computer Architecture and Computer Architecture and

ImplementationImplementation

Montek SinghMontek Singh

Mon., Nov. 25, 2002Mon., Nov. 25, 2002

Topic: Topic: Storage Systems (Disk Technology) Storage Systems (Disk Technology)

– contd.– contd.

2

Ensembles of DisksEnsembles of Disks Key ideaKey idea

Use collection of disk drives to improve characteristics Use collection of disk drives to improve characteristics of disk systems (storage capacity, bandwidth, etc.)of disk systems (storage capacity, bandwidth, etc.)

Used in mainframes for a long timeUsed in mainframes for a long time RAIDRAID

Redundant Array of Inexpensive Disks (original 1988 Redundant Array of Inexpensive Disks (original 1988 acronym)acronym)

Redundant Array of Independent Disks (redefined in Redundant Array of Independent Disks (redefined in 1992)1992)

3

Improving Bandwidth with Disk Improving Bandwidth with Disk ArraysArrays Arrays of independent disk drivesArrays of independent disk drives

Similar to high-order interleaving in main memoriesSimilar to high-order interleaving in main memories Each file assigned to different disk driveEach file assigned to different disk drive Simultaneous access to filesSimultaneous access to files Load balancing issuesLoad balancing issues

File striping/disk striping/disk interleavingFile striping/disk striping/disk interleaving Single file distributed across array of disksSingle file distributed across array of disks Similar to low-order interleaving in main memoriesSimilar to low-order interleaving in main memories Each logical I/O request corresponds to a data stripeEach logical I/O request corresponds to a data stripe

Data stripe divided into number of equal sized Data stripe divided into number of equal sized stripe unitsstripe unitsStripe units assigned to different disk unitsStripe units assigned to different disk units

Two kinds of striping depending on size of stripe unitTwo kinds of striping depending on size of stripe unitFine-grained striping: Stripe unit chosen to balance loadFine-grained striping: Stripe unit chosen to balance loadCoarse-grained striping: Larger stripe unitCoarse-grained striping: Larger stripe unit

4

Improving Availability with Disk Improving Availability with Disk ArraysArrays Failure rates of Disk Arrays can be quite high:Failure rates of Disk Arrays can be quite high:

MTTF of large-system disks approaches 1,000,000 MTTF of large-system disks approaches 1,000,000 hourshours

MTTF of PC-class disks approaches 150,000 hoursMTTF of PC-class disks approaches 150,000 hours But, array of 1,000 PC-class disks has MTTF of 150 But, array of 1,000 PC-class disks has MTTF of 150

hourshours

All schemes to cope with low MTTF aim to “fail All schemes to cope with low MTTF aim to “fail soft”soft” Operation should be able to continue while repair is Operation should be able to continue while repair is

mademade Always depends on some form of redundancyAlways depends on some form of redundancyIndependent addressing Fine-grained striping Coarse-grained striping

No redundancy Conventional RAID-0Replication (RAID-1) RAID-1Parity RAID-3 RAID-4, RAID-5ECC RAID-2

5

RAID-0RAID-0

Strip 12Strip 8Strip 4Strip 0

Strip 13Strip 9Strip 5Strip 1

Strip 14Strip 10Strip 6Strip 2

Strip 15Strip 11Strip 7Strip 3

Striped, non-redundantStriped, non-redundant Parallel access to multiple disksParallel access to multiple disks Excellent data transfer rate (for small strips)Excellent data transfer rate (for small strips) Excellent I/O request processing rate (for large strips)Excellent I/O request processing rate (for large strips)

Typically used for applications requiring high Typically used for applications requiring high performance for non-critical dataperformance for non-critical data

6

RAID-1RAID-1

Strip 3Strip 2Strip 1Strip 0

Strip 3Strip 2Strip 1Strip 0

Mirrored/replicated (most costly form of redundancy)Mirrored/replicated (most costly form of redundancy) I/O request rate: good for reads, fair for writesI/O request rate: good for reads, fair for writes Data transfer rate: good for reads; writes slightly slowerData transfer rate: good for reads; writes slightly slower

Read can be serviced by the disk with the shorter seek distanceRead can be serviced by the disk with the shorter seek distance Write must be handled by both disksWrite must be handled by both disks

Typically used in system drives and critical filesTypically used in system drives and critical files Banking, insurance dataBanking, insurance data Web (e-commerce) serversWeb (e-commerce) servers

7

Combining RAID-0 and RAID-1Combining RAID-0 and RAID-1

Strip 12Strip 8Strip 4Strip 0

Strip 13Strip 9Strip 5Strip 1

Strip 14Strip 10Strip 6Strip 2

Strip 15Strip 11Strip 7Strip 3

Strip 12Strip 8Strip 4Strip 0

Strip 13Strip 9Strip 5Strip 1

Strip 14Strip 10Strip 6Strip 2

Strip 15Strip 11Strip 7Strip 3

Can combine RAID-0 and RAID-1:Can combine RAID-0 and RAID-1: Mirrored stripes (RAID 0+1, or RAID 01)Mirrored stripes (RAID 0+1, or RAID 01)

Example: picture aboveExample: picture above Striped Mirrors (RAID 1+0, or RAID 10)Striped Mirrors (RAID 1+0, or RAID 10)

Data transfer rate: fair for reads and writesData transfer rate: fair for reads and writes Reliability: goodReliability: good Efficiency: poor (100% overhead in terms of disk Efficiency: poor (100% overhead in terms of disk

utilization)utilization)

8

RAID-3RAID-3

b0 b1 b2 b3 P(b)

Fine-grained (bit) interleaving with parityFine-grained (bit) interleaving with parity E.g., parity = sum modulo 2 (XOR) of all bitsE.g., parity = sum modulo 2 (XOR) of all bits

Disks are synchronized, parity computed by disk controllerDisks are synchronized, parity computed by disk controller When one disk fails… When one disk fails… (how do you know?)(how do you know?)

Data is recovered by subtracting all data in good disks from parity Data is recovered by subtracting all data in good disks from parity diskdisk

Recovering from failures takes longer than in mirroring, but failures Recovering from failures takes longer than in mirroring, but failures are rare, so is okayare rare, so is okay

Hot spares used to reduce vulnerability in reduced modeHot spares used to reduce vulnerability in reduced mode Performance:Performance:

Poor I/O request ratePoor I/O request rate Excellent data transfer rateExcellent data transfer rate Typically used in large I/O request size applications, such as Typically used in large I/O request size applications, such as

imaging or CADimaging or CAD

9

RAID-2RAID-2

b0 b1 b2 b3 f0(b) f1(b) f2(b)

Hamming codes capable of correcting two or more Hamming codes capable of correcting two or more erasureserasures E.g., single error-correcting, double error-detecting (SEC-DED)E.g., single error-correcting, double error-detecting (SEC-DED)

Problem with small writes (similar to DRAM cycle Problem with small writes (similar to DRAM cycle time/access time)time/access time)

Poor I/O request ratePoor I/O request rate Excellent data transfer rateExcellent data transfer rate

10

RAID-4RAID-4

Coarse-grained striping with parityCoarse-grained striping with parity Unlike RAID-3, not all disks need to be read on each readUnlike RAID-3, not all disks need to be read on each read

New parity computed by computing difference between old and new New parity computed by computing difference between old and new datadata

Drawback:Drawback: Like RAID-3, parity disk involved in every write; serializes small readsLike RAID-3, parity disk involved in every write; serializes small reads

I/O request rate: excellent for reads, fair for writesI/O request rate: excellent for reads, fair for writes Data transfer rate: fair for reads, fair for writesData transfer rate: fair for reads, fair for writes

Blk 12Blk 8Blk 4Blk 0

Blk 13Blk 9Blk 5Blk 1

Blk 14Blk 10Blk 6Blk 2

Blk 15Blk 11Blk 7Blk 3

P(12-15)P(8-11)P(4-7)P(0-3)

11

RAID-5RAID-5

Blk 12Blk 8Blk 4Blk 0

P(12-15)Blk 9Blk 5Blk 1

Blk 13P(8-11)

Blk 6Blk 2

Blk 14Blk 10P(4-7)Blk 3

Blk 15Blk 11Blk 7

P(0-3)

Key Idea: reduce load on parity disk Key Idea: reduce load on parity disk Block-interleaved Block-interleaved distributed paritydistributed parity Multiple writes can occur simultaneouslyMultiple writes can occur simultaneously Block 0 can be accessed in parallel with Block 5Block 0 can be accessed in parallel with Block 5

First needs disks 1 and 5; second needs disks 2 and 4First needs disks 1 and 5; second needs disks 2 and 4 I/O request rate: excellent for reads, good for writesI/O request rate: excellent for reads, good for writes Data transfer rate: good for reads, good for writesData transfer rate: good for reads, good for writes Typically used for high request rate, read-intensive data lookupTypically used for high request rate, read-intensive data lookup

12

Removable MediaRemovable Media Magnetic Tapes:Magnetic Tapes:

Have been used in computer systems as long as disksHave been used in computer systems as long as disks Similar magnetic technology as disksSimilar magnetic technology as disks

have followed similar density improvements as diskshave followed similar density improvements as disks Long strips wound on removable spoolsLong strips wound on removable spools Benefits: can be essentially of “unlimited” length; Benefits: can be essentially of “unlimited” length;

removableremovable Drawbacks: only offer sequential access; wear outDrawbacks: only offer sequential access; wear out Used for archival backups; but on the way outUsed for archival backups; but on the way out

Optical Disks: CD’s and DVD’sOptical Disks: CD’s and DVD’s CD-ROM/DVD-ROM: removable, inexpensive, random accessCD-ROM/DVD-ROM: removable, inexpensive, random access CD-R/RW, DVD-R/RAM: writable/re-writableCD-R/RW, DVD-R/RAM: writable/re-writable Becoming immensely popular: replacing floppy drives, tape Becoming immensely popular: replacing floppy drives, tape

drivesdrives Flash: flash cards, memory sticks, digital filmFlash: flash cards, memory sticks, digital film Microdrives: small portable hard disks (up to 1GB)Microdrives: small portable hard disks (up to 1GB)

13

Interfacing I/O Subsystem to CPUInterfacing I/O Subsystem to CPUWhere do we connect I/O Where do we connect I/O

devices?devices? Cache?Cache? Memory? Memory?

Low-cost systems: I/O bus = Low-cost systems: I/O bus = memory bus!memory bus!

– CPU and I/O compete for busCPU and I/O compete for bus

How does CPU address I/O How does CPU address I/O devs.?devs.? Memory-mapped I/OMemory-mapped I/O

Portions of address space Portions of address space assigned to I/O devicesassigned to I/O devices

I/O opcodes (not very popular)I/O opcodes (not very popular)

How does CPU synch with I/O?How does CPU synch with I/O? Polling vs. interrupt-drivenPolling vs. interrupt-driven Real-time systems: hybridReal-time systems: hybrid

Clock interrupts periodicallyClock interrupts periodically CPU polls during interruptCPU polls during interrupt

14

Does I/O Performance Matter?Does I/O Performance Matter? The “No” Argument:The “No” Argument:

““There is always another process to run while one process There is always another process to run while one process waits for I/O to complete”waits for I/O to complete”

Counter-arguments:Counter-arguments: Above argument applies only if throughput is sole goalAbove argument applies only if throughput is sole goal Interactive software and personal computers lay more Interactive software and personal computers lay more

emphasis on response time (=latency), not throughputemphasis on response time (=latency), not throughput Interactive multimedia, transaction processingInteractive multimedia, transaction processing

Process switching actually increases I/O (paging traffic)Process switching actually increases I/O (paging traffic) Desktop/Mobile computing: only one person/computer Desktop/Mobile computing: only one person/computer

fewer processes than in time-sharingfewer processes than in time-sharing Amdahl’s Law: benefits of making CPU faster taper off if Amdahl’s Law: benefits of making CPU faster taper off if

I/O becomes the bottleneckI/O becomes the bottleneck

15

Does CPU Performance Matter?Does CPU Performance Matter?Consequences of Moore’s Law:Consequences of Moore’s Law:

Large, fast CPU’sLarge, fast CPU’s Small, cheap CPU’sSmall, cheap CPU’s

Main goal: keeping I/O devices busy, not keeping CPU busyMain goal: keeping I/O devices busy, not keeping CPU busy– bulk of the hardware cost is in I/O peripherals, not CPUbulk of the hardware cost is in I/O peripherals, not CPU

Shift in focus:Shift in focus:From computation to communicationFrom computation to communicationReflected in change of terminology:Reflected in change of terminology:

– 1960’s-80’s: Computing Revolution1960’s-80’s: Computing Revolution– 1990’s-present: Information Age1990’s-present: Information Age

16

Does Performance Matter?Does Performance Matter?Performance is not the problem it once was!Performance is not the problem it once was!

15 years of doubling CPU speed every 18 months 15 years of doubling CPU speed every 18 months (x1000)(x1000)

Disks also have steadily become bigger and fasterDisks also have steadily become bigger and faster

Most people would prefer more reliability than Most people would prefer more reliability than speed:speed: Today’s speeds come at a cost: frequent crashesToday’s speeds come at a cost: frequent crashes Program crashes Program crashes frustration frustration Disk crashes Disk crashes hysteria hysteria

Reliability, availability, dependability, etc. are the key Reliability, availability, dependability, etc. are the key termsterms

Client-server model of computing has made reliability the Client-server model of computing has made reliability the key criterion for evaluationkey criterion for evaluation

– E.g., UNC-COMPSCI IMAP server: 300 MHz or something, E.g., UNC-COMPSCI IMAP server: 300 MHz or something, 99.9… % uptime99.9… % uptime