Post on 19-Jan-2016
description
transcript
12004 Morgan Kaufmann Publishers
Chapters 8
Storage, Networks, and Other Peripherals
22004 Morgan Kaufmann Publishers
32004 Morgan Kaufmann Publishers
Outline
• 8.1 Introduction
• 8.2 Disk Storage and Dependability
• 8.3 Networks
• 8.4 Buses and Other Connections between Processors, Memory, and I/O Devices
• 8.5 Interfacing I/O Devices to the Processor, Memory, and Operating System
• 8.6 I/O Performance Measures: Examples from Disk and File Systems
• 8.7 Designing an I/O System
• 8.8 Real Stuff: A Digital Camera
• 8.9 Fallacies and Pitfalls
• 8.10 Concluding Remarks
• 8.11 Historical Perspective and Further Reading
42004 Morgan Kaufmann Publishers
8.1 Introduction
52004 Morgan Kaufmann Publishers
Keywords
• I/O requests Reads or writes to I/O devices
62004 Morgan Kaufmann Publishers
Interfacing Processors and Peripherals
• I/O Design affected by many factors (expandability, resilience)
• Performance:— access latency — throughput— connection between devices and the system— the memory hierarchy— the operating system
• A variety of different users (e.g., banks, supercomputers, engineers)
Disk Disk
Processor
Cache
Memory- I/O bus
Mainmemory
I/Ocontroller
I/Ocontroller
I/Ocontroller
Graphicsoutput
Network
Interrupts
72004 Morgan Kaufmann Publishers
I/O
• Important but neglected
“The difficulties in assessing and designing I/O systems haveoften relegated I/O to second class status”
“courses in every aspect of computing, from programming tocomputer architecture often ignore I/O or give it scanty coverage”
“textbooks leave the subject to near the end, making it easierfor students and instructors to skip it!”
• GUILTY!
— we won’t be looking at I/O in much detail
— be sure and read Chapter 8 in its entirety.
— you should probably take a networking class!
82004 Morgan Kaufmann Publishers
I/O Devices
• Very diverse devices— behavior (i.e., input vs. output)— partner (who is at the other end?)— data rate
92004 Morgan Kaufmann Publishers
Figure 8.2 The diversity of I/O devices.
Device Behavior Partner Data rate (Mbit/sec)
Keyboard Input Human 0.0001
Mouse Input Human 0.0038
Voice input Input Human 0.2640
Sound input Input machine 3.0000
Scanner Input Human 3.2000
Voice output Output Human 0.2640
Sound output Output Human 8.0000
Laser printer Output Human 3.2000
Graphics display Output Human 800.0000-8000.0000
Modem Input or output Machine 0.0160-0.0640
Network/LAN Input or output Machine 100.0000-1000.0000
Network/wireless LAN Input or output Machine 11.0000-54.0000
Optical disk Storage Machine 80.0000
Magnetic tape Storage Machine 32.0000
Magnetic disk storage machine 240.0000-2560.0000
102004 Morgan Kaufmann Publishers
8.2 Disk Storage and Dependability
112004 Morgan Kaufmann Publishers
Keywords
• Nonvolatile Storage device where data retains its value even when power is removed.
• Track One of thousands of concentric circles that makes up the surface of a magnetic disk.
• Sector One of the segments that make up a track on a magnetic disk; a sector is the smallest amount of information that is read or written on a disk.
• Seek The process of positioning a read/write head over the proper track on a disk.
• Rotation latency Also called delay. The time required for the desired sector of a disk to rotate under the read/write head; usually assumed to be half the rotation time.
• Small computer systems interface (SCSI) A bus used as a standard for I/O devices.
122004 Morgan Kaufmann Publishers
Keywords
• Redundant arrays of inexpensive disks (RAID) An organization of disks that uses an array of small and inexpensive disks so as to increase both performance and reliability.
• Striping Allocation of logically sequential blocks to separate disk to allow higher performance than a single disk can deliver.
• Mirroring Writing the identical data to multiple disks to increase data availability.
• Protection group The group of data disks or blocks that share a common check disk or block.
• Hot swapping Replacing a hardware component while the system is running.
• Standby spares Reserve hardware resources that can immediately take the place of a failed component.
132004 Morgan Kaufmann Publishers
I/O Example: Disk Drives
• To access data:— seek: position head over the proper track (3 to 14 ms. avg.)— rotational latency: wait for desired sector (.5 / RPM)— transfer: grab the data (one or more sectors) 30 to 80 MB/sec
Platter
Track
Platters
Sectors
Tracks
142004 Morgan Kaufmann Publishers
Disk Read Time
• Q : What is the average time to read or write a 512-byte sector for a typical disk rotating at 10,000 RPM? The advertised average seek time is 6 ms, the transfer rate is 50 MB/sec, and the controller overhead is 0.2 ms. Assume that the disk is idle so that there is no waiting time.
• A:
ms2.92.001.00.30.6ms2.0MB/sec 50
KB 5.0
RPM 000,10
rotation 5.0ms0.6
If the measured average seek time is 25% of the advertised average time. The answer is
1.5ms + 3.0ms + 0.01ms + 0.2ms = 4.7ms
152004 Morgan Kaufmann Publishers
Figure 8.3 Six magnetic disks, varying in diameter from 14 inches down to 1.8 inches.
162004 Morgan Kaufmann Publishers
Figure 8.4 Characteristics of three magnetic disks by a single manufacturer in 2004.
Characteristics Seagate ST373453 Seagate ST3200822
Seagate ST94811A
Disk diameter (inches) 3.50 3.50 2.50
Formatted data capacity (GB) 73.4 200.0 40.0
Number of disk surfaces (heads) 8 4 2
Rotation speed (PRM) 15,000 7200 5400
Internal disk cache size (MB) 8 8 8
External interface, bandwidth (MB/sec)
Ultra320 SCSI, 320 Serial ATA, 150 ATA, 100
Sustained transfer rate (MB/sec) 57-86 32-58 34
Minimum seek (read/write) (ms) 0.2/0.4 1.0/1.2 1.5/2.0
Average seek read/write (ms) 3.6/3.9 8.5/9.5 12.0/14.0
172004 Morgan Kaufmann Publishers
Mean time to failure (MTTF) (hours) 1,200,000@25℃ 600,000@25℃ 330,000@25℃
Warranty (years) 5 3 —
Nonrecoverable read errors per bits read <1 per <1 per <1 per
Temperature, vibration limits (operating) 5-55 , ℃400Hz@0.5G
0-60 , ℃350Hz@0.5G
5-55 , ℃400Hz@1G
Size: dimensions (in.), weight (pounds) 1.0”X4.0”X5.8”, 1.9 lbs
1.0”X4.0”X5.8”, 1.4 lbs
0.4”X2.7”X3.9”, 0.2 lbs
Power: operating/idle/standby (watts) 20?/12/ — 12/8/1 2.4/1.0/0.4
GB/cu. In., GB/watt 3GB/cu.in., 4GB/W 9GB/cu.in., 16GB/W 10GB/cu.in., 17GB/W
Price in 2004, $/GB $400,
$5/GB
$100,
$0.5/GB
$100,
$2.50/GB
1510 1410 1410
182004 Morgan Kaufmann Publishers
Figure 8.5 Summary of studies of reasons for failures.
Operator Software Hardware System Year data collected
42% 25% 18% Data center (Tandem) 1985
15% 55% 14% Data center (Tandem) 1989
18% 44% 39% Data center (DEC VAX) 1985
50% 20% 30% Data center (DEC VAX) 1993
50% 14% 19% U.S. public telephone network 1996
54% 7% 30% U.S. public telephone network 2000
60% 25% 15% Internet services 2002
192004 Morgan Kaufmann Publishers
Figure 8.6 RAID for an example of four data disks showing extra check disks per RAID level and companies that use each level.
202004 Morgan Kaufmann Publishers
Figure 8.7 Small write update on RAID 3 versus RAID4
212004 Morgan Kaufmann Publishers
Figure 8.8 Block-interleaved parity (RAID 4) versus distributed block-interleaved parity (RAID 5)
222004 Morgan Kaufmann Publishers
8.3 Networks
232004 Morgan Kaufmann Publishers
Networks are growing in popularity over time, and unlike other I/O devices, there are many books and courses on them. For readers who have not taken courses or read books on networking, Section 8.3 on the CD gives a quick overview of the topics and terminology, including internetworking, the OSI model, protocol families such as TCP/IP, long-haul networks such as ATM, local area networks such as Ethernet, and wireless networks such as IEEE 802.11.
242004 Morgan Kaufmann Publishers
8.4 Buses and Other Connections between Processors, Memory, and I/O Devices
252004 Morgan Kaufmann Publishers
Keywords
• Bus transaction A sequence of bus operations that includes a request and may includes a request and may include a response, either of which may carry data. A transaction is initiated by a single request and may take many individual bus operations.
• Processor-memory bus A bus that connects processor and memory and that is short, generally high speed, and matched to the memory system so as to maximize memory-processor bandwidth.
• Backplane bus A bus that is designed to allow processors, memory. And I/O devices to coexist on a single bus.
• Synchronous bus A bus that includes a clock in the control lines and a fixed protocol for communicating that is relative to the clock.
• Asynchronous bus A bus that uses a handshaking protocol for coordinating usage rather than a clock; can accommodate a wide variety of devices of differing speeds.
262004 Morgan Kaufmann Publishers
Keywords
• Handshaking protocol A series of steps used to coordinate asynchronous bus transfers in which the sender and receiver proceed to the next step only when both parties agree that the current step has been completed.
• Split transaction protocol A protocol in which the bus is released during a bus transaction while the requester is waiting for the data to be transmitted, which frees the bus for access by another requester.
272004 Morgan Kaufmann Publishers
I/O Example: Buses
• Shared communication link (one or more wires)• Difficult design:
— may be bottleneck— length of the bus— number of devices— tradeoffs (buffers for higher bandwidth increases latency)— support for many different devices— cost
• Types of buses:— processor-memory (short high speed, custom design)— backplane (high speed, often standardized, e.g., PCI)— I/O (lengthy, different devices, e.g., USB, Firewire)
• Synchronous vs. Asynchronous— use a clock and a synchronous protocol, fast and small
but every device must operate at same rate andclock skew requires the bus to be short
— don’t use a clock and instead use handshaking
282004 Morgan Kaufmann Publishers
I/O Bus Standards
• Today we have two dominant bus standards:
292004 Morgan Kaufmann Publishers
Figure 8.10 The asynchronous handshaking protocol consists of seven steps to read a word from memory and receive it in an I/O device.
302004 Morgan Kaufmann Publishers
312004 Morgan Kaufmann Publishers
Figure 8.11 Organization of the I/O system on a Pentium 4 PC using the intel 875 chip
Parallel ATA(100 MB/sec)
Parallel ATA(100 MB/sec)
(20 MB/sec)
PCI bus(132 MB/sec)
CSA(0.266 GB/sec)
AGP 8X(2.1 GB/sec)
Serial ATA(150 MB/sec)
Disk
Pentium 4processor
1 Gbit Ethernet
Memorycontroller
hub(north bridge)
82875P
MainmemoryDIMMs
DDR 400(3.2 GB/sec)
DDR 400(3.2 GB/sec)
Serial ATA(150 MB/sec)
Disk
AC/97(1 MB/sec)
Stereo(surround-
sound) USB 2.0(60 MB/sec)
. . .
I/Ocontroller
hub(south bridge)
82801EB
Graphicsoutput
(266 MB/sec)
System bus (800 MHz, 604 GB/sec)
CD/DVD
Tape
10/100 Mbit Ethernet
322004 Morgan Kaufmann Publishers
Figure 8.12 Two Pentium 4 I/O chip sets from Intel.
332004 Morgan Kaufmann Publishers
8.5 Interfacing I/O Devices to the Processor, Memory, and Operating System
342004 Morgan Kaufmann Publishers
Keywords
• Memory-mapped I/O An I/O scheme in which portions of address space are assigned to I/O devices and reads and writes to those addresses are interpreted as commands to the I/O device.
• I/O instructions A dedicated instruction that is used to give a command to an I/O device and that specifies both the device number and the command word (or the location of the command word in memory).
• Polling The process of periodically checking the status of an I/O device to determine the need to service the device.
• Interrupted-driven I/O An I/O scheme that employs interrupts to indicate to the processor that an I/O device needs attention.
• Direct memory access (DMA) A mechanism that provides a device controller the ability to transfer data directly to or from the memory without involving the processor.
• Bus master A unit on the bus that can initiate bus requests.
352004 Morgan Kaufmann Publishers
Other important issues
• Bus Arbitration:
— daisy chain arbitration (not very fair)
— centralized arbitration (requires an arbiter), e.g., PCI
— collision detection, e.g., Ethernet
• Operating system:
— polling
— interrupts
— direct memory access (DMA)
• Performance Analysis techniques:
— queuing theory
— simulation
— analysis, i.e., find the weakest link (see “I/O System Design”)
• Many new developments
362004 Morgan Kaufmann Publishers
Figure 8.13 The Cause and Status registers.
372004 Morgan Kaufmann Publishers
8.6 I/O Performance Measures : Examples from Disk and File Systems
382004 Morgan Kaufmann Publishers
Keywords
• Transaction processing A type of application that involves handling small short operations (called transactions) that typically require both I/O and computation. Transaction processing applications typically have both response time requirements and a performance measurement based on the throughput of transactions.
• I/O rate Performance measure of I/Os per unit time. Such as reads per second.
• Data rate Performance measure of bytes per unit time, such as GB/second.
392004 Morgan Kaufmann Publishers
Impact of I/O on system performance
• Q : Suppose we have a benchmark that executes in 100 seconds of elapsed time, where 90 seconds is CPU time and the rest is I/O time. If CPU time improves by 50% per year for the next five years but I/O time doesn’t improve, how much faster will our program run at the end of five years?
A:• We know that
Elapsed time = CPU time + I/O time
100 = 90 + I/O time
I/O time = 10 seconds
The new CPU times and the resulting elapsed times are computed in the following table:
<next page>
402004 Morgan Kaufmann Publishers
After n years CPU time I/O time Elapsed time % I/O time
0 90 seconds 10 seconds 100 seconds 10 %
1 60 seconds 10 seconds 70 seconds 14 %
2 40 seconds 10 seconds 50 seconds 20 %
3 27 seconds 10 seconds 37 seconds 27 %
4 18 seconds 10 seconds 28 seconds 36 %
5 12 seconds 10 seconds 22 seconds 45 %
5.1
90
5.1
60
5.1
40
5.1
27
5.1
18
The improvement in CPU performance over five years is
However, the improvement in elapsed time is only
and the I/O time has increased from 10% to 45% of the elapsed time.
5.712
90
5.422
100
412004 Morgan Kaufmann Publishers
8.7 Designing an I/O System
422004 Morgan Kaufmann Publishers
I/O System Design
• Q: Consider the following computer system:– A CPU that sustain 3 billion instructions per second and averages
100,000 instruction in the operating system per I/O operation
– A memory backplane bus capable of sustaining a transfer rate of 1000 MB/sec
– SCSI Ultra320 controllers with a transfer rate of 320 MB/sec and accommodating up to 7 disks
– Disk drives with a read/write bandwidth of 75 MB/sec and an average seek plus rotational latency of 6 ms.
If the workload consists of 64 KB reads (where the block is sequential on a track) and the user program needs 200,000 instructions per I/O operation, find the maximum sustainable I/O rate and the number of disks and SCSI controllers required. Assume that the reads can always be done on an idle disk if one exists (i.e., ignore disk conflicts).
432004 Morgan Kaufmann Publishers
• A:
second
I/Os625,15
1064
101000
I/Oper Bytes
bandwidth Busbus of rate I/O Maximum
so KB, 64 transfer I/OEach
second
I/Os000,10
10)100200(
103
I/Oper nsInstructio
rateexecution n Instrucito
CPU of rate I/O Maximum
3
6
3
9
MB/sec56.9ms9.6
KB64
imeTransfer t
sizeTransfer rateTransfer
ms9.6MB/sec75
KB64ms6
imeTransfer t imerotation t Seek disk at I/Oper Time
The maximum number of disks per SCSI bus is 7, which won’t saturate this bus. This means we will need 69/7, or 10 SCSI buses and controllers.
442004 Morgan Kaufmann Publishers
8.8 Real Stuff : A Digital Camera
452004 Morgan Kaufmann Publishers
Figure 8.14 The Sanyo VPC-SX500 with Flash memory card and IBM Microdrive.
462004 Morgan Kaufmann Publishers
Figure 8.15 Characteristics of three storage alternatives for digital cameras.
Characteristics Sandisk Type I compactFlash SDCFB-128-768
Sandisk Type II compactFlash SDCFB-1000-768
Hitachi 4 GB Microdrive
DSCM-10340
Formatted data capacity (MB) 128 1000 4000
Bytes per sector 512 512 512
Data transfer rate (MB/sec) 4 (burst) 4 (burst) 4 – 7
Link speed to buffer (MB/sec) 6 6 33
Power standby/operating (W) 0.15/0.66 0.15/0.66 0.07/0.83
Size: height X width X depth (inches) 1.43 X 1.68 X 0.13 1.43 X 1.68 X 0.13 1.43 X 1.68 X 0.16
Weight in grams (454 grams/pound) 11.4 13.5 16
Write cycles before sector wear-out 300,000 300,000 Not applicable
Mean time between failures (hours) > 1,000,000 > 1,000,000 (see caption)
Best price (2004) $40 $200 $480
472004 Morgan Kaufmann Publishers
Figure 8.16 The system on a chip (SOC) found in Sanyo digital cameras.
482004 Morgan Kaufmann Publishers
8.9 Fallacies and Pitfalls
492004 Morgan Kaufmann Publishers
Fallacies and Pitfalls
• Fallacy: the rated mean time to failure of disks is 1,200,000 hours, so disks
practically never fail.
• Fallacy: magnetic disk storage is on its last legs, will be replaced.
• Fallacy: A 100 MB/sec bus can transfer 100 MB/sec.
• Pitfall: Moving functions from the CPU to the I/O processor, expecting to
improve performance without analysis.
502004 Morgan Kaufmann Publishers
Multiprocessors
• Idea: create powerful computers by connecting many smaller ones
good news: works for timesharing (better than supercomputer)
bad news: its really hard to write good concurrent programs many commercial failures
Cache
Processor
Cache
Processor
Cache
Processor
Single bus
Memory I/ONetwork
Cache
Processor
Cache
Processor
Cache
Processor
Memory Memory Memory
512004 Morgan Kaufmann Publishers
Questions
• How do parallel processors share data?— single address space (SMP vs. NUMA)— message passing
• How do parallel processors coordinate? — synchronization (locks, semaphores)— built into send / receive primitives— operating system protocols
• How are they implemented?— connected by a single bus — connected by a network
522004 Morgan Kaufmann Publishers
Supercomputers
93 93 94 94 95 95 96 96 97 97 98 98 99 99 00
500
400
300
200
100
0
Single Instruction multiple data (SIMD)
Cluster(network ofworkstations)
Cluster(network ofSMPs)
Massivelyparallelprocessors(MPPs)
Shared-memorymultiprocessors(SMPs)
Uniprocessors
Plot of top 500 supercomputer sites over a decade:
532004 Morgan Kaufmann Publishers
Using multiple processors an old idea
• Some SIMD designs:
• Costs for the the Illiac IV escalated from $8 million in 1966 to $32 million in 1972 despite completion of only ¼ of the machine. It took three more years before it was operational!
“For better or worse, computer architects are not easily discouraged”
Lots of interesting designs and ideas, lots of failures, few successes
542004 Morgan Kaufmann Publishers
Topologies
a. Crossbar
P0
P1
P2
P3
P4
P5
P6
P7b. Omega network
P0
P1
P2
P3
P4
P5
P6
P7
a. 2-D grid or mesh of 16 nodes
b. n-cube tree of 8 nodes (8 = 23 so n = 3)
552004 Morgan Kaufmann Publishers
Clusters
• Constructed from whole computers
• Independent, scalable networks
• Strengths:
– Many applications amenable to loosely coupled machines
– Exploit local area networks
– Cost effective / Easy to expand
• Weaknesses:
– Administration costs not necessarily lower
– Connected using I/O bus
• Highly available due to separation of memories
• In theory, we should be able to do better
562004 Morgan Kaufmann Publishers
• Serve an average of 1000 queries per second
• Google uses 6,000 processors and 12,000 disks
• Two sites in silicon valley, two in Virginia
• Each site connected to internet using OC48 (2488 Mbit/sec)
• Reliability:
– On an average day, 20 machines need rebooted (software error)
– 2% of the machines replaced each year
In some sense, simple ideas well executed. Better (and cheaper) than other approaches involving increased complexity
572004 Morgan Kaufmann Publishers
Concluding Remarks
• Evolution vs. Revolution
“More often the expense of innovation comes from being too disruptive to computer users”
“Acceptance of hardware ideas requires acceptance by software people; therefore hardware people should learn about software. And if software people want good machines, they must learn more about hardware to be able to communicate with and thereby influence hardware engineers.”