RAID-x: A New Distributed Disk Array for I/O-Centric Cluster Computing
Kai Hwang, Hai Jin, and Roy Ho
2
Outline
Introduction
RAID
Orthogonal Striping and Mirroring
Trojans Cluster Experiments
Cooperative disk drivers
Benchmark Experiments
Striped Checkpointing on RAID-x
3
Introduction
RAID-X Redundant array of inexpensive disks at level x Provides
High Bandwidth distributed I/O processing on a Serverless Cluster where server
functions are distributed among client hosts Based on Orthogonal striping and mirroring (OSM) Cooperative Disk Drivers (CDD) are used to
implement the OSM at the kernel level Maintains data consistency without using NFS or Unix system
calls
4
Distributed RAID-x
Must have these Capabilities:1. A single I/O space (SIOS) for all disks in cluster.
2. High scalability, availability and compatibility with current cluster architectures and applications.
3. Local and remote disk I/O operations performed with comparable latency.
Implies. Total transparency to users. Utilize all disks without knowing the physical locations of
the data blocks.
5
Orthogonal Striping and Mirroring
OSM Provides: Improvement in Parallel I/O Bandwidth Hides disk mirroring overhead Enhances scalability and reliability of cluster
computing applications.
Eliminates Small write problem which affects RAID-5.
Has advantages of both RAID-1 and Chained Declustering
6
Advantages of RAID-1 & Chained Declustering
RAID-1: Mirroring and Duplexing100% redundancy of data = no rebuild, just
a copyTwice the Read transaction rate of single
disks
Chained Declustering Load Balancing
7
Architecture of RAID-x vs. chained declustering RAID
Bj original data blocks.
Mj mirrored blocks.4 disks with mirroring groups involving 3 consecutive disk blocks.
ex. M0, M1, M2, mirroring blocks for B0, B1, B2 data blocks.
Diff mirroring groups are in diff shadings.
8
Architecture of RAID-x
Orthogonal mapping. no data block and it’s image are mapped to the same disk.
Data blocks are striped across all disks on the top half of the disk array, like RAID-0.
Means that for large writes blocks can be written in parallel to all disks in the stripe simultaneously.
Image blocks are “clustered” in the same disk vertically. Clustered images in the mirroring group are
simultaneously updated in the background. Resulting in lower latency and higher bandwidth in RAID-x.
10
Performance of 4 RAID Architectures
Write operations are improved.
The same bandwidth potential as Raid-0 and chained declustering.
Improvements from declustering are mainly in parallel writes. For large array size improvement approaches a factor of 2.
Tolerate single disk failures (RAID-5).
11
Trojans Cluster
16 Pentium II/400 MHz processors
RedHad Linux v. 6.0
PC engines (nodes) were connected by 100 Mbps Fast Ethernet
Each node is attached with a 10GB disk 16 disks = 160 GB single
I/O space
12
Distributed RAID-x Architecture
3 disks for each node, 4 nodes
Stripe groups B0, B1, B2, B3
accessed in parallelConsecutive stripe groups (B0, B1, B2, B3), (B4, B5, B6, B7), (B8, B9, B10, B11) accessed in pipeline fashion because they are retrieved from disk groups attached to the same SCSI buses
4x3 RAID-x architecture with orthogonal striping and mirroring
P: processor, M:memoryCCD:cooperative disk driver, Dj the jth disk, Bi: the ith data block, Bi’: the ith mirrored image in a shaded box
13
Distributed RAID-x Arch. cont.
n-by-k RISK-x Stripe group w/in disk
blocks on n disks Mirroring group n-1 blocks
on 1 disk Images of all data blocks in
stripe group are saved to two disks
Block addressing scheme strips across all nk disks sequentially and repeatedly
n = degree of parallelism K = depth of pipelining
4x3 RAID-x architecture with orthogonal striping and mirroring
P: processor, M:memoryCCD:cooperative disk driver, Dj the jth disk, Bi: the ith data block, Bi’: the ith mirrored image in a shaded box
14
Single I/O space in a Distributed RAID
A global virtual disk with a SIOS formed by cooperative disks
crucial to building scaleable cluster of computersIf not shared then I/O must be handled by time consuming system class through centralized file server (ex NSF)Enabled by CDDs at Linux kernel level
15
Cooperative Disk Driver (CDD) Architecture
Internal design of CDD arch.
Establish SIOS single global virtual disk
Storage Manager Receives and processes the
I/O requests
Client modules Redirects local I/O requests
to remote disk managers
16
CDD Architecture cont.
Data Consistency Module Maintains data consistency at Driver level that result from
distributed disks updating cached copies of same data block
Can Run in 3 different states Storage Manager: coordinates use of local disk storage by
remote nodes Client: accessing remote disks through remote disk
managers both
17
CDD
Allows serverless clusters
Offers remote disk access directly at kernel level
18
I/O Bandwidth vs. request number
Performance of 4 I/O subsystem architectures
For large read/write20MB file striped across all
disks in arrayFocuses on parallel I/O
capacity of disk arrayUncached filesClient reads only private fileAll reads performed
simultaneously
19
I/O Bandwidth vs. request number
Performance of 4 I/O subsystem architectures
Small read/write32KB data One block of stripe group
Results for small are very close to the results for large
21
Achievable I/O Bandwidth and Improvement Factor
Improvement factor of 16 clients over 1 on USC Trojans ClusterRAID-x demonstrates the highest improvement factor among the three RAID Arch.Almost 3x increase on RAID-X from 1-16
22
Elapsed Time in Executing the Andrew Benchmark
On 4 I/O subsystems with respect to increase of number of client requests up to 32.
NSF results RAID-x results
23
Striped Checkpointing on the RAID-x
Distribute data blocks and their mirrored images orthogonallyStriped staggering in coordinated checkpointing on the RAID-x disk array Successive stripes are accessed
in a staggered manner from diff. Stripes on successive 4-disk groups.
Staggering implies pipelined access of disk array
24
Striped Checkpointing on the RAID-x
Using OSM each striped checkpointing file has its mirrored image on its local disk. For each node, transient
failures can be recovered from its mirrored image in local disk. Permanent failures can be recovered from striped checkpointing.
25
Conclusions
RAID-x shows strength in building distributed, high-bandwidth, I/O storage for serverless PC or workstation clusters.OSM Architecture exploits full stripe bandwidthReliability Clustered mirror on on local disks and orthogonal
striping across distributed disks. Matches RAID-5 (recovery from single disk failures)
I/O performance is better than RAID-1 and 5Highly scalable with distributed control
26
Questions
What does Orthogonal striping and mirroring (OSM) provide?
What is RAID-x?
Explain the Cooperative Disk Driver (CDD).
Compare RAID-x to Petal.