Slide 1
DESIGN, IMPLEMENTATION, AND PERFORMANCE ANALYSIS OF THE ISCSI PROTOCOL FOR SCSI OVER
TCP/IP
ByAnshul Chadda (Trebia Networks)-Speaker
Ashish Palekar (Trebia Networks)Narendran Ganapathy (Sun Microsystems)R.D.Russell (University of New Hampshire)
Slide 2
Overview
• Background – SCSI Concepts– SAN– iSCSI– SCSI Subsystem in Linux
• iSCSI Initiator Implementation• Resources Used• Basic Performance Analysis• Conclusions and Future Work• Questions
Slide 3
What is SCSI?
• Acronym for Small Computer System Interface
• SCSI device identified by ID• Logical Units present within the SCSI device• Logical Unit Numbers: for logical addressing
TARGET Hard disk
INITIATOR(Host)
LUN 0 LUN 1 LUN 2 LUN 3
ID 7 ID 0
SCSI bus
Slide 4
SCSI Terms
Limitations• Length of the SCSI bus (25 m) • Number of SCSI devices supported• Single host point-to-point
INITIATOR TARGET
COMMAND
STATUS
MESSAGE IN
MESSAGE OUT
DATA IN
DATA OUT
Slide 5
Storage Area Network (SAN)
• Host accesses the remote storage as a device• Unit of Access: BLOCK• Security, Consistency, Integrity Issues at
BLOCK level
SAN
ApplicationClient 1
Block Access Protocol
ApplicationClient 2
Disk
Slide 6
New SAN Approaches
TCP/IP/Ethernet Network
ApplicationClient 1
Block Access Protocol
ApplicationClient 2
Disk
Ideal Solution
Our Approach
ApplicationClient 1
Block Access Protocol
ApplicationClient 2 Disk
TCP/IP/Ethernet Network
Target Emulator
Slide 7
iSCSI Protocol (an IETF standard)
• Requires Reliable Transport Layer (TCP)• Session established between Initiator and Target• Multiple connections within a session • Header size
– Large (48-byte) Basic header segment– Additional Header Segment can be present
• Sequence Numbers • Header and Data Digests optional• Security and Authentication optional• Parameter Negotiation• Target Discovery Mechanism
Slide 8
User space
Kernel space
Target EmulatorInitiator
User Application
File System
SCSI
SCSI HBA Driver
Disk
User Application
File System
SCSI
iSCSI Subsystem
Disk
Transport Network
Network Stack
Network Stack
SCSI
iSCSI
Subsystem
SCSI HBA Driver
Standard SCSI SCSI Over Network
Slide 9
SCSI Subsystem in Linux SCSI Upper level
SCSI Mid level
SCSI Host Bus Adapter
(Low level driver)
Target (Disk)
Target (Disk)
Target (Disk)
TransportNetwork
SCSI Host Bus Adapter
(Low level driver)
Target (Disk)
Target (Disk)
Target (Disk)
TransportNetwork
SCSI Command SCSI Status,Response
SCSI Command SCSI Status,Response
Slide 10
iSCSI Initiator Implementation
Slide 11
Resources Used For Testing
• Initiator– 2.4 GHz CPU, 1GB Memory– Fast/Gigabit Ethernet Cards– Linux Kernel (2.4.19)– UNH iSCSI Initiator kernel modules
• Target– 2.4 GHz CPU, 1GB Memory – Fast/Gigabit Ethernet Cards– Linux Kernel (2.4.19)– UNH iSCSI Target kernel modules
Slide 12
Performance Metrics & Variables
• Metrics: Bandwidth, Latency and % CPU utilization
• Variables – Initiator Scatter-Gather List Size – Ethernet Link Speed– Target Sector Size– Header & Data Digests– TCP Nagle Algorithm
Slide 13
Baseline Performance Values
• For the following Performance Variable values:– Initiator Scatter-Gather List Size=64 – Ethernet Link Speed=1000 Mbps– Target Sector Size=512 B– Header & Data Digests=OFF– TCP Nagle Algorithm=OFF
• READ:– Bandwidth=50.6761 MB/s, – %CPU util. On target=18.65%– %CPU util. On initiator=27%
• WRITE:– Bandwidth=53.3937 MB/s – %CPU util. On target=14.3%– %CPU util. On initiator=26.9%
Slide 14
Effect of Initiator Scatter-Gather list size on Bandwidth
50.6761
46.0205
53.3937
49.841
0
10
20
30
40
50
60
Ban
dwid
th (
MB
/s)
1 2
READ WRITE(9.18% BW decrease) (6.65% BW decrease)
(~10%CPU util. Increase) (~10%CPU util. increase)
scatter-gather list size=64
scatter-gather list size=8
Slide 15
Effect of Ethernet Link Speed on Bandwidth
50.6761
11.0265
53.3937
11.0297
0
10
20
30
40
50
60
Ban
dwid
th (
MB
/s)
1 2
READ WRITE(78.24% BW decrease) (79.4% BW decrease)
(~50 %CPU util. decrease ) (~50%CPU util. decrease)
1000 MB Ethernet Link: % Bandwidth util.=42%
100 MB Ethernet Link: %Bandwidth util.=88%
Slide 16
Effect of Target Sector Size on Bandwidth
50.676152.5764 53.3937
60.2563
0
10
20
30
40
50
60
70
Ban
dwid
th (
MB
/s)
1 2
READ WRITE(3.8% BW increase) (12.9% BW increase)
(~10%CPU util. decrease) (~10%CPU util. decrease)
Target Sector Size=512B
Target Sector Size=4096B
Slide 17
Effect of Header & Data Digests on Bandwidth
50.6761
42.1175
53.3937
39.8592
0
10
20
30
40
50
60
Ban
dwid
th (
MB
/s)
1 2
READ WRITE(16.8% BW decrease) (25.3% BW decrease)(~8% CPU util. increase) (~8% CPU util. increase)
Header, Data Digests OFF
Header, Data Digests ON
Slide 18
Effect of TCP Nagle Algorithm on Bandwidth
50.676152.1742 53.3937 53.3481
0
10
20
30
40
50
60
Bandw
idth
(M
B/s
)
1 2
READ WRITE(2.96% BW increase) (0.085% BW decrease)(<5% CPU util. Change) (<5% CPU util. Change)
Nagle Algorithm=OFF
Nagle Algorithm=ON
Slide 19
Conclusions
• To Increase Bandwidth– Increase Initiator Scatter-Gather List Size – Increase Ethernet Link Speed– Increase Target Sector Size – Turn OFF Header & Data Digests– TCP Nagle Algorithm has negligible effect
Slide 20
Future Work
• Run multiple simultaneous applications on the initiator system to utilize the CPU and bandwidth
• Do Latency measurements too• Detailed Performance Analysis for each network
stack involved• Analyze the effect of iSCSI protocol parameters
on performance. • Test hardware implementation for TCP/IP and
iSCSI• Compare with Fiber Channel technology for
bandwidth, latency and CPU utilization
Slide 21
Acknowledgements
• SUN/EMC for funding the project• Inter-Operability Lab. And Trebia
Networks for providing system resources
Slide 22
Questions???