18
Storage Networks DANIEL J. WORDEN APress Media, LLC
Storage Networks
DANIEL J. WORDEN
APress Media, LLC
Storage Networks Copyright © 2004 by Daniel J. Worden Originally published by Apress in 2004
All rights reserved. No part of this work may be reproduced or transmitted in any form or by
any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher.
ISBN 978-1-59059-298-4 ISBN 978-1-4302-0694-1 (eBook) DOI 10.1007/978-1-4302-0694-1
Trademarked names may appear in this book. Rather than use a trademark symbol with every
occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.
Lead Editor: Jim Sumser
Technical Reviewer: Jonathan Hassell
Editorial Board: Steve Anglin, Dan Appleman, Ewan Buckingham, Gary Cornell, Tony Davis,
John Franklin, Jason Gilmore, Chris Mills, Steven Rycroft, Dominic Shakeshaft, Jim Sumser, Karen Watterson, Gavin Wray, John Zukowski
Assistant Publisher: Grace Wong
Project Manager: Beth Christmas
Copy Manager: Nicole LeClerc
Copy Editor: Ami Knox
Production Manager: Karl Brooks
Production Editor: JanetVail
Compositor: Diana Van Wmlde
Proofreader: Elizabeth Berry
Indexer: Valerie Perry
Artist: Kinetic Publishing
Cover Designer: Kurt Krames
Manufacturing Manager: Tom Debolski
In the United States: phone 1-800-SPRINGER, email orders@springer-ny. corn, or visit http: 1 IWviW. springer -ny. corn. Outside the United States: fax +49 6221 345229, email orders@springer. de, or visit http: 1 /WviW. springer. de.
For information on translations, please contact Apress directly at 2560 Ninth Street, Suite 219, Berkeley, CA 94710. Phone 510-549-5930, fax 510-549-5939, email [email protected], or visit http: 1 /WviW. apress. corn.
The information in this book is distributed on an "as is" hasis, without warranty. Although every
precaution bas been taken in the preparation ofthis work, neither the author{s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work.
For my brothers: John mrich-Thanks for making our payroll, so many times!
Verner mrich-Your compassion is an inspiration. Tom mrich-Working as an executive with your direction was a career high point.
Viggo mrich-My son is a better man because you were "in loco parentis." Bill Ulrich-You go to impressive distances to set such a good example.
Stephen Worden-How could I ever forget OokaDokah? Donald Worden-Even as an eight-year-old, you could always out-quip me.
Contents at a Glance
About the Author .............................................................................................. xvii
About the Technical Reviewer ..................................................................... xix
Acknowledgments .................................................................................................. xxi
Introduction ...................................................................................................... xxiii
Chapter 1 A Halk Around the Technology ............................................ l
Chapter 2 Hard Disk Technology ........................................................... 33
Chapter 3 Networks ..................................................................................... 51
Chapter 4 SCSI: The Specification, Architecture, and Command Set ...................................................................... 85
Chapter 5 RAID Levels and Logical Volumes ................................. 103
Chapter 6 File Systems and Network Protocols .......................... 115
Chapter 7 SAN Backup and Recovery ................................................... 129
Chapter 8 Storage Security .................................................................. 143
Chapter 9 Storage Networks Software and Content .................... 165
Chapter 10 Do-It- Yourself Network Storage Components ........... 189
Chapter 11 Vendor Offerings .................................................................. 213
Chapter 12 Making the Business Case for Storage Networks .. 233
Chapter 13 Future Directions for Storage Networks .................. 261
Afterword .............................................................................................................. 275
Index ...................................................................................................................... 277
v
Contents
About the Author .............................................................................................. xvii
About the Technical Reviewer ..................................................................... xix Acknowledgments .................................................................................................. xxi
Introduction ...................................................................................................... xxiii
Chapter 1 A Walk Around the Technology ................................. 1
Getting Started ..................................................................................................... 1 The Network Is the Computer-Again ............................................................. 2 Class and Instance .............................................................................................. 3 In the Beginning-There Was the ISOIITU/OSI Network Model ........... 3 SAN Components ....................................................................................................... 4 Benchmarking Data Throughput ........................................................................ 8 Get on the Bus ..................................................................................................... 10 Industry Standard System Bus Architectures ........................................ 12 Proprietary Bus Architectures .................................................................... 13 InfiniBand .............................................................................................................. 14 Host Bus Adapters .............................................................................................. 15 Gigabit Ethernet ................................................................................................ 16 iSCSI ........................................................................................................................ 17 Fibre Channel ....................................................................................................... lB Putting Together the Hardware Puzzle ..................................................... 19 Topologies .............................................................................................................. 20
Networked Storage: NAS Is SAN Spelled Backwards ...................................... 22 SAN-Switched Network Storage ..................................................................... 23 NAS Gateway ..................................................................................................... 24
Writing to Disk ................................................................................................... 24 A Word on Disks ................................................................................................... 25
ATA Disk Drives ................................................................................................. 25 SCSI .................................................................................................................... 26
Connecting the Dots .......................................................................................... 26 Switch Technology .............................................................................................. 28
Layer 2 Switches ................................................................................................ 29 Layer 3 Switches ................................................................................................ 29 Layer 4 Switches ................................................................................................ 29 Comparing Switch Layers: Performance and Service .................................... 30
vii
Contents
Orchestrating the Arrangement .................................................................... 30 Storage Device and Network Resource Management.. .................................. 31 Data and Policy Management .......................................................................... 31 Virtualization ..................................................................................................... 31
Summary .................................................................................................................... 32
Chapter 2 Hard Disk Technology ................................................... 33
Dissecting a Drive ............................................................................................ 33 Mechanical Components ...................................................................................... 34
What Could Go Wrong? ........................................................................................ 35 My Head on a Platter ........................................................................................ 35 Fitness to Task ................................................................................................... 37
Not So Fast ........................................................................................................... 37
Scuzzy ...................................................................................................................... 38 Serial vs. Parallel Interfaces .................................................................. 39 That's SMART ......................................................................................................... 40 Addressing the Disk .......................................................................................... 44
ATA/IDE Drives .................................................................................................. 45
Hard Disk Performance ...................................................................................... 46
SCSI Addressing ................................................................................................... 47
SCSI Devices ......................................................................................................... 48
LUN Masking ..................................................................................................... 48
SCSI Command Set ........................................................................................... 49 Summary .................................................................................................................... 49
Chapter 3 Networks .................................................................................. 51
IEEE 802 .................................................................................................................. 51 Premise Wiring ................................................... , ................................................. 53
Category 5 and More Cable ......................................................... , .................... 54
Connectors ........................................................................................................ 56 Ethernet Network Cards ......................... , ......................................................... 56
Media Access Control .............. , ........................................................................ 57
Header Construction ........................................................................................ 60 Bridges and Switches ........................................................................................ 60 Datagrams .......................................................................................................... 61 802.1Q and 802.1P ............................................................................................. 61
TCP liP Routing ..................................................................................................... 61 Nonroutable Protocols and Addresses ............................................................ 62 Firewalls ............................................................................................................. 63
ISO Layer 1 and Layer 2 ................................................................................. 64
Fibre Channel from the Ground Up ............................................................. 64
viii
Contents
Let There Be Light at the End of the Tunnel ..................................... 65 Point-to-Point Essentials .................................................................................. 65 Lasers One Oh One ........................................................................................... 66 Photodiodes ....................................................................................................... 67 Propagation over Distance ............................................................................... 68
Loss Through Dispersion ................................................................................. 70 The 3 Rs .................................................................................................................. 71 EDFA ........................................................................................................................... 72 Modes in Fiber-Optic Cable ........................................................................... 72 Multiplexing Signals ........................................................................................ 73
Time Division Multiplexing .............................................................................. 73 Wavelength Division Multiplexing ................................................................... 74
Protocol Independence ...................................................................................... 75 Connectors .............................................................................................................. 75 Topologies .............................................................................................................. 75
FC-AL ................................................................................................................. 76 Fibre Channel ....................................................................................................... 76
FC-0 Fibre Channel Defined ............................................................................ 77 FC-1 .................................................................................................................... 77 InterSyn1bol Interference ................................................................................. 79 8b/10b ................................................................................................................ 80 FC-2 .................................................................................................................... 80 FC-3 .................................................................................................................... 81 FC-4 .................................................................................................................... 82
Parallel SCSI, Ethernet, and Fibre Channel.. ...................................... 82 Fibre Channel over Copper Cable vs. Parallel SCSI ........................................ 83 SCSI Commands over Fiber-Optic Cable ........................................................ 83
Summary .................................................................................................................... 84
Chapter 4 SCSI: The Specification, Architecture, and Command Set ................................................................ 85
Common Command Set ............................................................................................ 86 SAM: The SCSI Architecture Model ............................................................. 86 SCSI Communications .......................................................................................... 88 SCSI Messaging ..................................................................................................... 91 SCSI Commands ....................................................................................................... 92 Processing Multiple Commands ...................................................................... 93 Encapsulating Protocols Within SCSI Communications ...................... 94 SCSI Enclosure Services ................................................................................. 94 Fibre Channel SCSI ............................................................................................ 98
SCSI-FCP ............................................................................................................ 98 iSCSI ........................................................................................................................ 99
iSCSI Protocol .................................................................................................... 99 Summary .................................................................................................................. 101
ix
Contents
Chapter 5 RAID Levels and Logical Volumes ..................... 103
RAID Levels and What They Mean ................................................................ l03 Striping-aka RAID Level 0 ............................................................................ 104 Spanning: When Only the Very Biggest Will Do ............................................ 104 Mirroring-aka RAID 1 ................................................................................... 105 Mirrored Stripe Set-aka RAID 0+ 1 ............................................................... ] 06 Stripe with Parity-aka RAID 3 ...................................................................... 106 RAID5 .............................................................................................................. 107
RAID Read/Write Operations ......................................................................... l08 Stripe Setting ................................................................................................... 108 Mirroring .......................................................................................................... l 09
Stripe with Parity ............................................................................................. 1 09 Distributed Parity ....................................................................................... ..... 11 0
Hardware RAID ..................................................................................................... llO Software RAID ..................................................................................................... lll RAID Strategies ................................................................................................. lll Logical Volume Management ........................................................................... lll
Logical Volume Capabilities ........................................................................... 112 Summary .................................................................................................................. 114
Chapter 6 File Systems and Network Protocols ............. 115
What Is a File System? ................................................................................. 115 Shared File Systems ........................................................................................ 116
Network File System ........................................................................................ l17 Network Information Services ....................................................................... 118 Server Message Block ...................................................................................... 118
Journaling File Systems ............................................................................... 121 Distributed File Systems ............................................................................. 123
Andrew File System ......................................................................................... l23 DCE/DFS .......................................................................................................... l24 Microsoft Dfs .................................................................. ; ................................ 124 Coda ................................................................................................................. 124 lntermezzo ....................................................................................................... l24 Lustre ................................................................................................................ 124 Sistina Global File System ............................................................................... l25
Enterprise Volume Management Systems ................................................... 125 Summary .................................................................................................................. 127
X
Contents
Chapter 7 SAN Backup and Recovery .......................................... 129
The Fundamental Value of Data .................................................................. 130 Protecting Your Interests ........................................................................... 131 The Perils of Pauline .................................................................................... 132 The Sarbanes-Oxley Act of 2002 ................................................................ 133 Speed Costs-How Fast Did You Want to Go? .......................................... 134 It's the Recovery, Not the Backup, That Counts ............................. 134
Keyword Search ............................................................................................... 135 Backup Strategy by Data Type .................................................................... 135
File Systems Backup ........................................................................................ 136 Drive-to-Drive Backup .................................................................................... 13B
Snapshot Copy ................................................................................................. 138 Clone Copy ...................................................................................................... 138 Mirror Copy ..................................................................................................... 139 Remote Copy ................................................................................................... 139 Extended Copy ................................................................................................ 140
Restore, Recover, and Fail Over .............................................................. 140 What Was the Question Again? .................................................................... 141 78s, 45s, 33s, 8-Tracks, Cassettes, COs and MP3s ........................ 141 Summary .................................................................................................................. 142
Chapter 8 Storage Security ............................................................ 143
Can't Read What You Can't Get .................................................................. 143 It's Classified ................................................................................................. 144 Gates of the Great Wall ............................................................................... 145 Tunneling .............................................................................................................. 145 Secure Socket Layers ...................................................................................... 146 IPSec ...................................................................................................................... 147 The Keys to Enlightenment ........................................................................... 14B Digital Signatures .......................................................................................... 149 Digital Certificates ...................................................................................... 149 How Encryption Works ...................................................................................... 150 Drilling into IPSec ........................................................................................ 151 IK£ ........................................................................................................................... 153 FCSec ...................................................................................................................... 153 SAN Fabric Security ........................................................................................ 154
Digital Certificates .......................................................................................... . 155 Hard Zoning ..................................................................................................... 155 SoftZoning ....................................................................................................... 155 LUN Masking ................................................................................................... 155 CHAP: The Challenge Authentication Protocol ............................................ 156
xi
Contents
Attack Types ....................................................................................................... 156 Denial of Service {DoS) ................................................................................... 156 Malformed Packets ......................................................................................... 156 IP Packet Fragments {FragAttack) ................................................................. 157 Worms .............................................................................................................. 157 Buffer Overflows .............................................................................................. 157 Application Attacks ......................................................................................... 158
The Theory in Practice ................................................................................. 158 ACL Configuration .......................................................................................... 159 VPN Authentication ........................................................................................ 160
Impact and Implications for Storage Networks ................................. 160 Get a GRIP ............................................................................................................ 161 Encrypted Storage ............................................................................................ 162 Summary .................................................................................................................. 163
Chapter 9 Storage Networks Software and Content ......................................................................... 165
Managing Storage Network Resources ....................................................... 166 Storage Network Management ...................................................................... 166 Storage Resource ManagementTools ............................................................ 167
Virtualization ................................................................................................... 173 Policy-Driven Storage Management ........................................................... 174
The Autonomic Model for Storage ................................................................. l74
Integrated Solutions Console ......................................................................... 175 Proprietary Products vs. Open Standards ..................................................... 178
Accessing Storage Assets ............................................................................. 178 Database Servers ............................................................................................. 178 Mail Servers ..................................................................................................... 179 Office Documents ........................................................................................... 180
Content for Storage Networks .................................................................... 181 The Information Life Cycle ............................................................................. 182 ILMValue Proposition .................................................................................... 183 Metalogical Representations .......................................................................... 184 Metadata Mechanisms ................................................................................... 184
Modeling Your Storage Network Content ................................................. 186 Enter UML ....................................................................................................... 187
Summary .................................................................................................................. 188
xii
Contents
Chapter 10 Do-It-Yourself Network Storage Components ....................................................... 1B9
Fitness to Task ................................................................................................. 190 Network Storage Experiment ......................................................................... 191 Elements of the Test ...................................................................................... 191 The Premise ......................................................................................................... 193 Theory and Practice ........................................................................................ 194 The Results ......................................................................................................... 194 Comparisons ......................................................................................................... 196
SCSI vs. ATA/66 ................................................................................................ 196 10/100vs. GbE ................................................................................................. 198 NTFS vs. F.AT32 ................................................................................................ 198 ATA/66 vs . .ATA/100 ......................................................................................... 199 SMB vs. NFS ..................................................................................................... 200 Linux File Systems ........................................................................................... 201 FTP ................................................................................................................... 201 Spanned, Mirrored, and Mirrored Striped Drives ........................................ 201 Sustainable Performance ................................................................................ 202 Cat 5 vs. Cat 6 ................................................................................................... 203 Jumbo Frames ................................................................................................. 203
Tuning GbE Storage Network Components ................................................. 204 Methodology ......................................................................................................... 205 My Requirements ................................................................................................. 206 Practical Application .................................................................................... 20B Lessons Learned ................................................................................................. 211 Summary .................................................................................................................. 212
Chapter 11 Vendor Offerings ............................................................ 213
SAN Component Vendors .................................................................................... 213 Consumer NAS ....................................................................................................... 214 Leaders of the Pack ........................................................................................ 214
IBM ................................................................................................................... 215 Hewlett Packard .............................................................................................. 219 Microsoft .......................................................................................................... 221 NetApps ............................................................................................................ 222 Cisco ................................................................................................................. 223 De11 ................................................................................................................... 224 EMC .................................................................................................................. 225 Hitachi .............................................................................................................. 226 Sun Microsystems ........................................................................................... 227 QLogic .............................................................................................................. 227 Emulex ............................................................................................................. 228
xiii
Contents
Comparisons ......................................................................................................... 229
What Did I Buy? ................................................................................................. 230
Summary .................................................................................................................. 232
Chapter 12 Making the Business Case for Storage Networks ................................................. 233
Use Cases .............................................................................................................. 234
The Value of the Hardware Replacement Use Case ...................................... 237
Use Case Description ...................................................................................... 237
The Value of the Server Resource Administration Consolidation Use Case .................................................................................. 240
Structuring the Feasibility Study Template ...................................... 240
Objective .......................................................................................................... 241
Terms ofReference .......................................................................................... 241
Existing Situation ............................................................................................ 242 Problem Statement ......................................................................................... 242
Proposed Solutions ......................................................................................... 242
Recommendation ............................................................................................ 242
Gross Benefits .................................................................................................. 243
Estimated Costs ............................................................................................... 243
Net Benefits ..................................................................................................... 243
Building Business Cases ............................................................................... 243 Begin at the End .............................................................................................. 244
The Objective ................................................................................................ ... 244
Terms ofReference .......................................................................................... 244
Server Consolidation ...................................................................................... 245 A Tale of Two Companies ............................................................................... 246
Analyzing the Existing Situations ................................................................... 248 Analyzing the Problem Statements .......................................................... , ..... 249
The Problem in Context ................................................................................. 250
Proposed Solutions .......................................................................................... 251
Doing Nothing As an Option .......................................................................... 251
Unless ............................................................................................................ 251
Proposing Storage Solutions .......................................................................... 251
Segregating the Impact ................................................................................... 252
The Decision Matrix ........................................................................................ 252
Proposed Solutions for Company B ...... : ........................................................ 254
Recommendations .......................................................................................... 254
The Financial Case .......................................................................................... 255
Quantifiable Gross Benefits ............................................................................ 255
Nonquantifiable Gross Benefits ..................................................................... 256
xiv
Contents
Estimated Costs ............................................................................................... 257 Net Benefits ..................................................................................................... 257 Company B: Another Set of Requirements ................................................... 258 Is IT Feasible? ................................................................................................... 259
Virtualization As a Benefit ...................................................................... 259 Summary .................................................................................................................. 260
Chapter 1.3 Future Directions for Storage Networks ... 261
Third Chip from the Sun ............................................................................... 262 ASICs Get the Job Done ................................................................................. 263 Microdrive Technology .................................................................................... 264 Silicon Photonics ............................................................................................ 264 Quantum Cryptography ...................................................................................... 265
Quantum Computing ............................................................................................ 266 Exploding Demand from Shrinking Devices ............................................ 267 Virtualization ................................................................................................... 268 Organizing the Circus .................................................................................... 269 Autonomics ............................................................................................................ 270 Summary .................................................................................................................. 273
Afterword .......................................................................................................... 275
Index .................................................................................................................... 277
XV
About the Author
Daniel Worden has been an IT manager, architect, and administrator since 1982. He obtained his first Unix root account in 1984 working with Pyramid and Sun equipment. His first exposure to fibre channel attached storage arrays was in 1996 while providing systems administration services to a multinational oil company. He has also been hands-on in retail, hospital, and video production environments.
During the 1990s, Daniel and his wife Marie built a 25-employee software and services company, which they sold to a publicly traded firm in 2001. He currently offers his consulting skills for interesting projects as they arise. Recently, he published a series of tutorials on autonomic computing tools.
Storage Networks is his fifth computer book. Previous work has been translated into Chinese and Japanese. More information about the author is available at http ://daniel.worden.net.
xvii
About the Technical Reviewer
Jonathan Hassell is a systems administrator and IT consultant residing in Raleigh, NC. He is currently employed by one of the largest departments on campus at North Carolina State University, supporting a computing environment that consists of Windows NT, 2000, XP, Server 2003, Sun Solaris, and HP-UX machines.
Hassell has extensive experience in networking technologies and Internet connectivity. He currently runs his own web hosting business, Enable Hosting, based out of Charlotte, NC. He is involved in all facets of the business, including finances, marketing, operating decisions, and customer relations.
Jonathan's previously published work includes RADIUS, published by O'Reilly and Associates; and Hardening Windows, published by Apress. He has also written monthly columns for the Wmdows 2000 Magazine Network and WindowsiTSecurity.com. His work has been published in CMP's Publish magazine and Pinnacle's Lima: AppDev newsletter, and he is a frequent contributor to Security Focus, the leading online security magazine, and PC Pro, a renowned computing magazine in the United Kingdom.
Hassell's latest book, Managing Windows Server 2003, will be published by O'Reilly and Associates in early 2004.
xix
Acknowledgments
IN nm CREATION of a work like this book, there are a great many people involved, none more than Jim Sumser, in every step from the initial gleam on. I couldn't possibly thank everyone by name who contributed in important ways to the writing, editing, and production of this book. Thank you all!
I would like to take this opportunity to doff my cap in thanks to the entire Apress team-you are a truly exceptional group: Ami Knox, for your improvements to my prose, punctuation, and capitalization, grazie, mille grazie; Beth Christmas, from expediting my contract through the most genteel and effective project shepherding I've ever before enjoyed; Jonathan Hassell, your even-handed praise and critique both encouraged and challenged me to bring out my best in this book; and two more "Is" on the team, Janet Vail in Production and Julie Miller in Marketing-thank you both for your efforts.
Alexander Worden's keen eye and deft hand resulted in some of the better diagrams in the book-thanks, kiddo.
Carla Bayha, without your introduction to Studio B, no one would be reading this. Thanks so much. And thank you, Laura Lewin, though agents never get the thanks they deserve.
The greatest acknowledgement goes to Marie, my spouse and partner, for the myriad ways she was and continues to be the best helpmeet ever, and to those family and friends who encouraged and supported me through this process.
Of course, for all the support and diligent review by others, any flaws in conception or construction of this book remain my responsibility alone.
xxi
Introduction
THANK YOu FOR picking up Storage Networks. When I was initially approached about writing this book, my first thought was the Fortune 500 market for Storage Area Networks (SANs) was pretty well serviced already. Then it dawned on me that twin revolutions in both storage and network technologies were opening up new opportunities and simultaneously presenting a variety of new issues for IT professionals to overcome.
This book, Storage Networks, is the result of my efforts to make sense of the explosion of products that increase network speed, link more nodes, and store ever-greater amounts of data over geographically disparate areas.
More than anything else, my objective for this book was to organize those issues and break them down into comprehensible, related pieces. I wanted to evaluate the components of Storage Area Network solutions so you, the Reader, would be better equipped to plan, implement, and administer this critical resource in your own environment.
To accomplish this, I felt I had to link theory and practice, SAN and NAS, Ethernet and Fibre Channel into a cohesive treatment of the technologies as an integrated set. A tall order to be sure.
Along the way, my review of technology standards in one chapter led me later to the selection of certain configuration options in proof-of-concept testing. These allowed me to achieve returns in performance tuning of 20 percent and more.
The theoretical concept informs the practical approach, and this book reflects that dynamic.
Rather than get immersed in the depth and breadth of any one component of storage networks, my aim here was to pull it all together for you. I wanted to deal with the "down and in" detail of each layer of the ISO network model as it applies to storage networks, and relate that to an "up and out" review of vendor offerings, data life cycle, and business considerations.
By the time you finish the book, I expect you'll be comfortable with the barrage of tech-spec numbers associated with any storage network product, and be better able to determine how it could fit within your environment.
If you're someone who's been tasked with implementing or administering a storage network, I kept your interests firmly in mind as well. Likewise, I hoped to appeal to anyone looking to get grounded in the technologies for the first time. Those of you with many years of experience in the field may also find a few gaps in your knowledge of disk and networking technology filled in.
xxiii
Introduction
:xxiv
Where possible, I related ideas and concepts to products on the market when I wrote the book, but I made an effort to make the discussion relevant beyond the latest release from any particular vendor. You should find the book applicable to Storage Area Network technology for quite some time to come.
In any case, this book was meant to be read rather than referenced, and I hope you find it an instructive companion on your way to grappling with storage network technology.
-Daniel J. Worden
