March 2003 Copyright University of Michigan How the University of Michigan’s SAN Pays For Itself University of Michigan Administrative Information Services Chris Wood Nancy Medd March 26, 2003 Copyright University of Michigan 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.
Transcript
March 2003 Copyright University of Michigan
How the University of Michigan’s SAN Pays For Itself
University of
Michigan Administrative Information Services
Chris Wood Nancy Medd
March 26, 2003Copyright University of Michigan 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.
March 2003 Copyright University of Michigan
About the University of Michigan
• Founded in 1817
• Three campuses – Ann Arbor – 19 schools & colleges,
University Hospitals & Health Centers Dearborn – 4 schools & colleges Flint – 4 schools & colleges
• Research & Educational Units 35 Centers 18 Institutes
University of
Michigan Administrative Information Services
March 2003 Copyright University of Michigan
University Profile—Fall 2002
Student enrollment – 54,131
Faculty – 5,977
Staff – 23,123
University of
Michigan Administrative Information Services
March 2003 Copyright University of Michigan
Introduction
Michigan Administrative Information Services
• Administrative computing arm of the University of Michigan
• NOT login servers, research, student systems, hospital…
March 2003 Copyright University of Michigan
Pre-SAN Environment …
• PeopleSoft Financials Higher Education Human Resources
• Oracle (55 databases)
• AIX (35 servers)
• Windows (61 servers)
March 2003 Copyright University of Michigan
… Pre-SAN Environment
• 22 Terabytes of attached storage 14 TB unprotected 8 TB protected with mirrors (3.5 TB usable)
• 2100 SSA disks
• 13 TB occupied
• 9 TB allocated but not occupied
• 58% of allocated storage was occupied
March 2003 Copyright University of Michigan
March 2003 Copyright University of Michigan
Problems We Faced…
• Machine room was full (floor space and UPS)
• Large numbers, sizes and speeds of disks were hard to manage and inventory
• Disk failures were more common as the number of disks increased (129 failures in the past 12 months)
• Hardware maintenance costs increased
March 2003 Copyright University of Michigan
…Problems We Faced
• Databases were multiplying, both in size and number
• Unused storage was rarely where it was needed
• Database replication between hosts was slow
• Disaster Recovery was increasingly important
March 2003 Copyright University of Michigan
We Purchased a SAN
SAN=Storage Area Network
• Smart disk arrays
• Fiber channel directors
• Host adapters connecting all of the pieces
March 2003 Copyright University of Michigan
Post-SAN Environment
• PeopleSoft, Oracle, AIX
• 2 McData directors
• 3 IBM ESS 800s (sharks)
• 19 TB of storage, all protected 352 disk on sharks 32 SSA disks 70% occupied
• FlashCopy on one shark
March 2003 Copyright University of Michigan
March 2003 Copyright University of Michigan
The Implementation
March 2003 Copyright University of Michigan
Where we are in the Implementation
• Completed design (3 months)
• Halfway through implementation (3 months into 6 month plan)
• All test and development servers are complete
• Some production servers have moved
March 2003 Copyright University of Michigan
Implementation Timeline
March 2003 Copyright University of Michigan
Implementation: What We Expected
• Steep learning curve to master management of the new equipment
• Lots of mistakes in the initial configuration• Migration process would quickly become
trivial • Minimal disruption to hosts• Lots of DBA resistance• Not many people outside system support
and database administration would care
March 2003 Copyright University of Michigan
Implementation Surprises: Bad…
• Machine room couldn’t handle the new equipment Insufficient floor space Not enough UPS power Floor wasn’t strong enough
• Migrations never became trivial
• More outages than expected Software installation Data moves
March 2003 Copyright University of Michigan
…Implementation Surprises: Bad
• Security concerns caused us to drop some servers from the SAN plan
• Aggressive implementation schedule left little time to train remainder of the team
• Database administrators were suspicious of both reliability and performance on the SAN
• Many people in the organization were interested in all of the details (the spotlight is on)
March 2003 Copyright University of Michigan
Implementation Surprises: Good…
• Many people in the organization were interested in the SAN
• It was easier than expected to learn to manage the new hardware
• There were no large scale design mistakes
March 2003 Copyright University of Michigan
…Implementation Surprises: Good
• The changes that forced outages for each data move allowed us to rethink our data layouts
• We could migrate some test databases during scheduled refreshes
• While the database administrators are still skeptical, they’re softening
March 2003 Copyright University of Michigan
Implementation: What We’d Do Differently
• Address security earlier to avoid planning moves for machines that never should have moved to the SAN
• Spread the implementation over more time
• Train all system support staff earlier so the workload could be distributed
March 2003 Copyright University of Michigan
Changes to the way we work
• Restructured databases so ongoing space management is easier
• Increased thresholds for when we add space since it is easier and faster to allocate SAN storage
• Developed new storage request tracking procedures
• Began to proactively order disk
March 2003 Copyright University of Michigan
Scorecard
How did we do?
March 2003 Copyright University of Michigan
Problems we didn’t solve…
• Storage needs are still increasing• Database replication isn’t, in general,
faster Didn’t buy FlashCopy for all sharks
Software is expensive Additional disks to support FlashCopy are
expensive Additional disks would have meant more
frames FlashCopy only works within a shark
March 2003 Copyright University of Michigan
…Problems we didn’t solve
• Our DR situation hasn’t improved; however, we expect it to once the SAN implementation is complete.
March 2003 Copyright University of Michigan
Problems we solved…
• Machine room space and power situation has improved
• Storage management is much easier since software, not hardware controls it all
• Increasing storage needs are met with fewer physical disks Larger capacity disks Less wasted space
March 2003 Copyright University of Michigan
…Problems we solved
• Disk failures are less common (0 in 5 months)
• Saved money on maintenance Prepaid, so prices are locked in Maintenance costs are lower on new disks
than old Fewer physical disks reduces costs
• Nightly replication of production databases is faster
March 2003 Copyright University of Michigan
Unexpected Wins
• Many fewer hardware error calls to system support
• Fewer outages
• Maintenance windows are shorter because systems boot faster
• Performance features in the disk arrays allow use of much larger capacity disks
March 2003 Copyright University of Michigan
Features for the Future
• Peer-to-Peer Remote Copy to copy one shark to another for disaster recovery
• LAN-free backup to move backup traffic from the LAN to the SAN
• Server-free backup to move backup processing from servers to a dedicated SAN-aware backup engine
March 2003 Copyright University of Michigan
How the SAN pays for itself…
March 2003 Copyright University of Michigan
… How the SAN pays for itself …
• Projected SAN payback of 4.5 years
March 2003 Copyright University of Michigan
… How the SAN pays for itself
• We’re ahead of schedule for payback Using disk space more efficiently Requiring justification for disk requests
results in better storage planning Increased thresholds for when we add space
since it is easier and faster to allocate SAN storage
