Datacenter Health CheckTerri-Lynn ThayerAVP/CIO, Computing & Information ServicesEDUCAUSE Enterprise Technology Conference, May 2007

Copyright Terri-Lynn B. Thayer 2007

This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Presentation Overview Datacenter blood pressure is rising Pre-assessment work Reliability goals Assess existing space, power,

cooling, fire suppression, & security Assessment results & options Recommendations for a renovation Key considerations in site selection

Datacenter Blood Pressure is Rising Growing Demands

Increased number of servers Increase in breadth of customer base – taking in

previously distributed computing Technology Changes

Storage growth Power/cooling needs – new boxes are smaller but they

are energy hogs and they are hot, hot, hot (10X power for a fully populated rack and 3-4X power to cool it down)

Business Resumption Concerns 24 x 7 demands Well publicized disasters - 911, Katrina Many university datacenters today lack standby power

generation Research Support

National trend for R1s to take on more support centrally Our Datacenters are Old

Pre-Assessment Work Capacity Planning and Growth Analysis

Determine a planning horizon Identify services likely to be provided from

your datacenter during that time period Business Continuity and Disaster

Recovery Objectives How long can your University operate

without a functioning datacenter? Do you have a cold or hot site? Should you consider a multiple datacenter

approach?

More Homework Research computing support decision

is fundamental Review your insurance

How much do you have Other requirements of your insurer

Get professional help with the assessment Involve your university facilities engineers Seek advice from outside professionals

who are familiar with modern datacenter design and operation

What cost/risk profile is your institution comfortable with? Reliability goals

Numerical Rankings Terminology Summary Definition

(1) Unreliable Shared building power and cooling; no generator

(2) Partially Isolated, Unreliable

Dedicated power system; shared cooling system; unconditioned power; non-redundant air conditioning; no generator

(3) Isolated Unreliable Dedicated power and cooling systems; unconditioned power; non-redundant dedicated air conditioning units; no generator

(4) Isolated Conditioned Dedicated power and cooling systems; conditioned power; non-redundant dedicated A/C units; no generator

(5) Isolated Improved Dedicated power and cooling systems; uninterruptible power system; non-redundant dedicated A/C units; no generator

(6) Isolated, Mostly Reliable

Dedicated power and cooling systems; uninterruptible power system; redundant dedicated A/C units; no generator

(7) Reliable Dedicated power and cooling systems; uninterruptible power system; redundant dedicated A/C units; generator

(8) Reliable Redundant Dedicated power and cooling systems; redundant UPS systems; redundant dedicated A/C units; redundant generators

(9) Ultra-Reliable Redundant power train; redundant cooling system; redundant UPS systems; redundant dedicated A/C units; redundant generator systems; redundant fuel system

(10) State of the Art

Redundant power train; redundant cooling system, redundant UPS systems, redundant dedicated A/C units; redundant generator systems; redundant fuel system; site hardened for weather and geographic exposures; location minimizes exposure to jurisdictional closure from hazardous spill, terrorism, or similar risks.

BRUNS-PAK Data Center Reliability Ranking

BRUNS-PAK • 999 NEW DURHAM ROAD • EDISON, NJ 08817(732) 248-4455 • Fax: (732) 248-3644 • http://www.bruns-pak.com

Datacenter Evaluation

Space Electrical System Mechanical System Fire Protection System Security

Space Square footage of conditioned

space Raised floor

Access Elevators Door size

Machine room layout Furnishings, racks, command center Is the space expandable? Is this a multi purpose facility?

Electrical/Power Considerations Source and costs Patch Panel/Power Control Units/ Power

Distribution Units Standby power

Uninterruptible Power Supply (UPS) Redundant/non-redundant Battery type (wet vs dry), capacity, and monitoring

Generator Type Power and cooling

Are the systems expandable? Delicate balance

Other Surge protection, lightning protection, grounding

Mechanical Systems Evaluation Cooling & humidity control

Chilled water – do you have a dedicated chiller?

Computer Room Air Conditioner - CRAC units - # and location

Capacity and reliability Heat Detectors

Airflow distribution Water sensors

Fire Protection Systems Detection

Smoke alarms Heat detectors Air sampling

Abatement Halon system (production banned in

1994) Full flooding clean agent system

FM-200, NAF, Inergen etc

Sprinkler system Wet Pre-action

Security

Physical access to the facility elevators & doors Caged areas and visitors Multi-purpose facility Door access system

Windows Monitoring

Closed circuit TV

Ancillary Services and Support Ancillary Services

Tape storage Secure storage/staging Paper storage Test/setup lab Printers and print support General storage room Break room

Other staffing and services which are provided from your datacenter

Machine hosting and associated SLAs others University departments groups external to the University

Assessment Results & Options Most of us will find that our

datacenters are not adequate for the anticipated growth over the next five years

Majority will identify power and cooling as the most significant issue Cooling and the power to cool will be

the number one issue Space constraints will be the

runner up

Options

Renovate Build a new datacenter Both of the above Multi-datacenter campus Outsource or Hosting

Recommendations for Renovation Implement standby power generation

capable of supporting both power and cooling

Remove ancillary services from machine room and relocate to other spaces

Trade off between space and density is a complex issue High density racking results in significant

heat and power provision issues It is general cheaper to provide more space

than to keep a small space with high density equipment adequately powered and cooled

Improve Air Flow & Circulation Provide additional space between racks

to promote air circulation Open up plenum space by relocating

cabling to overhead trays Increase height of the raised floor if

possible Consider new cooling solutions and

rack technologies – everything old is new again Chilled water is far more efficient than cool

air for heat removal Reconfigure the layout to implement a

“double hot aisle/cold aisle” configuration

Distribute high density racks

Double Hot Aisle/Cold Aisle hot aisle/cold aisle layout is

where cold air is segregated in front of equipment cabinets and hot exhaust air is expelled behind equipment cabinets. This layout eliminates the direct transfer of hot exhaust air from one system into the intake air of another system

Double A CRAC unit is located between two

hot aisles

Site Selection for a New Facility An opportunity to consider cost

Corporate world has moved their datacenters in some cases quite remote from the rest of their operation which allows them to consider

Power costs Real estate cost Labor costs

Look for a Old Supermarket Single story Single use facility Slab Few windows Lots of open space around the

building Loading docks and delivery truck

access

Other Considerations Voice and data connectivity Flooding and other weather

related issues If moving to a multi-datacenter

approach as part of a business continuity plan then consideration should be given to put the two datacenters at a sufficient distance to reduce dependence on the same power grid and to minimize weather and other regional disasters

Staffing Implications Data Center Managers will need to be

more skilled in the area of environmental issues, engineering, and server technologies

Facilities organizations may need to devote more time and specialization to cooling and power technologies related to the datacenter

Managing data center renovation or build projects will be resource intensive and may result in downtime for key services

Go Green Reduce energy costs (datacenter build

may be more expensive) Legislation Environmental concerns and

institutional plans to reduce carbon emissions

Vendor products Rack and server cooling technologies CO2 for cooling, DC power systems

Design Considerations Solar panels and wind energy Heat recycling

Conclusion Our datacenters are under significant stress If we take a close look we will find that most of

us will experience power and cooling problems in the near future. Cooling and the power to cool will be the most substantial issue we face.

There are new technologies and best practices which will provide some relief

Many of us will build new datacenters over the next five years and we should consider remote locations, outsourcing, and green IT solutions

These project will require both significant financial resources as well as IT and Facilities staff time. We may need to employ new skill sets

It is highly recommended that you engage professional assistance to evaluate your facility and to assist in renovation and new build designs

Additional Resources

www.stonesoup.org Past Meetings Spring 2006 meeting Data Center Futures Workshop

Presentations