High Density Data CentersFraught with Peril

Richard A. Greco, Principal

EYP Mission Critical Facilities®, Inc.

2

Microprocessors Trends

Reprinted with the permission of The Uptime Institute from a white paper titled Heat Density Trends in Data Processing,

computer Systems and Telecommunications Equipment Version 1.1

3

Power Consumptions will Continue to Increase

• More transistors and higher clock speeds will result in more power consumption which in turn lead to greater production of heat.

• Projections of power consumption increases by 2X every four years.

4

Power Consumptions

• The fact is that the majority of servers available today have high power consumptions.

3U 2U

Albatros Server DS/5U 1.3 kW

1U 3U BladeTotal Load (kW) 3.0 - 7.3 3.2 - 5.5 3.2 - 12.6 7.9 - 12.0 6.0 - 14.0

Technology Impact on 42U Enclosure

5U

14 - 28Number of Outlets 8 - 16 14 - 28 21 - 42 42

- - - -

- - .4 kW -

IBM X Series 360 .37 kW

Sun Fire B1600 - -

Compaq Proliant DL380 - -

IBM X Series 300 -

- - 1.0 kW

- - - -

.4 kW - -

5

Cooling Microprocessors

• Within three to five years, researchers at Intel Corp., Hewlett-Packard Co. and IBM predict computer makers will have to move beyond fans and adopt cooling mechanisms such as radiators and liquid cooling systems to avoid potential overheating.

6

How Will These Trends Impact the Data Center?

• High density data centers– Faster chips, more heat, more boxes

• Low to medium density data centers with hot spots– Faster chips, more heat, but fewer total

number of boxes

• Unknown effect

7

What is a High Density Data Center?

“A high compute density data center of today is characterized as one consisting of thousands of racks with multiple computing units…The heat dissipation from a rack containing such computing units exceeds 10kW.”

Source: Thermal Considerations in Cooling Large Scale High ComputeDensity Data Center, Patel et al.

8

What EYP MCF Believes is the Practical Limit to a High Density Data Center

• A high density data center is one in which the power to the raised floor and the computing equipment exceeds 150 watts/sf over the entire raised floor. This is equivalent to approx. 3.5 kW/cabinet in an efficiently arranged data center.

• This is the point where traditional all air data centers begin to have significant limitations.

Cooling Strategies for the Conventional Data Center

10

Example of a Traditional All Air Data Center at 150 w/sf

• 15,000 sf, 620 24x36 cabinets, 3.5 kW/cabinet, 2170 kW cooling required

• Cabinets on hot aisle/cold aisle – 16 ft aisle to aisle

• 23 or 24 CRACs (lay-out dependent) total at 110 kW each (375.3 MBH), 3 redundant

• 620 perforated tiles at 530 CFM eachmin. req.

• (24) 200 kW PDU’s (2N configuration)

11

Two Approaches to Cooling the Data Center

• We use Computational Fluid Dynamics (CFD) to determine the preferred arrangement– Option 1: CRACs on opposite walls – Option 2: CRACs in middle and on one wall

12

Option 1: CRACs On Opposite Walls

• 150 w/sf• Typical 15,000 sf

data center• 620 cabinets

@ 3.5 kW/cabinet• (24) 110.2 kW

CRAC units • (24) 225

KVA PDUs

13

Front In, Back Out, Hot Aisle / Cold Aisle Configuration for Cabinet

• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)

Cold Hot ColdAisle Aisle Aisle

Supply Air is 55º F Return Air is 85º F

14

Option 1: CRACs On Opposite Walls,3’ Raised Floor

15

Option 1: CRACs On Opposite Walls, 3’ Raised Floor – 2 Units Failed

16

Option 2: CRACs in Middle and On One Wall

• 150 w/sf• Typical 15,000 sf

data center• 620 cabinets

@ 3.5 kW/cabinet• (23) 110.2 kW

CRAC units • (24) 225

KVA PDUs

17

Option 2: CRACs in Middle and On One Wall, 3’ Raised Floor

Cooling Strategies forHigh Density Data Centers

19

High Density Data Center – 240 w/sf

• Typical 15,000 sf100’ by 150’

• 462 cabinets @ 7.9 kW/cabinet 3650 kW cooling

• 462 air grates @ 1200 cfm ea

• (40) 110.2 kW CRAC units

• (32) 225 KVA PDUs (2N)

20

High Density Data Center, 4 Rows of CRACs, 5’ Raised Floor

21

High Density Data Center, 4 Rows of CRACs, 5’ Raised Floor – 3 Units Failed

22

To Overcome Airflow Variations Use Fan Powered Cabinets

• Ensures airflow to cabinet regardless of pressure in floor and airflow through air grates, however re-circulation will occur if sufficient air from under floor is not delivered.

• Cabinet fans supplement server fans

23

Fan Powered Cabinets: Front In, Back Out, Footprint Vented, Fan Assisted

• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)

Supply Air is 55º F

Discharge Air is 85º F Raised Floor

Cabinet Fan

24

Fan Powered Cabinets: Front In, Back Out, Fan Assisted

• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)

Supply Air is 55º F

Discharge Air is 85º F Raised Floor

Cabinet Fans

25

• Air is drawn through the front of the cabinet (blue) and discharged through duct work (red)

Fan Powered Cabinets: Front In, Top Out, Fan Assisted

Supply Air is 55º F

Raised Floor

Cabinet Fans

Hot Air Plenum

Discharge Air is 85º F

Return Air Ceiling Plenum is 85º F Ceiling

26

Fan Powered Cabinets: Front In, Top Out, Footprint Vented, Fan Assisted

• Air is drawn through the bottom of the cabinet (blue) and discharged through the top (red)

Raised Floor

Supply Air is 55º F

Cabinet FansDischarge Airis 85º F

27

Raised Floor

Supply Air is 55º F

Discharge Airis 85º F Ceiling

Cabinet Fans

Fan Powered Cabinets: Front In, Top Out, Footprint Vented, Forced Ventilated• Air is drawn through the bottom of the cabinet

(blue) and discharged through duct work (red)

28

High Density Data Center without CRACs

• 325 w/sf • Typical 11,200 sf

high density data center

• 462 cabinets @ 7.9 kW/cabinet

• 462 cabinet coolers • (32) 225

kVA PDUs

29

Water Cooled Cabinet: Front In, Back Out

• Air is drawn through the front of the cabinet (blue) and discharged through the back (blue)

Supply Return Piping

Raised Floor

Cooling Coil

Cabinet Fans

Discharge Air is 72º F

30

Supply Return Piping

Raised Floor

Water Cooled Cabinet: Footprint Vented

Cool Air

Cabinet Front

Fan and CoilLocation underCabinet

CabinetRear

Building Floor

31

• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)

Refrigerant Cooled Cabinet

Raised Floor

Heat Exchanger

Refrigerant Pump

Discharge Air is 85º F

45º F Water from Chilled Water Plant

60º F

32

• Air is drawn through the front of the cabinet (blue) and discharged through the back (red)

Refrigerant Cooled Cabinet: Front In, Back Out

Raised FloorSupply Air is 55º F

Piping to Heat Exchanger

Discharge Air is 85º FOverhead Fan Coil

Supply Air

33

Inherent Problems with High Densities

• Air Distribution Problems– Inadequate raised floor height and excessive

openings in perforated tiles can result in high velocity airflows.

– Bypassing of air within cabinets– Re-circulation of air outside of cabinets

• Today users report a greater number of server failures in top 1/3 of cabinets

34

Inherent Problems with High Densities

• Cabinet fans are generally single power supply.

• Cabinet fan power should be from a separate PDU with redundancy.

• Identical cabinets may be required.

• UPS system must be larger or separated to incorporate fan power (up to 350 w/cabinet). For our example of 462 cabinets – 162kW of fan power.

35

Inherent Problems with High Densities

• What about the rate-of-change at the microprocessor level during transitions from utility to generators. To my knowledge this has not be addressed by the industry.

• Is uninterruptible cooling mandatory?– If so, what are the implications?

• Costs• Reliability• Redundancy• Maintainability

36

Other Considerations

• Numerous additional monitoring points.• Addition of cooling liquids in data center

environment.• Fire Zones.• Noise.• Seismic. • Computational Fluid Dynamic modeling is

required for design and recalculation when hardware changes.

37

Other Considerations

• It will be mandatory to have close coordination between facilities department and IT hardware planners.– Does not exist today

• Generally higher risks with lower margins for errors.

38

Prognostications

• “…late 1990’s requirements of 100 watts per square foot of average power now exceed 200 watts per square foot, and will double again by 2004.”

META Report - August 2, 2001 by Rich Evans

39

Prognostications

• “In fact, the 2004 400-watts-per-square-foot requirement will again double by 2008, so designers must work this into longer-range facilities plans so that space can be fully used, as opposed to 40 percent to 50 percent utilization because of insufficient power and cooling.”

META Report – August 2, 2001 by Rich Evans

40

Prognostications

• “… Data centers utilize much less than 50% of the physical and power infrastructure.”

Source: Cost of Overbuilding by Domenic Alcaro

41

Prognostications

• “Current estimates of data center power requirements are greatly overstated because they are based on criteria that incorporate oversized, redundant systems, and several safety factors.”

Source: Data Center Power Requirements: Measurements from Silicon Valley,

Mitchell-Jackson, Koomey, Nordman, Blazek

42

Prognostications

• “Furthermore most estimates assume data centers are filled to capacity. For the most part, these numbers are unsubstantiated.”

Source: Data Center Power Requirements: Measurements from Silicon Valley,

Mitchell-Jackson, Koomey, Nordman, Blazek

43

So What’s Real?

• Today’s data centers incorporate significant amounts of legacy equipment, slow change out due to economy.

• Many data centers have significant amounts of available space on the raised floor.

44

Current Load Densities

DATA CENTER TYPE WATTS/SF KW/CABINET

Enterprise 20 to 60 0.5 to 1.5

Internet/Co-location 40 to 90 1.0 to 2.0

Managed Services 60 to 110 1.5 to 2.5

High Density 150+ 3.5+

45

My View

• Most projections of watts/square foot are overstated.

• Millions of dollars will be spent on unused infrastructure.

• There no longer are real drivers for compaction.

• Generally there is ample raised floor available.

46

My View

• High density data centers will not become a reality.

• Low/medium density data centers will have hot spots due to high density computing equipment.

47

My View of What to Do

• Manage your hardware installations

• Don’t buy IT gear that is problematic

• Don’t build yourself a problem– Existing air delivery systems work– Highly engineered systems are expensive to

build and operate and significantly reduce reliability, redundancy and flexibility.

High Density Data Centers

Richard A. Greco, Principal

EYP Mission Critical Facilities®, Inc.

www.eypmcf.com