White Paper - Infrastructure in an energy-efficient data centre...This white paper, focusing on the...

White Paper - Infrastructure in an energy-efficient data centre Verena Schwalfenberg Dipl.-Ing. Markus Schmidt

Copyright © 2008

All rights reserved.

Rittal GmbH & Co. KG Auf dem Stützelberg

D-35745 Herborn

Phone +49(0)2772 / 505-0 Fax +49(0)2772/505-2319

www.rittal.de www.rimatrix5.de

V10

Infrastructure in an energy-efficient data centre

Contents

List of figures ........................................................................................................................................... 3

Abbreviations........................................................................................................................................... 4

Executive Summary................................................................................................................................. 5

Introduction.............................................................................................................................................. 6

Energy-related problems in data centres ................................................................................................ 7

Infrastructure ....................................................................................................................................... 7

Development of electricity costs and consumption ............................................................................. 7

Reasons for the increasing energy consumption.............................................................................. 10

Concepts and options for optimization .................................................................................................. 13

Power supply and distribution ........................................................................................................... 13

Cooling .............................................................................................................................................. 14

Room cooling................................................................................................................................ 14

LCP............................................................................................................................................... 16

Air/water heat exchanger.............................................................................................................. 16

Free cooling .................................................................................................................................. 17

Outlook .................................................................................................................................................. 19

2


List of figures Figure 1 Development of electricity costs (industrial) in Germany.......................................................... 7

Figure 2 Procurement costs for new servers compared to the costs for power & cooling...................... 8

Figure 3 Increase in server energy density ............................................................................................. 9

Figure 4 Relationship between operating costs over 3 years and server procurement costs ................ 9

Figure 5 Watts per US$ 1,000 for 1U servers ....................................................................................... 10

Figure 6 RimatriX5 efficiency monitor ................................................................................................... 11

Figure 7 Efficiency of a 40kW UPS ....................................................................................................... 12

Figure 8 RimatriX5 “Pay-as-you-grow” concept .................................................................................... 13

Figure 9 Flexible scalability of PMC 200 ............................................................................................... 14

Figure 10 Raised-floor cooling with cold / hot-aisle design ................................................................... 15

Figure 11 Cold-aisle containment.......................................................................................................... 15

Figure 12 Cold-aisle containment & LCP Inline cooling ........................................................................ 16

Figure 13 Average annual temperatures in Germany ........................................................................... 17

3


Abbreviations

CO2 - Carbon dioxide

COP - Coefficient of performance

CPU - Central processing unit

IDC - International Data Corporation

IT - Information technology

kW - Kilowatt

kWh - Kilowatt hour

LCP - Liquid Cooling Package

MTTR - Mean time to repair

PDM - Power Distribution Module

PDR - Power Distribution Rack

PMC - Power Modular Concept

DC - Data centre

SI-EER - Site infrastructure energy ratio

UPS - Uninterruptible power supply

4


Executive Summary

This white paper, focusing on the infrastructure in energy-efficient data centres, provides an overview

of the basic energy problems in existing data centres. It explains which components in an

infrastructure place the biggest load on the energy budget.

Innovative solutions are needed as additional power consumption leads to increased waste heat,

which in turn has be cooled using energy-intensive methods. The ever-rising electricity costs also

make this a top priority. It is predicted that by 2010, electricity and cooling costs will account for 50%

of the total costs of a data centre1.

There are a number of reasons for this rise in energy consumption, including the growing demands on

computing power and the continuous increase in the packaging density in the server area. At the

same time, there is also huge potential savings offered by e.g. IT chillers, which often remains

untapped.

RimatriX5 is one solution that helps improve energy efficiency. It is a modular concept from a single

source that meets the demands for an energy-efficient data centre infrastructure.

5

1 Source: International Data Corporation (IDC) 2006


Introduction

Energy efficiency is one of the key terms used in IT today in conjunction with data centres. There are

numerous reasons for this development. The increasing demands on computing power, rising server

packaging densities, more waste heat, CO2 emissions and, above all, the huge rise in energy costs

automatically push for energy-efficient solutions to help cut electricity consumption.

A few years ago, the day-to-day energy costs for data centre operation played a relatively insignificant

role. More emphasis was placed on procurement costs for the IT hardware, such as servers, storage

systems and network technology. Over time, this technology has become increasingly powerful, which

has caused power consumption to rise accordingly. It is now clear that this level of power consumption

leads to considerable costs and problems. That is why manufacturers of IT hardware have

increasingly been integrating energy-saving mechanisms into their products. These include flexible

clock rates, lower operating voltages and sections of processor chips that can be deactivated as

necessary.

So far, little attention has been paid to the data centre infrastructures used to provide and distribute

the energy required by the IT hardware and to dissipate waste heat. In a worst case scenario, this

energy consumption is higher than the consumption of the IT hardware itself. Therefore, innovative

concepts and solutions are also required in the infrastructure to boost energy-efficient operation.

6


Energy-related problems in data centres

Infrastructure

The infrastructure of a data centre must solve two basic problems in relation to energy:

Reliable provision of energy

Dissipation of the heat that is generated

These are both key cost factors in day-to-day operation and have a major impact on the efficiency of a

data centre.

In simplified terms, the efficiency of a data centre can be defined as follows:

DC

ITDC nconsumptiopower

nconsumptiopowerefficiency__

=

ytechnonetworkstorageservernconsumptiopower IT log__ ++=

)(__ chillercoolerPDMUSVnconsumptiopowernconsumptiopower ITDC ++++=

The efficiency of a data centre depends on the infrastructure technology required for the

uninterruptible power supply (UPS), power distribution (PDM), coolers and water chillers. The lower

the efficiency levels of these individual components, the poorer the overall efficiency of the data

centre.

If the infrastructure components exhibit a poor level of efficiency, this has a double impact on the

overall system – the energy costs rise and the server's power consumption increases.

Development of electricity costs and consumption

Figure 1 Development of electricity costs (industrial) in Germany2

Electricity costs for the commercial sector have been increasing steadily since 2000, and the trend

looks set to continue as a result of the worldwide demand for energy.

7

2 Source: Energieagentur NRW (Energy Agency NRW)


Furthermore, servers today consume more power than they did a few years ago despite state-of-the-

art architectures, which is causing the absolute consumption values for the infrastructure to skyrocket.

Figure 2 Procurement costs for new servers compared to the costs for power & cooling3 It is not surprising, therefore, that the power and cooling costs are assuming a bigger share of the

overall costs of a data centre. In Figure 2, the IDC forecasts that energy costs will account for around

40% of the total IT budget in 2010, while other experts predict that this figure may even rise to 50%.4

In addition to high energy consumption and the associated costs, increased energy consumption is

giving rise to a further problem in that existing data centres can no longer handle the power

requirements and waste heat.

3 Source: International Data Corporation (IDC) 2006

8

4 Source: (“CEO Guide to Green Computing”, businessweek.com 2007/05).


Figure 3 Increase in server energy density5

The energy density in data centres increased more than 10-fold between 1994 and 2004 alone. This

rise places extreme demands on the infrastructure of the power supply (UPS), power distribution and

cooling concept. This is because every watt of power supplied must also be removed from the data

centre in the form of waste heat.

Figure 4 Relationship between operating costs over 3 years and server procurement costs6 According to calculations from the Uptime Institute, the server operating costs skyrocket compared to

procurement costs when users continue to work with old concepts and structures. In the best case

(projection B)7 the increase is still considerable, but projection A8 is realistic. However, the old

infrastructures must be modernised to achieve these values.

5 Source: Datacom Equipment Power Trends and Cooling Applications, 2005

9

6 Source: Data Center Energy Efficiency and Productivity, Uptime Institute 7 State-of-the-art


Further evidence of an obsolete infrastructure is its SI-EER value. The site infrastructure energy

efficiency ratio is the ratio between the energy supplied and the energy that reaches the IT hardware.

It is the reciprocal value of EfficiencyDC.

According to the calculations of the Uptime Institute, this value came to an average of 2.5 in 2007.

This means that of the 2.5 watts supplied, only 1 watt reaches the IT hardware. In a data centre

covering 2,800 square meters, reducing the SI-ERR to 2.0 would cut energy costs by around

US$ 1 million a year.

Reasons for the increasing energy consumption

To be able to implement cost-saving measures and therefore run a data centre more efficiently, it is

important to identify the reasons behind the rising energy consumption.

The first and most obvious reason for this is the fact that the performance in the server area is increasing all the time. Moore’s Law9 states that the complexity of the chip circuitry doubles every 24 months. This doubling of integration density goes hand in hand with a significant increase in computing power. As a result of the growing demands, more and more servers are being operated in each data centre. Therefore, the total power consumption of data centres rises even though the value for “watt per computing power” falls significantly. This phenomenon becomes clear if you consider how many server watts could be purchased over the

years for US$ 1,000.

Figure 5 Watts per US$ 1,000 for 1U servers10

From 2000 to 2006 alone, the power per US$ 1,000 increased more than 13-fold. Even in best case

scenarios, the possible developments (projections A and B) predict a rise to 157 watt/US$ 1,000.

One reason for the considerable increase in recent years is the blade server. Its extremely compact

design means that a very high power density can be generated in the rack. A rack equipped with

several blade centres can easily result in power consumption greater than 20kW, which then has to be

cooled. With an SI-EER of 2.5, the data centre has an input power of 50kW for just one rack!

8 Best Practice

10

9 A law devised by Gordon Moore in 1965 that states that that the number of transistors that can be inexpensively placed on an integrated circuit increases exponentially, doubling approximately every two years 10 Source: Data Center Energy Efficiency and Productivity, Uptime Institute

http://en.wikipedia.org/wiki/Integrated_circuit

http://en.wikipedia.org/wiki/Exponential_growth


Many servers are bought to host one or two special services. These applications often only place a

load of between 10 and 20 percent on the server, which means that 80% of the CPU’s computing time

is spent on idle operation. This would not be a major problem if power consumption was scaled linear

to the computing load. Unfortunately, this is not the case. A server with only a 20% load needs around

75% of the energy that it needs in productive operation with a 70 – 80% load. This problem cannot be

solved by the infrastructure, but it places a tremendous load on it.

Figure 6 RimatriX5 efficiency monitor

The RimatriX5 efficiency monitor illustrates the problem with infrastructures that are oversized or not

operated to their full capacity. The zoom view shows that the efficiency of the data centre (bottom

graph, blue curve) drops rapidly to less than 10% when the IT load (top graph, green curve) is far

below the normal load.

11

During the planning of a data centre, the infrastructure is often designed to cater for maximum server

configuration. In other words, the capacities of the UPS and cooler in particular far outstrip the

requirements in the first few years of operation. In addition to causing very high procurement costs, at

worst the devices also operate at less than 30% of their maximum load and, therefore, with a

significantly lower level of efficiency.


Efficiency 40kW UPS

88.0%

89.0%

90.0%

91.0%

92.0%

93.0%

94.0%

95.0%

15% 30% 52% 76% 102%

Load

Effic

ienc

y

Wirkungsgrad

Figure 7 Efficiency of a 40kW UPS

When operated to 90% of its capacity, a 40kW UPS, for example, has an efficiency level of approx.

94.5%. However, if it is operated at just 25% of its capacity, the efficiency level drops to 91%.

With a power supply of 100kW, a drop in the efficiency level of 3.5 percentage points increases power

losses by 3.5kW. Over a year, this amounts to 30,660kWh or €3,066 (based on electricity costs of

€0.10 per kWh). This loss of €3,066 is caused purely by poor sizing, which also has to be paid for at

the investment stage (bigger UPS = higher procurement costs). Similar calculations can also be done

for cooling.

Solutions in this area must, therefore, ensure that components deliver optimum efficiency throughout

the service life of a data centre.

Cost savings Runtime kWh Saving [€]

1 year 30,660 3,066 5 years 153,300 15,330

10 years 306,600 30,660 Potential savings from improved UPS efficiency11

12

11 Reduced power loss of 3.5kW


Concepts and options for optimization

Power supply and distribution

As mentioned in the previous section, two main factors impact on the power supply of a data centre –

efficiency and sizing.

Sizing means choosing the right-sized components for the system, and it should be based on the

demand and the final expansion stage of the data centre. When data centres were planned in the

past, only the maximum configuration was factored into the equation “to be on the safe side”. This

configuration is often only achieved in the final years of a data centre – if at all.

To enable infrastructure components, such as UPS and power distribution, to operate at maximum

efficiency during the initial phase, they must be configured slightly smaller, with the option of

expansion at a later date. This balancing act can be realised with modular systems such as RimatriX5

from Rittal.

1

2

3

Figure 8 RimatriX5 “Pay-as-you-grow” concept12 Legend 1 – Current power consumption 2 – Scalable, demand-oriented and gradual adaptation of capacity (RimatriX5) 3 – Overcapacity configured with conventional DC technology x-axis: Service life in years y-axis: Installed infrastructure in % / room capacity Figure 8 demonstrates the pay-as-you-grow concept from RimatriX5. The components for UPS, power

distribution and cooling grow as the data centre expands.

This modular concept can be illustrated using the example of Rittal PMC 200 UPS.

13

12 Source: Rittal GmbH


Figure 9 Flexible scalability of PMC 200

The PMC 200 can be operated using three 20kW modules as an (n+1)-redundant 40kW UPS. If new

servers demand more power, there is no need to buy new or additional UPSs. The power output can

be increased simply by inserting another module whilst the system is operational. These modules are

available in several classes ranging from 8 to 40kW, enabling flexible adaptation to the power

consumption required. The modular n+1 redundant design offers energy-saving benefits for both

upgrades and operation. A second UPS is not required in the event of failure – an additional module

with less power loss is sufficient. The modular concept also boosts availability. During service

scenarios, a faulty module can be replaced in just a few minutes. This results in a very low MTTR

(mean time to repair). The state-of-the-art system design without a transformer cuts down on the use

of copper in production and makes for above-average efficiency of more than 95%.

The same modular concept with all its energy-saving benefits can also be realised for power

distribution. Rittal offers such a system with its Power Distribution Rack (PDR) and Power Distribution

Modules (PDM).

Cooling

Room cooling The second biggest factor that impacts on the efficiencyDC value is data centre cooling. Even with an

average SI-ERR of 2.0, a data centre with a 100kW IT load creates almost 200kW heat losses that

have to be cooled.

The traditional method is room cooling, whereby cold air is blown through the room – or better,

through a raised floor into the DC. In older data centres, this arrangement can cool a heat load of

between 1 and 2kW per rack; in newer, better planned DCs, this value increases to 5kW. However,

this solution demands very accurate implementation of the cold-aisle concept. Figure 10 shows this

type of cold-aisle scenario.

14


Figure 10 Raised-floor cooling with cold / hot-aisle design13 It is essential that hotspots or heat leakages, which can cause the cold and warm air to mix, are

avoided. This reduces the efficiency of the cooling output considerably and threatens the reliable

operation of the servers at these points.

Figure 11 Cold-aisle containment14

The next optimisation step is cold-aisle containment, whereby cold air is directed using additional

ceilings, walls and doors. This method is some 10-20% more efficient than traditional cold-aisle

cooling without containment.


15



Figure 12 Cold-aisle containment & LCP Inline cooling15

LCP If the cold air supplied via the raised floor is no longer sufficient, LCP Inline – i.e. cooling devices

inserted between the racks – is the next step to optimise the cooling efficiency. LCP Inline devices

from Rittal are the ideal solution, offering a high level of efficiency and excellent reliability. With a

Liquid Cooling Package (LCP), the air is cooled via an air/water heat exchanger, enabling very high

cooling outputs.

However, if several fully-equipped blade centres are installed in the racks, the cooling load in the

enclosure is too high for traditional or optimised room cooling concepts. In this case, rack-based

cooling is virtually compulsory. The encapsulated rack is cooled and does not affect the temperature in

the server room. The LCP Modular or LCP Plus system can be used to cool the rack. The cooling

output of these systems, which are also based on air/water heat exchangers, can be adapted and

extended step-by-step via cooling modules, in line with the RimatriX5 pay-as-you-grow concept. The

maximum cooling outputs for LCP Modular and LCP Plus are 20kW and 30kW respectively.

In very critical applications, a redundant design can be realised with the LCP system with 2 LCPs

being used to cool one rack.

Air/water heat exchanger In cooling solutions using air/water heat exchangers, the cooling water must be made available with a

specific inlet temperature and the heated water from the return flow must be cooled again. IT chillers

are usually used for this purpose. The power of the IT chiller must be adapted to the waste heat that is

generated. Rittal offers a finely-graded range with cooling outputs between 15kW and 462kW. These

power levels in themselves indicate that cooling requires an enormous amount of energy, coupled with

substantial costs and CO2 emissions.

16



Annual average temperatures in Germany In degrees Celsius

long-term average

linear trend

Thermometer values on the rise

Figure 13 Average annual temperatures in Germany16 Free cooling Free cooling offers good savings potential for IT chillers. A free cooling system uses ambient air to

cool the heated cooling water. This method can always be used to generate cold water when outside

temperatures are low.

Free cooling aims to cut operating costs and reduce greenhouse gases. Hot air is extracted without

the use of a compressor, which reduces the total electrical energy required by the IT chiller, thus

making it much more efficient.

In standard applications, the IT chiller can be operated in free cooling mode with an outside

temperature of around 10 degrees Celsius. On average, the temperature in Germany is below 10

degrees for almost 50% of the year. As air-conditioning in a data centre usually runs 24/7, the savings

potential is also almost 50%. In standard cooling mode, a 462kW chiller requires an input power of

96.25kW.

Cooling efficiency Q=462kW; coefficient of performance17: COP=4.8

Required input power for the IT chiller: kWkWCOPQP 25.96

8.4462

===

When the IT chiller is operated in free cooling mode, the required input power drops to around 10kW.

Based on an electricity price of 10 cent/kWh, this results in potential savings of almost €38,000 every

year!

16 Source: DWD / FAZ graphic

17

17 Thermal efficiency of heat pumps, identical with the term coefficent of performance


Sample calculation for IT chiller costs with and without free cooling

Pow

er c

onsu

mpt

ion

of

chill

er w

ith c

ompr

esso

r [k

W]

Ope

ratin

g ho

urs

per

yeea

r

Cos

ts [€

]

Pow

er c

onsu

mpt

ion

of

chill

er w

ithou

t co

mpr

esso

r [kW

]

Ope

ratin

g ho

urs

per

yeea

r

Cos

ts [€

]

Tota

l cos

ts p

er y

ear [

€]

Without free cooling

96 8,760 84,096 10 0 0 84,096

With 30% free cooling

96 6,132 2,452 10 2,628 2,628 61,495


96 4,380 42,048 10 4,380 4,380 46,428


96 2,628 25,228 10 6,132 6,132 31,36018

Compared to costs without free cooling, a system that uses 50% free cooling saves 376,680 kWh a

year, which is equivalent to €37,668/year.

This system works by feeding the water via an air/water heat exchanger to the outside air where it is

cooled by temperature balancing with the ambient temperature (<10°C). This process can be

enhanced by fans. This method is efficient because it dispenses with the need for a compressor which

would consume power.

18 The calculation is based on an electricity price of 10 cent/kWh

18

Outlook

The importance of energy efficiency for data centre operators will continue to grow. The need for

action will also increase in view of the costs, energy availability and environmental factors. RimatriX5

from Rittal is an innovative and modular concept from a single source that meets the infrastructural

requirements for a modern, energy-efficient data centre. The modular design of RimatriX5 offers a

wide variety of options for expanding and modernising existing data centres, regardless of whether the

operator wants to install a complete new cooling system or just a single rack.

The next optimisation steps for the RimatriX5 range are already in development and will ensure that

existing and new installations will continue to be energy-efficient in the future, too.

Date post:	12-Aug-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

White Paper - Infrastructure in an energy-efficient data centre...This white paper, focusing on the...

Documents