Liquid cooling and heat re-use experiences · Cooling Total Liquid Cooling •Air is passed through...

transcript

Liquid cooling

and heat re-use

experiences

Gert SvenssonDep. Director PDC, KTH

November 11, 2015

PDC Center for

High Performance Computing

Contents

• PDC Overview

• Heat re-use at PDC

• Liquid cooling

• Case studies

PDC Center for

KTH and PDC

• KTH Royal Institute of Technology is the largest

technical university in Sweden.

• PDC Center for High Performance Computing is

the largest supercomputer centre in Sweden

located at KTH.

2015-11-11 3STOCKHOLM

PDC Center for

PDC’s Mission

• Operation of world-class:

• Supercomputers

• Data storage & long time data archive

• Provide world-class:

• User support

• Training

• Education

• mainly to Swedish academia

• Take part in world-class research in the area

PDC Center for

Funding

• Part of SNIC (6 centers in a meta center)

• Most funding from the Swedish Research Council (VR) and KTH

• Many EU projects

• Some industrial collaboration (mainly Scania)

• Note: Academic use is free

PDC Center for

How is a supercomputer build today?

Cluster

NodesSimilar to computerwithout keyboardand monitor

Processors

PDC Center for

PDC Main System: Cray XC40 Beskow

PDC Center for

Cooling at PDC background

• Listed buildings

• KTH located between the ”Royal

national city park” and the ”city

centre”

• District cooling since 2004

– Environmentally friendly (sea

water and central heat re-use)

– Has been fairly expensive but this

has changed recently

– Backup cooling needed

PDC Center for

Energy consumption at PDC

PDC Center for

First Heat Re-Use Project 2009 - 2014

• Cray XE6 600 kW

• Retrofitted an air cooled system with

water cooling

• Heating of one building (the Chemistry

• Heating the incoming air to the building

PDC Center for

Collecting high enough temperature

• Our computer room air condition (CRAC) units takes water

input of 8 °C and produce 18 °C

• Room temperature normally around 20 °C

• CRAC units not sufficiently high temperature

• The Cray XE60 computer two versions:

• Water cooled (too low temperature)

• Air cooled (air output 35 – 40 °C)

• Decided to purchase the air cooled version and design the

cooling ourselves

PDC Center for

Cooling the computer

• Hot air collected by

industrial air – water heat

exchangers

• Custom made chimneys

attach the computer

racks and the heat

exchangers

PDC Center for

Server Racks

35 - 44 °C 35 - 44 °C

Water 16 °C

~30 °C

16 °C

Air-Water Heat Exchangers

From: Computer Room Air Conditioners CRAC)

PDC Center for

Building to heat

• Nearby chemistry

lab undergoing

renovations

• Mainly heated by

• Needs larger

amount of air

because of

chemicals

PDC Center for

Transporting water to the chemistry lab

• Use existing pipes from the district cooling system

• Change direction of flow when in re-use mode

Chemistry Building

PDC HeatRecyclingLoop(winter)

PDC Center for

Water distribution

• One heat exchanger for

re-use

• One for district cooling

• If everything fails use tap

waterDistrict Cooling System

Always external supply only in this section Own supply(winter)

PDC Computer Hall

CrayCRAC

18°C 19

°C33.5 °CExternal Heat Exchangers

16 °C

Tap Water 33

°C19°C

PDC Center for

Lessons learned

• Possible to build something custom-made but

lot of work

• Keep it as simple as possible

• Re-use existing infrastructure

• Different perspective in computer industry vs.

building industry

• Requires cold climate but

• Heat can be made into cold using absorption

chillers

PDC Center for

Evaluation Winter 2012 - 2013

Saved 540 MWh district cooling

Saved 270 MWh district heating

Saved 50 kEuro per season

Saved 50 000 kg CO2 per season

-Oct-1

-Jan-1

-Mar-1

Recovered power (kW) Outdoor temperature (°C)

PDC Center for

Second heat re-use project

• New Cray XC40 700 kW

• Water cooled

• Central heat re-use facility with heat pump

PDC Center for

Cray XC40 water cooling (hybrid cooling)

• CPU board is air cooled but the air is cooled by heat

exchangers and fans in the adjacent racks

• Simple solution but with lower water temperature

and some energy loss by the fans (more like cooling

doors)

PDC Center for

Cray XC40 water cooling

PDC Center for

Central heat re-use at KTH

PDC Center for

Experience

• Supercomputers excellent

heaters

• Almost constant heat

• Heat re-use really works and

pays off•Energy to KTH reduced

23,700 MWh/year (25

•CO2 reduced 890 000

kg/year (44 %)

PDC Center for

Why liquid cooling

Liquid cooling is being adopted for a variety of

reasons:

• Silence (No fans)

• Resilience (More stable, efficient cooling)

• Efficiency (Fans replaced by pumps)

• Environment & Cost

– Higher temperature -> More free cooling

– Higher temperature -> Possibility for heat re-use

• Total Cost of Ownership (TCO)

• Hostile or Complex Environments (Not

depending on clean air)

PDC Center for

Different types of cooling

Air Cooling

Indirect Liquid Cooling

Direct Liquid Cooling

Total Liquid Cooling

• Air is passed through servers and then through a rear door or in row air-water heat exchanger.

• Liquid is taken direct to some components

• Fans are still needed.

• All components are cooled directly by liquid. Air side losses are minimised.

• Inside the DC, the system has no fans and breathes no air.

Slide courtesy by Iceotope

PDC Center for

ASHRAE Water Cooling

American Society of Heating, Refrigerating & Air-Conditioning Engineers

PDC Center for

Direct Liquid Cooling Widely Available

• Cooling with liquid more or less close to the

CPU/memory etc.

• Advantages:

• Low extra energy used for cooling (liquid pump)

• Efficient cooling – possibility to run CPU faster

• Depends of how close to CPU

• Higher liquid temperature than going through

• Free-cooling for longer time of the year

• Possibility for heat re-use

PDC Center for

PUE is not that important!

Heat re-use gives

more savings than

improving PUE

PDC Center for

Problems and some solutions

• Corrosion and bacteria in hot water• Filters, devices to take away air, chemicals, black

magic…

• Computers short life time (4 years vs. 10 -30 for infrastructure)

• Lack of standard for liquid cooling

• Different temperatures and pressures

• Different requirements on clean water

• Flexible solutions required!

PDC Center for

Experience

• High temperature is

everything• Collect heat close to the CPU

• Best: Direct Liquid Cooling

• Encapsulate the heat

• Don’t mix high and low temp

PDC Center for

Case study: HP Apollo 8000

• Closed passive heat pipes on the CPU cards

• Heat is transferred via heat transfer to a separate

liquid loop in the rack

PDC Center for

Heat Pipe Demo

PDC Center for

HP Case study continued

PDC Center for

HP case study continued

PDC Center for

HP case study.

• Modularized plumbing system

PDC Center for

2015-11-11STOCKHOLM 37

Another liquid: Mineral oil

• Green Revolution Cooling

• Fluid is dielectric

• Electronics can be submerged in

the fluid (not moving parts)

• One phase

• Heat is transported by circulation

• High temp possible (60C)

PDC Center for

Yet another liquid: 3M Novec

• Fluid is dielectric

• Electronics can be submerged in the fluid

• One phase

• Heat is transported by circulation

• Novec 7300 with boiling point 98°C is used

• Two phase system

• Boiling and Condensation

• Novec 649 with boiling point 49°C is used

• 3M claims the fluid is environmental safe (but I doubt)

• The fluid is very expensive

PDC Center for

Very simple submerged cooling

• Interconnect company doing cooling

on the side

• Box of Intel Phi with Extoll

interconnect

• Boiling Novec

• Water cooled coil

PDC Center for

2015-11-11STOCKHOLM 40

PDC Center for

Iceotope

PDC Center for

Iceotope: Secondary Coolant flow

45 degrees in

Up to 55 degrees out

PDC Center for

Rack with highest power

• RSC

• 400 kW/rack

• 1.2 PFLOPS

• 1024 Intel® Xeon Phi™ 5120D

PDC Center for

Lot of products available but not much

standardization

• Water temperatures

• Water quality

• Connectors

• ASHRAE is doing some

standards

Liquid cooling activity

PDC Center for

gert@pdc.kth.se

Questions?

Liquid cooling and heat re-use experiences · Cooling Total Liquid Cooling •Air is passed through...

Documents