Managing Green Datacenters Powered by Hybrid Renewable Energy Systems

Intelligent Design of Efficient Architectures Laboratory (IDEAL)Department of Electrical and Computer Engineering

University of Florida

Presented by Chao LiICAC

Jun 20, 2014

Chao Li, Rui Wang, Tao Li, Depei Qian, Jingling Yuan

2 / 30

Power is a Big Issue in the Cloud Era

• Current global server market– 30 GW power demand– Doubles every 5 years [2]

Power Budget

2010

2015

2020

Everything is in the Cloud

Ever-increasing user data

Endless data processing

More servers are needed!

[1] http://oraclestorageguy.typepad.com/

2] Report to Congress on Server and Data Center Energy Efficiency, EPA, 2007

05

101520253035

Zett

aB

yte

s

Impending Data Explosion

Structured Unstructued

[1]

3 / 30

×30

4 / 30

[1] C. Belady, Projecting Annual New Datacenter Construction Market Size, Global Foundation Services, 2011

[2] DCD Industry Census 2012: Energy, http://www.dcd-intelligence.com/

[3] http://energyalmanac.ca.gov/electricity/total_system_power.html

USA

China

U.K.

Japan

Brazil

France

Benelux

Canada

Germany

Russia

8 TWh

3 TWh

2 TWh

2 TWh

2 TWh

1 TWh

1 TWh

1 TWh

1 TWh

1 TWh

Australia

India

1 TWh

1 TWh

The increase in server energy demand (2012-2013)[2]

• The global data center electricity usage in 2012: 300 ~ 400 TWh– 2% of global electricity usage– Expected to triple by 2020 [1]

Server Footprint Continues to Expand

302 TWhTOTAL ENERGYwas consumed

in CA in 2012 [1]

5 / 30

The Sustainability Issue: Energy Cost

[3] https://www.gov.uk/government/organisations/department-of-energy-climate-change

• Escalating energy consumption drives data center cost up– Need to think alternative power provisioning solutions

[1] Conference report: The Future of the Data Centre, http://www.information-age.com

[2] Ken Brill, The Economic Meltdown of Moore’s Law and the Green Data Center

0%

50%

100%

150%

200%

2000 2003 2006 2009 2012

The 3-Year Energy Expenditure(% of Total IT Equipment Cost)

[1,2]

[3]

20

40

60

80

100

120

140

160

180

Re

tail

Pri

ce I

nd

ex

Historical Electricity Prices in UK

Average Compared to 2005Benchmark for 2005

6 / 30

0%

20%

40%R

uss

ia

Fran

ce

Ital

y

Bra

zil

Spai

n

Ch

ina

Me

xico

No

rdic

s

Can

ada

Turk

ey

Be

ne

lux

USA

Ge

rman

y

Ind

ia UK

Jap

an

% Performing Carbon Monitoring

Hurricane Sandy, 2012(Northeastern US)

Typhoon Haiyan, 2013(Southeast Asia)

• The greenhouse effect & climate change

• 1MW data center → 10~15 Kt CO2 yearly

• Data centers are carbon-constrained: – They must cap carbon emissions

The Sustainability Issue: CO2 Emission

7 / 30

Greenpeace (http://www.greenpeace.org)

Regional Greenhouse Gas Initiative (http://www.rggi.org)

Union of Concerned Scientists (http://ucsusa.org)

[1] Energy & Environment Consumer Survey , Pike Research, 2012

Major Driving Power of Low-Carbon IT

• Non-profit organizations campaign for sustainability

• Government regulations and initiatives

Environmental Protection Agency (http://www.epa.gov/ )

EU Emission Trading Scheme (http://ec.europa.eu/clima )

California's SB X 1-2 Law: requiring 33% renewables by 2020

[1]• The green energy concept has mass appeal among consumers

Bio-fuel

Wind Energy

Solar EnergyFavorable

Very Favorable

69%

65%

54%

8 / 30

Carbon-Conscious Computing System Design

Many IT Companies start tointegrate non-conventional

clean energy solutions

Solar Panels Wind Turbines

Micro-TurbinesBio-mass

Fuel Cells

Batteries

9 / 30

Limitations of Prior Work

• Load management and supply management are decoupled– Must combine load management and supply management

• Only focus on certain specific type of renewable energy– Need to look at hybrid renewable energy systems.

10 / 30

Outline

Background: Hybrid Renewable Energy System for Data Centers

2. GreenWorks: A Framework for Multi-Source Powered System

1.

3. Multi-Source Driven Power Management for Data Centers

4. Evaluation and Discussion

MinuteskT (k+1)T

Sp

ee

dP

ow

er

Minutes

Load Power with

GreenWorks

11 / 30

Hybrid Energy Systems

• Baseload Power Supply– Biomass, gas turbine, etc.– Stable and controllable

Wind turbine

Gas turbine

Fuel cells

Solar panel

Energy storage

Utility

Power interface

Circuit breaker

Microgrid

Biomass energy

Central Controller

Diesel Generator

Server

Cluster

Transformer

ATS

UPS

• Intermittent Power Supply– Solar, wind, etc.– Time-varying output

• Backup Power Supply– Various batteries– Immediate response

12 / 30

Energy Balance Challenge

• Demand/Supply power mismatch problem– Requires fine-grained load and supply management

0 5 10 150

500

1000

1500

Time (hour)

Po

we

r (K

W)

Coarse-grained variation

0 60 120 180 2400

500

1000

Time (min)

Po

we

r (K

W)

moment-to-moment oscillation

• Cannot simply rely on any single type of power supply– Intermittent green power: cannot guarantee power output– Baseload power: too slow to follow the fluctuation– Battery: limited lifetime and capacity– Over-provisioning power to overcome the above issue?

13 / 30

GreenWorks: An Overview

• A synergy of various power supplies, as well as computing loads

Intermittent Pwr.

Baseload Pwr.

PDU

PDU

PDU

Server Racks

Server Racks

Server Racks

UPS

Sw

itch

ge

ar

Energy Source Infor.

Load Following Ctrl.

Load Power Infor

Pwr. Interface Circuit Breaker Perf. RecorderPwr. Modulator

Monitoring

& Ctrl.Stored Energy Infor.

AT

SPwr. Meter

UPS

UPS

Baseload

Laborer

Green ManagerGreenWorks

Mic

ro-g

rid

Ce

ntr

al

Co

ntr

olle

r

Energy

Keepers

Load

Brokers

14 / 30

Power Management Hierarchy

E

:

E E E

L

B

L

:

:

Tier-I: Datacenter Facility Level

(Adjusts Baseload Pwr. Supply)

Gre

en

Ma

na

ge

r

:

Tier-II: Cluster/PDU Level

(Manages Intermittent Pwr. Supply)

Tier-III: Rack Level

(Regulates Backup Pwr. Supply)

Power flow Ctrl. signal

::

• A multi-layer power integration and management strategy

15 / 30

Outline

Background: Hybrid Renewable Energy System for Data Centers

2. GreenWorks: A Framework for Multi-Source Powered System

1.

3. Multi-Source Driven Data Center Power Management

4. Evaluation and Discussion

MinuteskT (k+1)T

Sp

ee

dP

ow

er

Minutes

Load Power with

GreenWorks

16 / 30

VRM

Core Core

Core Core

VRM

VRM VRM• Use alternating processing speed

– Increase server performance with over-clocking (e.g. Turbo Boost)

– Give priory to jobs that are have higher anticipated ETI

• Maintain a lookup table, which contains– Job ID and job execution progress– Calculate anticipated execution time based

on the current progress and speed– ETI: Execution Time Increase (%)

ETI Job ID

DV

FS Ctrl.

. . .

Lookup Table

I/V se

nso

r

• Opportunistically Boost System Performance

Stage I: Adequate Power Supply Budget

17 / 30

• Calculate a discharge budget– The total energy that can be cycled through a battery is fixed

0

200

400

600

800

1000

1200

1400

0500

1000150020002500300035004000

10

%

20

%

30

%

40

%

50

%

60

%

70

%

80

%

90

%

10

0%

Th

rou

gh

pu

t (k

Wh

)

# o

f C

yc

les

Cycles to Failure Throughput

Manage solar energy usage based on

0

t

aggregated AhD D

/budget ratedD T Lifetime D

budget aggregatedD D

• Give load shedding priority if the discharge budget is low– Otherwise, use the stored energy to maintain server speed.

• Balancing load shedding and battery discharging

Stage II: Moderate Power Supply Drop

18 / 30

• Calculate a time budget, which evaluates:– whether a job could meet its deadline in the future with

frequency boosting techniques.

Time Budget = Remaining Runtime × D× P × S

S = μ ×(1 – 1 / Frequency Speedup) ;

D: the duty cycle of performing turbo boost

P: likelihood of receiving adequate renewable power

S: the execution time that 1s frequency boost can save

Stage III: Significant Power Supply Drop

• Deadline Driven, Power-Aware Load Shedding

• Choose load shedding if the time budget is enough– Otherwise, use stored energy to handle the power shortfall.

19 / 30

Managing Baseload at Coarse-Grained Intervals

Avg. UPS Capacity

Avg. Runtime Increase

Intermittent Power Infor.

Green manager

Baseload

Systems

Current Output Level

++∆ Baseload

Laborers

• Adjust the baseload power output based on – Moving average of the load power demand– Current power demand

• Can also add additional bonus power output– Based on the average battery capacity– Average performance of the workload

20 / 30

GreenWorks Power Management: A Summary

• Matching power demand to total power supply

Adjust Baseload Power

1. Abundant Power:Boost Load

2. Inadequate Power:Maintain Load

3. Power Emergency:Shed Load

Coarse-grained load management

Fine-grained load management

Energy storage

Balancer

Server Cluster

History Info.

CapacityController

Power cycling

DG

Po

we

r O

utp

ut New demand goal

Bonus budget

Ram

pin

g

Scheduled

adjustment

Current generation level

Green Manager

21 / 30

Outline

Background: Hybrid Renewable Energy System for Data Centers

2. GreenWorks: A Framework for Multi-Source Powered System

1.

3. Multi-Source Driven Data Center Power Management

4. Evaluation and Discussion

MinuteskT (k+1)T

Sp

ee

dP

ow

er

Minutes

Load Power with

GreenWorks

22 / 30

Verification Platform

• A three-layer simulation framework– Energy systems + Server systems + Job Scheduling

Power System Layer

Infrastructure Layer

HPC Job Scheduling Layer

Batch scheduler

Job queue

DatacenterModel

Profiler

HPC Traces[ Jon ID, subTime, waitTime, startTime, endTime, cpuNum, cpuTime…]

Server Power DatamaxFreq/staticPwr/dynPWr/ Turbo levels...

Resource TraceswindTimeSeriesData …

Wind Turbine Model

Baseload Power Model Battery ModelDischargeEvents

GreenWorksModulatorMonitor

AnalyzerPower Stats.

perf/pwr

Ctrl

Job Infor.

23 / 30

Evaluated HPC Data Center Traces

Data center workloads traces

Atlas

BlueGene

Thunder

DataStar

MetaC

iDataPlex

RICC

Seth

Short-running workloads

Long-running workloads

• Parallel Workload Archive– http://www.cs.huji.ac.il/labs/

parallel/workload/

Short job Inter-arrival time

Long job Inter-arrival time

24 / 30

Performance Comparison

0%

10%

20%

30%

40%

ETI

Shedding Boosting GreenWorks

• Less then 3% performance degradation, on average– Very close to an ideal battery based design (2.1%)

• About 12% performance degradation, for the 5% worst cases– Bettery than a our baseline, Boosting

25 / 30

Energy Efficiency

0%

1%

2%

3%

4%

5%

6%

S BG

W S BG

W S BG

W S BG

W S BG

W S BG

W S BG

W S BG

W S BG

W

Thunder DataStar Atlas BlueGene RICC MetaC Seth iDataPlex Avg.

Tota

lEn

erg

yLo

ss Battery Loss Inverter Loss

• Factors that affect energy utilization– Inverter energy loss and battery round-trip energy loss– GreenWorks maintains almost the same energy utilization

26 / 30

Battery Lifetime

• Provides better battery utilization– The lifetime of GreenWorks battery is very close to its

designated lifetime.

27 / 30

UPS Autonomy Time (Backup Time)

• On average, the stored energy level is 78% of rated capacity– Boosting: 70%. Shedding: 88%.

• Ensures a rated backup time for 20% of the time.– Better than Boosting but worse than Shedding.

0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

UPS Autonomy Time

Em

pir

ica

l C

DF

Shedding

Boosting

GreenWorks

Under-use

Over-use

28 / 30

Conclusions

• Renewable energy powered computing system design should not ignore the attributes of power supply

• Be very careful when boosting (e.g., over-clocking), maintaining (e.g., use batteries), or shedding (e.g., shutdown servers) the loads in data centers powered by hybrid renewable energy systems.

• A cross-layer and cross-component data center power management scheme could provide us much better design tradeoffs.

29 / 30

Thanks For Your Attention

30 / 30

Green Computing

•30