Cutting the Electric Bill for Internet-Scale SystemsAndreas AndreouCambridge University, R02aa773@cam.ac.uk

What’s this all about?

• Energy expenses are an increasingly important fraction of data center operating costs

• Electricity prices show both temporal and geographical variation

• Exploit variations in electricity prices for economic gain

Key observations• Electricity prices vary• Prices vary on an hourly basis• Often not well correlated at different locations• Substantial variations

• Large distributed systems already incorporate request routing and replication• Dynamic request routing to map clients to servers• Mechanisms to replicate data necessary to process requests at

multiples sites

Problem Specification• Large system composed of server clusters spread out

geographically• Map client requests to clusters such that the total electricity

cost is minimized• Assumptions• System fully replicated• Optimize for cost every hour• No knowledge of the future• Rate of change slow enough to be compatible with existing

routing mechanisms• Fast enough to respond to electricity market fluctuations• Incorporate bandwidth and performance goals as constraints

Terminology• Energy Elasticity• Degree to which energy consumed by a cluster depends on the

load placed on it• Ideally: no load, no power• Worst case: no difference between peak and idle power• State-of-the-art: idle power around 60% of peak

• Differential Duration• Number of hours one location is favored over another by more

than $5/MWh• PUE• Power usage effectiveness (measure of data center energy

efficiency)

Background

Wholesale Electricity Markets (1)• Generation• Government and independent power producers• Coal (~50%), natural gas (~20%), nuclear power (~20%),

hydroelectric generation (~6%)• Different regions, different power generation profiles

• Transmission• Producers and consumers are connected to an electric grid• 8 reliability regions

Wholesale Electricity Markets (2)• Market Structure• Each region managed by Regional Transmission Organization

(RTO)• RTO administer wholesale electricity markets• Auctioning mechanism:

• Producers present supply offers• Consumers present demand bids• Coordinating body determines flow and sets prices

• Market Types• Day-ahead markets• Real-time markets

Wholesale Electricity Markets (3)• Market Structure• Assumptions

• Real-time prices are known and vary hourly• Electric bill is proportional to consumption and indexed to wholesale

prices• Request routing behavior induced by our method doesn’t significantly

alter prices and market behavior

Daily Variation

Different Market Types• Hourly real-time (RT) market is more volatile than day-ahead

market

Hour-to-Hour Volatility

Geographic Correlation

Price Differentials

Differential Distributions

Time-of-Day

Differential Duration

Akamai: Traffic and Bandwidth• Over 2000 content provider customers in the US• 9-region traffic with electricity price data• Data covering 24 days worth of traffic• Traffic data of 5-minute intervals from public clusters

• Bandwidth costs are significant• Aggressively optimized to reduce bandwidth costs• 95/5 billing model

• Client-Server Distances• Use geographic distance as a coarse proxy for network

performance

Cluster Energy Consumption (1)• Roughly linear to its utilization

• Pidle : average idle power draw of single server

• Ppeak : average peak power draw of single server

• r: empirical derived constant• ut : average CPU utilization at time t

• • what is important in determining savings

Routing Energy• Increased path lengths will not alter energy consumption

significantly• Average energy for a packet to pas through is on the order of

2mJ• Incremental energy dissipated by each packet passing

through a core router would be as low as 50μJ per medium size packet

• New routes may overload existing routers• Additional bandwidth could lead to upgrade• Can ignore by incorporating 95/5 bandwidth constraints

Simulation Strategy• Real-time market prices for 29 different locations• Traffic data for Akamai public clusters in 9 of those• Data set spanning Jan 2006 through Mar 2009• Workload data set contains 5-minute samples in 25 cities• Period of 24 days and some hours• Discarded 7 and grouped remaining 18 cities to 9 clusters

• Akamai’s geographic server distribution• Two routing schemes• Akamai’s original allocation• Distance constrained electricity price optimizer

• Energy model as shown before

24 Days of Traffic (1)• Energy Elasticity

• Bandwidth Costs

24 Days of Traffic (2)• Distance and savings

39 Months of Prices• Derived from 24-day Akamai workload (US traffic only)

• Dynamic beats static

Results• Existing systems can reduce energy costs be at least 2%

without any increase in bandwidth costs or significant reduction in client performance• Google-like energy elasticity• Akamai-like server distribution• 95/5 bandwidth constraints

• Savings increase with energy elasticity• Fully elastic system with relaxed bandwidth constraints can

reduce energy cost be 30% (13% with bandwidth constraints)

• Allowing increase of client-server distances leads to increased savings

Considerations (1)• Not reacting immediately to price changes noticebly reduces

overall savings

Considerations (2)• Server operators should be able to negotiate contractual

arrangements

• Distributed systems with energy elastic clusters can be more flexible than traditional consumers

• Triggered demand response programs

Future Work• Implementing Joint Optimization

• RTO Interaction

• Weather Differentials

• Environmental Cost