Hacking the Conventional Data Center Infrastructure
Presented by: Kevin Brown
Confidential Property of Schneider Electric
Commonly Stated Benefits of Open Compute
• Simplification of servers
• Consolidation of server power supplies
• Eliminate centralized UPS and move to the rack with Li-ion
• Less downtime
• Less cost
Page 2 Confidential Property of Schneider Electric |
Critical questions arise about what is upstream of the rack…
• If I adopt Open Compute, what does my power architecture look like?
• Can I get 2N (i.e. tier 3) redundancy levels like I have today?
• How do I support traditional and Open Compute IT loads in the same data center?
Page 3 Confidential Property of Schneider Electric |
? Answering these questions will broaden adoption…
Important to understand the cost trade-offs
• Analyzed capital cost (material cost only)
• Compared traditional and Open Compute specific architectures
• From MV switchgear down to (and including) IT power supply
• Key assumptions:
• OCP PSU/BBU/shelf costs based on design by Schneider Electric
• Traditional server PSU costs & sizing based on various IT vendors/suppliers
• Models based on 9.6MW data center, 10 kW/rack
• Costs normalized to $/Watt
Based on the content of white paper 228, Analysis of Data Center Architectures Supporting Open Compute Project (OCP)
Page 4 Confidential Property of Schneider Electric |
Traditional vs. OCP-based designs
$2.77/watt
$1.53/watt
$2.08/watt
Traditional 2N
Open Compute Specific 1N
Open Compute Specific 2N
We think most will want to maintain 2N (Tier 3) redundancy
“Tiers” in a nutshell
Tier 1: Single power path to IT load; single points of failure Tier 3: Dual (redundant) power paths to IT load, concurrently maintainable
Page 6 Confidential Property of Schneider Electric |
Traditional 2N power architecture today… Typical 2N design for traditional IT loads
• 2N power paths from utility to load, Tier 3(ish)
• Adds complexities (“Belts & suspenders”):
– Load bank
– Ties
– Additional UPS output breakers
• Concurrently maintainable
Page 7 Confidential Property of Schneider Electric |
Open Compute Specific 1N architecture An example of a cost-reduced architecture to support OCP loads
• Aligns with the simplicity and cost-reduction mindset of OCP
• Open Compute servers with one PSU path
• Single path to the IT load
– with rack-based battery backup
– no centralized UPS
• Minimal breaker count
Traditional 2N vs. Open Compute Specific 1N
Page 8 Confidential Property of Schneider Electric |
0.87
0.22 0.15 0.34 1.53
0.03
2.77
Traditional 2N
Redundancy Rack system Open Compute
Specific 1N
Power supplies
Batteries UPS
45% capex savings… but where do these savings come from? $
/ wat
t
Traditional 2N vs. Open Compute Specific 1N
Page 9 Confidential Property of Schneider Electric |
0.87
0.22 0.15 0.34 1.53
0.03
2.77
Traditional 2N
Open Compute
Specific 1N
Power supplies
Batteries UPS Rack system Redundancy
31% savings is from redundancy differences… $
/ wat
t
Traditional 2N vs. Open Compute Specific 1N
Page 10 Confidential Property of Schneider Electric |
0.87
0.22 0.15 0.34 1.53
0.03
2.77
Open Compute
Specific 1N
Power supplies
Batteries UPS Rack system
Redundancy Traditional 2N
1% cost adder for rack system $
/ wat
t
0.87
0.22 0.15 0.34 1.53
0.03
2.77
Redundancy Traditional 2N
Batteries UPS Rack system
Open Compute
Specific 1N
Power supplies
Traditional 2N vs. Open Compute Specific 1N
Page 11 Confidential Property of Schneider Electric |
8% savings from eliminating upstream UPS…
$ / w
att
Traditional 2N vs. Open Compute Specific 1N
Page 12 Confidential Property of Schneider Electric |
0.87
0.22 0.15 0.34 1.53
0.03
2.77
UPS Rack system
Redundancy Traditional 2N
Open Compute
Specific 1N
Power supplies
Batteries
5% cost adder for Li-ion batteries
$ / w
att
Traditional 2N vs. Open Compute Specific 1N
Page 13 Confidential Property of Schneider Electric |
0.87
0.22 0.15 0.34 1.53
0.03
2.77
Open Compute
Specific 1N
Power supplies
Batteries UPS Rack system
Redundancy Traditional 2N
12% savings from power supplies
$ / w
att
• Redundancy • Concurrent maintainability
2N (or tier 3) is still important to many data centers
Page 14 Confidential Property of Schneider Electric |
Utility A
OC
32A
400A
400A
3200A
RPP
1250 A
1250 A
Utility B
32APod level
Rack level
400A
400A
3200A
RPP
1250 A
1250 A
PDU
ServerPSUBBU
ServerPSUBBU
PDU
PSU
PSU
1600A
Cooling
1600A
Cooling
MV level
LV level
G G G G G
630 A
G G GG
Open Compute Specific 2N architecture 2N simple design to support OCP IT loads
Page 15 Confidential Property of Schneider Electric |
reduces complexity by eliminating unnecessary cross-ties, additional breakers
saves cost with one battery path, one raw utility path
trusts the redundant server power supplies
Compared to traditional 2N, this architecture….
Cost difference of Traditional 2N vs. Open Compute Specific 2N
Page 16 Confidential Property of Schneider Electric |
2.080.22
0.01
2.77
0.160.22
0.40
Open Compute
Specific 2N
Power supplies
Batteries UPS Rack system
Architecture simplification
Traditional 2N
$ / w
att
25% capex savings…
Cost difference of Traditional 2N vs. Open Compute Specific 2N
Page 17 Confidential Property of Schneider Electric |
2.080.22
0.01
2.77
0.160.22
0.40
Power supplies
Open Compute
Specific 2N
Batteries UPS Rack system
Architecture simplification
Traditional 2N
$ / w
att A simplified 2N design costs $2.37/watt… (that accounts for more than half of the 25% savings)
Cost difference of Traditional 2N vs. Open Compute Specific 2N
Page 18 Confidential Property of Schneider Electric |
2.080.22
0.01
2.77
0.150.22
0.40
Open Compute
Specific 2N
Power supplies
Batteries UPS Rack system
Architecture simplification
Traditional 2N
Rack, UPS, battery, and PSU differences represent the remaining…
$ / w
att
What if I have a mix of Open Compute and traditional IT loads?
Page 19 Confidential Property of Schneider Electric |
$2.77
$1.53 $2.08 $2.15
0 to 0.23
Traditional 2N
Open Compute
Specific 1N
Open Compute
Specific 2N
Simplified 2N
Premium for traditional power supplies (PSUs) depends on mix of loads
A design that accommodates both types of loads
Assumes $0.46 traditional PSU cost per watt of IT load $
/ wat
t
Page 21 Confidential Property of Schneider Electric |
• Flexible architecture that allows for mix of traditional IT loads and OCP loads
• UPS is upstream to support both traditional and OCP loads
• To minimize cost, one power path with UPS, one without
• OCP loads have dual PSUs without BBUs
Example of a Simplified 2N design that accommodates both types of loads
$2.08 $2.08 $2.08
$0.07 $0.07 $0.07
$0.12 $0.23
100% OCP loads
50% OCP loads
0% OCP loads
9% premium
Premium for traditional PSUs*
Impact of load mix on cost premium
Premium for simplified architecture that accommodates both load types
Cost of Open Compute Specific 2N design
* Assumes $0.46 traditional PSU cost per watt of IT load
3% premium
15% premium
$ / w
att
PSU cost is a highly sensitive variable...
Page 23 Confidential Property of Schneider Electric |
$0.077 per PSU
watt
2 PSUs per
server X
3 PSU
oversizing X =
$0.46 per IT load
watt
Our baseline assumption for traditional PSUs:
Sensitivity analysis:
We varied the PSU oversizing down to 2 and up to 4.5, or $0.32 to $0.70 / IT load watt
$2.08 $2.08 $2.08
$0.07 $0.07 $0.07
$0.04 to
$0.25
$0.09 to
$0.47
100% OCP loads
50% OCP loads
0% OCP loads
Premium varies significantly as traditional PSU cost changes
Sensitivity analysis
3% premium
8% to 26% premium
5% to 15% premium
We assume a more right-sized PSU for OCP • 12 kW of PSU watts x 2 PSUs = 24 kW of PSU for
10kW of IT load
• PSU oversizing = 1.2
$ / w
att
Freely available resources to help with planning decisions
Page 25 Confidential Property of Schneider Electric |
Reference Designs • Designs to support OCP • One-line diagrams, bill of materials, layout drawings • www.schneider-electric.com/datacenterdesigns
TradeOff Tools • OCP vs. traditional cost comparison tool • Li-ion vs. VRLA TCO tool • www.tools.apc.com
White Papers • WP228, Analysis of Data Center Architectures
Supporting OCP Designs • WP229, Battery Technology for Data Centers:
VRLA vs. Li-Ion • www.whitepapers.apc.com
Key take-aways…
1. To increase adoption, critical questions about power infrastructure must be addressed
2. Today’s traditional 2N architectures have opportunity for simplification and cost reduction
3. Simplified 2N represents a small premium (3%) over OCP-specific 2N, and gives flexibility for mixed-loads
Page 26 Confidential Property of Schneider Electric |
Page 27 Confidential Property of Schneider Electric |
Lithium-ion Batteries
Benefits of lithium-ion batteries over VRLA batteries for UPS applications
• Higher energy density (70-260 kWh/kg vs. 15-50 kWh/kg)
• Double the life span (10-15 years vs. 4-6 years)
• 1/3rd smaller footprint
• Weighs 2/3rds less
• Faster re-charge time (1/2–1 hour vs. 6-12 hours)
• Improved cycle life (# of charge/discharge cycles) - >1000 cycles vs. <400 cycles
• Specified to operate at higher temperatures (40C) without impact on cycle life
Page 28 Confidential Property of Schneider Electric |
Li-ion batteries and Open Compute
• Li-ion technology has enabled the rack-based approach
• BUT… important to separate the discussion of battery type from where the battery goes…
Page 29 Confidential Property of Schneider Electric |
Where energy storage resides determines benefits:
• Centralized storage (upstream of rack)
– Flexibility to support mix of OCP and traditional IT
– Isolation of battery problems from IT equipment
– Frees up U space in the IT rack
– Minimize stranded capacity of batteries
– Less batteries to manage
– Filtering and transient protection from upstream UPS
• Decentralized storage (at the rack)
– Fault isolation
– Incremental deployment
– Maintainability
Questions?
Page 30 Confidential Property of Schneider Electric |