Post on 16-Dec-2015
transcript
The CDCE Project @ BNLThe CDCE Project @ BNL
HEPIX – LBLHEPIX – LBL
October 28, 2009October 28, 2009
Tony Chan - BNLTony Chan - BNL
BackgroundBackground
Rapid growth in the last few years caused space, power and Rapid growth in the last few years caused space, power and cooling problemscooling problems
Increasing capacity for RHIC/ATLAS and other activities Increasing capacity for RHIC/ATLAS and other activities cannot be accommodated with current facility cannot be accommodated with current facility infrastructureinfrastructure
Search for additional data center space began in 2007Search for additional data center space began in 2007
Update of talk originally given at HEPIX in St. Louis (Nov. Update of talk originally given at HEPIX in St. Louis (Nov. 2007)2007)
Vital StatisticsVital Statistics
Currently housing 165 racks of equipment (disk storage, Currently housing 165 racks of equipment (disk storage, cpu, network, etc) + 9 robotic siloscpu, network, etc) + 9 robotic silos
Approximately 35 PB of tape storage, 9 PB of disk storage Approximately 35 PB of tape storage, 9 PB of disk storage capacity and 10,200 computing corescapacity and 10,200 computing cores
Average power usage ~ 650 kW (~60% of maximum UPS Average power usage ~ 650 kW (~60% of maximum UPS capacity) with peak load ~ 790 kWcapacity) with peak load ~ 790 kW
Cooling capacity for a maximum of ~1000 kW Cooling capacity for a maximum of ~1000 kW
The Growth of ComputingThe Growth of Computing
0
10000
20000
30000
40000
50000
60000
70000
KSpecInt2000
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
(es
t.)
2011
(es
t.)
2012
(es
t.)
Total Distributed Storage Capacity Total Distributed Storage Capacity
0
5000
10000
15000
20000
25000
30000
35000
Sto
rag
e C
apac
ity
(TB
)
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
(es
t.)
2011
(es
t.)
2012
(es
t.)
Evolution of Space Usage Evolution of Space Usage
0
2000
4000
6000
8000
10000
12000
14000
Us
ag
e o
f D
ata
Ce
nte
r (s
q.
ft.)
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010(est.)
2011(est.)
2012(est.)
Capacity of old data center
Intel dual and quad-core deployed
Evolution of Power Usage Evolution of Power Usage
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Po
we
r U
sa
ge
(k
W)
2003 2004 2005 2006 2007 2008 2009 2010(est.)
2011(est.)
2012(est.)
Existing UPS Capacity
The Search For Solutions (1)The Search For Solutions (1)
Engaged Lab management – Spring 2007Engaged Lab management – Spring 2007
Discussion on possible solutions – Summer 2007Discussion on possible solutions – Summer 2007• CostCost• TimeTime• LocationLocation
Recommendation to Lab management – Fall 2007/Winter Recommendation to Lab management – Fall 2007/Winter 20082008• Two-phase solution to meet cost and time constraintsTwo-phase solution to meet cost and time constraints• Identify and renovate existing space to meet short-term requirementsIdentify and renovate existing space to meet short-term requirements• New building to meet long-term requirementsNew building to meet long-term requirements
Funding for two construction projects approved – Spring Funding for two construction projects approved – Spring 20082008
The Search for Solutions (2) The Search for Solutions (2)
Renovate existing floor space (US $0.6 million)Renovate existing floor space (US $0.6 million)• Tender award – April 2008Tender award – April 2008• Renovations begin – June 2008Renovations begin – June 2008• Renovations end -- October 2008Renovations end -- October 2008• Occupancy – November 2008Occupancy – November 2008
New building (US $5 million)New building (US $5 million)• Finalize design – May 2008Finalize design – May 2008• Tender award – June 2008Tender award – June 2008• Construction starts – August 2008Construction starts – August 2008• Construction ends – August 2009Construction ends – August 2009• Occupancy – October 2009Occupancy – October 2009
From first proposal to occupancy took 2½ yearsFrom first proposal to occupancy took 2½ years
Facility Development TimelineFacility Development Timeline
Recent Past Recent Past (2006-2007)(2006-2007)• More efficient use of facility resourcesMore efficient use of facility resources• Supplemental cooling system in existing facilitySupplemental cooling system in existing facility
Near-Term Near-Term (2008 to present)(2008 to present)• Renovation of (2000 ftRenovation of (2000 ft22) 185 m) 185 m22 of unused floor with 300 kW of of unused floor with 300 kW of
powerpower• New building with (6600 ftNew building with (6600 ft22) 622 m) 622 m22 and 1.0 MW of power and 1.0 MW of power• Mission-specific facility (redundant cooling, deep raised floors, Mission-specific facility (redundant cooling, deep raised floors,
etc)etc)• Room for ~ 150 racks and 7 robotic silosRoom for ~ 150 racks and 7 robotic silos
Long-Term Long-Term (2017 and beyond)(2017 and beyond)• New BNL data center with 25000 ftNew BNL data center with 25000 ft22 (2300 m (2300 m22) after 2018) after 2018
Where We Are Today (1)Where We Are Today (1)
Data Center Expansion (part 1) is similar to Data Center Expansion (part 1) is similar to existing facilityexisting facility• 12-in (30.48 cm) raised floor12-in (30.48 cm) raised floor• Redundant cooling capacityRedundant cooling capacity• No support for racks > 10 kWNo support for racks > 10 kW• No support for supplemental coolingNo support for supplemental cooling• Cable trays for power and networkCable trays for power and network
Data Center Expansion (part 2) was designed for Data Center Expansion (part 2) was designed for high density equipmenthigh density equipment• 30-in (76.2 cm) raised floor30-in (76.2 cm) raised floor• Support for racks > 10 kW (blades, half-depths, etc)Support for racks > 10 kW (blades, half-depths, etc)• Redundant cooling capacityRedundant cooling capacity• Support for racks > 2,500 lbs (1,135 kg)Support for racks > 2,500 lbs (1,135 kg)• 13-ft ceiling (4 m) for high-profile racks13-ft ceiling (4 m) for high-profile racks• Cable trays for power and networkCable trays for power and network• Support for supplemental coolingSupport for supplemental cooling• Environmentally-friendly buildingEnvironmentally-friendly building
Where We Are Today (2)Where We Are Today (2)
Facility expanded from 5000 ftFacility expanded from 5000 ft22 (465 m (465 m22) to 13,600 ft) to 13,600 ft22 (1260 (1260 mm22) of floor space) of floor space
Equipment capacityEquipment capacity• from ~150 to ~300 racksfrom ~150 to ~300 racks• from 6 to 13 robotic silosfrom 6 to 13 robotic silos
Infrastructure supportInfrastructure support• from ~1 to ~2.0 MW of UPS-backed power (up to 4 MW capacity)from ~1 to ~2.0 MW of UPS-backed power (up to 4 MW capacity)• Cooling capacity grew from ~1 to ~ 2 MW (up to 4 MW capacity)Cooling capacity grew from ~1 to ~ 2 MW (up to 4 MW capacity)
3 robotic silos and 6 racks of worker nodes first occupants 3 robotic silos and 6 racks of worker nodes first occupants of CDCE (October 2009)of CDCE (October 2009)
Is this sufficient until 2018?Is this sufficient until 2018?
Unresolved IssuesUnresolved Issues
Insufficient funds to:Insufficient funds to:• add 2add 2ndnd flywheel UPS for CDCE flywheel UPS for CDCE• diesel generator to support additionaldiesel generator to support additional flywheel UPS unitsflywheel UPS units• install additional 1 MW of cooling capacity (equipment already install additional 1 MW of cooling capacity (equipment already
purchased) purchased)
Estimated cost is additional US $2-3 millionEstimated cost is additional US $2-3 million• Estimate CDCE will exceed 2 MW of UPS power and cooling by Estimate CDCE will exceed 2 MW of UPS power and cooling by
20122012
Lead time to approve funds and pre-installation is Lead time to approve funds and pre-installation is 12 months 12 months decision by 2011 decision by 2011
Reason for Optimism?Reason for Optimism?
0
20
40
60
80
100
120
140
160
180
Total
Rack
Coun
t
2005 2006 2007 2008 2009 2010 (est.)
Multi-core cpu’s and Multi-core cpu’s and increasing storage increasing storage density have helped density have helped restrain a feared restrain a feared unsustainable growth in unsustainable growth in power and space needspower and space needs
Rack counts have not Rack counts have not increased at the same increased at the same rate as computing and rate as computing and storage deploymentsstorage deployments
Somewhat hopeful that Somewhat hopeful that continued technological continued technological gains will further restrain gains will further restrain data center growthdata center growth
SummarySummary Facility footprint nearly tripled since 2007Facility footprint nearly tripled since 2007
Applied lessons learned in design of data center expansion Applied lessons learned in design of data center expansion (part 2)(part 2)
Must increase cooling efficiency with new technologiesMust increase cooling efficiency with new technologies• Rack-top cooling units Rack-top cooling units • Rear-door heat exchangerRear-door heat exchanger• Hot aisle containmentHot aisle containment
Significant increases in power efficiency and technology Significant increases in power efficiency and technology (power supply, multi-core cpu, etc) is a positive (power supply, multi-core cpu, etc) is a positive development, but some unresolved issues remaindevelopment, but some unresolved issues remain