Grid Computingat the Large Hadron
Collider:Massive Computing at the
Limit of Scale, Space, Power and Budget
Dr Helge MeinhardCERN, IT Department
02-Jul-2009
CERN (1)
Conseil européen pour la recherche nucléaire – aka European Laboratory for Particle Physics Facilities for
fundamental research
Between Geneva and the Jura mountains, straddling the Swiss-French border
Founded in 1954
CERN (2)
20 member states ~3400 staff members,
fellows, students, apprentices
9000 users registered (~6500 on site) from more than 550
institutes in more than 80 countries
~ 910 MCHF (~550 MEUR) annual budget
http://cern.ch/
Physics at the LHC (1)
Matter particles: fundamental building
blocks
Force particles:bind matter particles
Physics at the LHC (2)
Four known forces: strong force,weak force, electromagnetism, gravitation
Standard model unifies three of them Verified to
0.1 percent level Too many free
parameters E.g. particle masses
Higgs particle Higgs condensate
fills vacuum Acts like ‘molasse’,
slows other particles down, gives them mass
Physics at the LHC (3)
Open questions in particle physics: Why are the parameters of the size as we
observe them? What gives the particles their masses? How can gravity be integrated into a unified
theory? Why is there only matter and no anti-matter in
the universe? Are there more space-time dimensions than
the 4 we know of? What is dark energy and dark matter which
makes up 98% of the universe? Finding the Higgs and possible new
physics with LHC will give the answers!
The Large Hadron Collider (1)
Accelerator for protons against protons – 14 TeV collision energy By far the world’s
most powerful accelerator
Tunnel of 27 km circumference, 4 m diameter, 50…150 m below ground
Detectors at four collision points
The Large Hadron Collider (2)
Approved 1994, first circulating beams on10 September 2008
Protons are bent by superconducting magnets (8 Tesla, operating at 2K = –271°C) all around the tunnel
Each beam: 3000 bunches of 100 billion protons each
Up to 40 million bunch collisions per second at the centre of each of the four detectors
The Large Hadron Collider (3) Incident on 19 September 2008
During attempt to ramp beam energy up to 7 TeV, a leak occurred in the cold mass causing significant loss of helium
Repair work is ongoing Instrumentation for detecting this kind of
problem being added Schedule: beam around mid October
2009, collisions around mid November 2009, running until autumn 2010 Collision energy: 5 + 5 TeV Short technical stop at Christmas 2009
LHC Detectors (1)ATLAS
CMS
LHCb
LHC Detectors (2)
2’200 physicists (including 450 students) from 167 institutes of 37
countries
LHC Data(1)
The accelerator generates 40 million bunch collisions (“events”) every second at the centre of each of the four experiments’ detectors
LHC Data (2)
Reduced by online computers that filter out a few hundred “good” events per second …
… which are recorded on disk and magnetic tape at 100…1’000 Megabytes/sec
15 Petabytes per year for four experiments
15’000 Terabytes = 3 million DVDs
1 event = few Megabytes
LHC Data (3)
CERN18%
All Tier-1s39%
All Tier-2s43%
CERN12%
All Tier-1s55%
All Tier-2s33%
CERN34%
All Tier-1s66%
CPU Disk Tape
30’000 CPU servers, 110’000 disks: Far too
much for CERN!
Summary of Computing Resource RequirementsAll experiments – 2008
From LCG TDR – June 2005
Total
CPU (MSPECint2000s) 142
Disk (Petabytes) 57
Tape (Petabytes) 53
CERN All Tier-1s All Tier-2s
25 56 61
7 31 19
18 35
Worldwide LHC Computing Grid (1) Tier 0: CERN
Data acquisition and initial processing
Data distribution Long-term curation
Tier 1: 11 major centres Managed mass
storage Data-heavy analysis Dedicated 10 Gbps
lines to CERN Tier 2: More than
200 centres in more than 30 countries Simulation End-user analysis
Tier 3: physicists’ desktops
Tier3physics
department
Desktop
Germany
USAUK
France
Italy
Taiwan
NordicCountries
Nether-lands
CERN Tier 0
Tier2
Lab a
Uni a
Lab c
Uni n
Lab m
Lab b
Uni bUni y
Uni x
grid for a physicsstudy group
SpainTier 1
grid for a regional group
Worldwide LHC Computing Grid (2) Grid middleware for “seemless”
integration of services Aim: Looks like single huge compute
facility Projects: EDG/EGEE, OSG Big step from proof of concept to stable,
large-scale production Centres are autonomous, but lots of
commonalities Commodity hardware (e.g. x86
processors) Linux (RedHat Enterprise Linux variant)
CERNComputer CentreFunctions:
WLCG: Tier 0, some T1/T2
Support for smaller experiments at CERN
Infrastructure for the laboratory
…
Requirements and Boundaries (1) High Energy Physics applications require
mostly integer processor performance Large amount of processing power and
storage needed for aggregate performance No need for parallelism / low-latency high-
speed interconnects Can use large numbers of components with
performance below optimum level (“coarse-grain parallelism”)
Infrastructure (building, electricity,cooling) is a concern Refurbished two machine rooms
(1500 + 1200 m2) for total air cooledpower consumption of 2.5 MWatts
Will run out of power in about 2011…
Requirements and Boundaries (2)
Major boundary condition: cost Getting maximum
resources with fixed budget…
… then dealing with cuts to “fixed” budget
Only choice: commodity equipment as far as possible, minimisingTCO / performance This is not always the
solution with the cheapest investment cost!
Purchased in 2004, now retired
The Bulk Resources – Event Data
Tapes/ server
s
Disk servers
CPU servers
Router
R
R
R
REthernetbackbone
(multiple 10GigE)
10GigE
Permanent storage on tape
Disk as temporary buffer
Data paths:tape diskdisk cpu
(simplified network topology)
CERN CC currently (July 2009)
5’700 systems, 34’600 processing cores CPU servers, disk servers, infrastructure
servers 13’900 TB usable on 41’500 disk
drives 34’000 TB on 45’000 tape cartridges
(56’000 slots), 160 tape drives Tenders in progress or planned
(estimates) 3’000 systems, 20’000 processing cores 19’000 TB usable on 21’000 disk drives
CPU Servers (1)
Simple, stripped down, “HPC like” boxes No fast low-latency interconnects
EM64T or AMD64 processors (usually 2),2 GB/core, 1 disk/processor
Open to multiple systems per enclosure Adjudication based on total performance
(SPECint2000, moving to SPECcpu2006 – all_cpp subset)
Power consumption taken into account
CPU Servers (2)
The Power Challenge (1)
Infrastructure limitations E.g. CERN: 2.5 MW for IT equipment
Clearly insufficient – need to fit maximum capacity into given power envelope
Additional creative measures required (water-cooled racks in air-cooled room)
Electricity costs money Electricity costs likely to raise (steeply)
over the next few years Saving 10 W is saving 88 kWh per year
The Power Challenge (2)
IT responsible of significant fraction of world energy consumption Server farms: 180…280 billion kWh per
year (20…31 million kW) CERN’s data centre is 0.1 per mille of this…
1…2% of the world’s energy consumption, annual growth rate: 16…23%
Responsibility towards mankind demands using the energy as efficiently as possible
Saving a few percent of energy consumption makes a big difference
Server Energy Consumption
Power supply Fans Processors Chipset Memory modules Disk drives VRMs (Voltage Regulator Modules) RAID controllers … What should we start looking at?
CERN’s Approach
Measure apparent (VA) power consumption in primary AC circuit CPU servers: 80% full load, 20% idle Infrastructure servers: 50% full load, 50%
idle Add element reflecting power
consumption to purchase price Currently about 6.50 EUR per VA
Adjudicate on the sum of purchase price and power adjudication element
Power Efficiency: Lessons Learned (1) Power efficiency increased by factor 12
in just a little over 4 years Power efficiency =
performance / power consumption Quantum steps:
Microarchitecture: Netburst to Core to Nehalem Multi-core: 1 to 2 to 4 per CPU For Core: 5000P (Blackford) to 5100 (San
Clemente) chipset Much improved PSU efficiencies
CERN retires servers aggressively after end of warranty period
Power Efficiency: Lessons Learned (2)
Need to benchmark concrete servers; generic statements on platform are void Beginning 2007: different servers with same
CPU, same chipset, same memory config resulted in proposals with 50% spread of power efficiency
Fostering energy-efficient solutions makes a difference Summer 2008: different techniques in
response to same call for tender differed by 60% in power efficiency
Power Efficiency: Lessons Learned (3)
Solutions with power supplies feeding more than one system usually more power-efficient There are more options than just
blades… Redundant power supplies are
inefficient Summer 2008: 1U server running on one
PSU module used 8.5% less power than running on two modules
Difference even larger in idle mode
Future (1)
Is IT growth sustainable? Demands continue to rise exponentially Even if Moore’s law continues to apply,
data centres will need to grow in number and size
IT already consuming 2% of world’s energy – where do we go?
How to handle growing demands within a given data centre? Demands evolve very rapidly, technologies
less so, infrastructure even at a slower pace – how to best match these three?
Future (2)
IT: Ecosystem of Hardware OS software and tools Applications
Evolving at different paces: hardware fastest, applications slowest How to make sure at any given time that
they match reasonably well?
Future (3)
Example: single-core to multi-core to many-core Most HEP applications currently single-
threaded Consider server with two quad-core
CPUs as eight independent execution units Model does not scale much further
Need to adapt applications to many-core machines Large, long effort
Summary The Large Hadron Collider (LHC) and its experiments
is a very data (and compute) intensive project LHC has triggered or pushed new technologies
E.g. Grid middleware, WANs High-end or bleeding edge technology not necessary
everywhere That’s why we can benefit from the cost advantages of
commodity hardware Scaling computing to the requirements of LHC is hard
work IT power consumption/efficiency is a primordial
concern We had the first circulating beams on 10-Sep-2008,
and have the capacity in place for the initial needs We are on track for further ramp-ups of the computing
capacity for future requirements
Summary of Computing Resource RequirementsAll experiments - 2008From LCG TDR - June 2005
CERN All Tier-1s All Tier-2s TotalCPU (MSPECint2000s) 25 56 61 142Disk (PetaBytes) 7 31 19 57Tape (PetaBytes) 18 35 53
Thank you