Evaluation of Computing Technologies

for the ALICE Data Acquisition

Klaus.Schossmaier@cern.chCERN PH-AID

Visitors from UK Department of Trade and Industry (DTI)30 June 2004

Visitors from UK DTI, 30 June 2004

2Evaluation of Computing Technologies for the ALICE DAQ

DAQ Architecture

LDC (Local Data Concentrator)

DSS (DAQ Services Server)

GDC (Global Data Collector)

event-buildingnetwork

DDL HLT HLT HLT (High Level Trigger)

FEE (Front End Electronics)

KVM (Keyboard Video Mouse)

TDS (Transient Data Storage)

storage network

PDS (Permanent Data Storage)

Visitors from UK DTI, 30 June 2004

3Evaluation of Computing Technologies for the ALICE DAQ

DAQ Installations

DAQ System DDL LDC+GDC+DSS

Location

Development + Evaluation

2 10 AID lab CERN

DDL Development 2 2 DDL lab CERN

Detector Groups 1 1 Institutes

Test Beams 2 4 Test Beam areas

ALICE Data Challenges 0 80 IT CERN

Reference Setup 6 7 AID lab CERN

Test + Commissioning 25 15 LHC Point 2

Final System 422 254 LHC Point 2

Visitors from UK DTI, 30 June 2004

4Evaluation of Computing Technologies for the ALICE DAQ

DAQ Examples

Test Beam for TPC Sector Test:May 2004, 4x1 system, 533 GB

Reference System:to be finished end of 2004

L3 rack

LDCs (1U, 2U, 4U)

GDCs (1U)

GE switches

KVM switch

FC switch

Disk arrays

Server

Visitors from UK DTI, 30 June 2004

5Evaluation of Computing Technologies for the ALICE DAQ

Computing Elements

Platforms Single/Dual/Quad CPU Rack-mount and VMEbus form-factor PCI or PCI-X slots for RORCs and FC adaptors Linux operating system (Red Hat) DATE software including custom drivers

Networking Gigabit Ethernet adaptors/switches Fiber Channel adaptors/switches

Data Storage Disk Array (RAID level 0 or 5)

Infrastructure KVM (keyboard, video, mouse) system PDUs (power distribution units)

Visitors from UK DTI, 30 June 2004

6Evaluation of Computing Technologies for the ALICE DAQ

Linux Platforms

VMEbus SBC

Opteron

AthlonMP

Quad Xeon

Itanium

Dual Xeon

Pentium II and III

NetServer

Visitors from UK DTI, 30 June 2004

7Evaluation of Computing Technologies for the ALICE DAQ

32-bit Machines

Supermicro X5DPR-iG2 [Broadberry]

Intel Server Board SE7501HG2 [Elonex]

Most likely used as LDC and GDC

CPU: dual Xeon 2.4 GHz, 512 KBChipset: Intel E7500, 400 MHz FSBMemory: 1 GB, 266 MHz DDR SDRAMDisks: 80 GB IDE Gb Ethernet: 82546EB, RJ45 portsGraphics: ATI Rage XL, 8 MB Slots: 1x PCI-X 64/66, 1x PCI-X 64/133

CPU: dual Xeon 2.66 GHz, 512 KBChipset: Intel E7501, 533 MHz FSBMemory: 2 GB, 266 MHz DDR SDRAMDisks: 120 GB IDE Gb Ethernet: 82546EB, RJ45 portsGraphics: ATI Rage XL, 8 MB Slots: 2x PCI-X 64/100, 1x PCI-X 64/133, 3x PCI 32/33

Linux:Red Hat 7.3

Visitors from UK DTI, 30 June 2004

8Evaluation of Computing Technologies for the ALICE DAQ

64-bit Machines

2-way 1U server [Newisys 2100]

CPU: dual Opteron 244 (1.8 GHz), 1MB L2 cacheChipset: AMD 8131 PCI-X bridge, AMD 8111 SouthbridgeMemory: 8 GB, 266 MHz DDR SDRAM (in banks) Disks: 36.7 GB ULTRA320 SCSI disksGb Ethernet: 2x BCM5703X, RJ45 portsGraphics: Trident Blade 3D, 8 MB PCI-X slots: 1x 64/100, 1x 64/133Periphery: PS/2, USB, CD-ROM, floppy, etc.Service Processor: web server for administration

32-bit Linux: Red Hat 7.3 64-bit Linux: UnitedLinux 1.0 CERN Enterprise Linux 3

x86-64 architecture: - seamless 32-bit and 64-bit mode- tools gcc 2.96 (32-bit), gcc 3.2.1 (32- and 64-bit)- DATE runs under Red Hat Linux 7.3- ROOT and mySQL runs under United Linux 1.0

Presentation: CERN Computing Seminar 3rd Sep 2003

Visitors from UK DTI, 30 June 2004

9Evaluation of Computing Technologies for the ALICE DAQ

VMEbus Computer

Concurrent Technologies Plc. (CCT):One-slot VME64 single board computers6U form factor, PC-compatible

CPU: Pentium III @ 850MHzChipset: 440BX/ZX/DXMemory: 256 MB, 100 MHz ECCFast Ethernet: 82559ER, RJ45 portGraphics: CT69030, 4 MBExpansion: PMC slot, no PCI slotStorage module: 10 GB disk, floppy driveLinux: Red Hat 6.2 and 7.2 and 7.3

VMEbus access A16/24/32, D8/16/32, BLT:- Tundra Universe II chip- linuxvme-1.10.01driver- single-cycle rate: 3.5 MB/s read, 14 MB/s write- DMA64 rate: 43 MB/s read, 44 MB/s write

Reports: ALICE-INT-2001-034, ALICE-INT-2000-019

VP CP1/P34-11together withDS MSS/IFP-14

Visitors from UK DTI, 30 June 2004

10Evaluation of Computing Technologies for the ALICE DAQ

Benchmarking CPUs

0

20000

40000

60000

80000

100000

120000

0 500 1000 1500 2000 2500 3000

MHz

uben

ch/#

CP

U

Pentium II Pentium III Opteron Athlon MP Xeon Itanium

64-bit

32-bit

Visitors from UK DTI, 30 June 2004

11Evaluation of Computing Technologies for the ALICE DAQ

Benchmarking Memory

0

200

400

600

800

1000

1200

1400

1600

1800

100 MHz 133 MHz 133 MHz 266 MHz 500 MHz

SDRAM SDRAM DDRSDRAM

DDRSDRAM

RDRAM

MB

/s min

max1x Stream:

Visitors from UK DTI, 30 June 2004

12Evaluation of Computing Technologies for the ALICE DAQ

Network Infrastructure

Network technologies Fast/Gigabit/10Gigabit Ethernet for event-

buildingaggregate bandwidth: 2.5 GB/s

Fiber Channel (FC) for storage networkaggregate bandwidth: 1.25 GB/s

System issues Network controller Linux kernel + driver Network switch

Evaluation Transfer rate (message size, TCP socket size) CPU load (% CPU per MB/s) Long-term stability

Visitors from UK DTI, 30 June 2004

13Evaluation of Computing Technologies for the ALICE DAQ

Benchmarking Ethernet

3C996 (BCM5700 controller)

Pro/1000 MT server (82545EM controller)

Visitors from UK DTI, 30 June 2004

14Evaluation of Computing Technologies for the ALICE DAQ

Readout Performance

0

10000

20000

30000

40000

50000

60000

10 100 1000 10000 100000 1000000 10000000

Bytes

ev

en

ts/s

ec

1xDDL 2xDDL

0

20

40

60

80

100

120

140

10 100 1000 10000 100000 1000000 10000000

Bytes

MB

/se

c

1xDDL 2xDDL

1x LDC:- Intel Server Board SE7501HG2- 2 pRORC + 2 DDL + 2 FEE- 100KB pages, 250MB physmem- Red Hat Linux 7.31x GDC:- Supermicro X5DPE-G2- 64MB IPC for event building- Red Hat Linux 7.3

DDL

2x FEE:- Front-End Emulator- 100 MB/s

GE

Test Setup

Visitors from UK DTI, 30 June 2004

15Evaluation of Computing Technologies for the ALICE DAQ

Transient Data Storage

Connection technologies DAS (Direct Attached Storage): scaling problems NAS (Network Attached Storage): not conclusive SAN (Storage Attached Network): good results with FC

Disk Arrays Product from DotHill: 12x 36.7 GB based on FC disks Product from Infortrend: 12x 300 GB based on IDE disks Going to test products based on SATA technology Dual 2 GBit/s ports, rack-mount Configuration: RAID level, file system (ext2, xfs), etc.

Fiber Channel Fabric Host Bus Adaptor (HBA): products from QLogic Switch: product from Brocade with 15 ports

Visitors from UK DTI, 30 June 2004

16Evaluation of Computing Technologies for the ALICE DAQ

Benchmarking Disk Arrays

Infortrend IFT-6330-12F2D

DotHill SANnetII 200

Visitors from UK DTI, 30 June 2004

17Evaluation of Computing Technologies for the ALICE DAQ

KVM System

Raritan Paragon UMT2161

2nd user (over IP)

1st user Cat5 cables

2 users, 16 ports

daisy chaining …

Keyboard

Video

Mouse

Computer Interface Module (CIM)

Visitors from UK DTI, 30 June 2004

18Evaluation of Computing Technologies for the ALICE DAQ

Résumé

ALICE DAQ is based on standard technology Technology is driven by the market Performance increases by almost constant

price Choosing from different vendors Fast evolution cycles need to be watched

Thorough evaluation of computing elements Doing functional tests and benchmarking Working close together with the manufacturer Ongoing task with documentation

Purchasing Need to follow the CERN regulations Installation happens in phases

Potentials Integration, management, surveys