NICS RP UpdateTeraGrid Round Table

March 10, 2011

Ryan BrabyNICS HPC Operations Group Lead

Management changes· Patricia Kovatch is the Interim NICS Project

Director.· Ryan Braby is the new HPC Operations and

Technology Integration Group Lead.– Started Jan 1st.– Over 12 years experience in HPC Systems

Administration and Integration.– Experienced with large IBM Power based systems,

BlueGene systems, Linux clusters, and Lustre.

Kraken XT5 Specifications

Compute processor type AMD 2.6 GHz Istanbul-6

Compute cores 112,896

Compute sockets 18,816

Compute nodes 9,408

Memory per node 16 GB (1.33 GB/core)

Total memory 147 TB

Peak system performance 1.17 PF

Interconnect topology 25 x 16 x 24 Torus/Seastar2+

Parallel file system space 3.3 PB (raw) 2.4 PB (usable)

Parallel file system peak performance 30 GB/s

Athena XT4 Specifications

Compute processor type AMD 2.3 GHz Barcelona-4Compute cores 18,048Compute sockets 4,512 quad-coreCompute nodes 4,512Memory per node 4 GB (1 GB/core) Total memory 17.6 TBPeak system performance 0.166 PFInterconnect topology 12 x 16 x 24 Torus/SeastarParallel file system space 100 TB (raw) 85 TB (usable)Parallel file system peak performance 10 GB/s

Nautilus SGI UltraViolet Specs

Compute processor type Intel ~2.0 GHz NehalemCompute cores 1024Compute sockets (nodes) 128 oct-coreMemory per core 4 GBTotal memory 4 TB (NUMA)Accelerators 16 NVIDIA Fermi GPUs (8 active)Peak system performance 8.2 TFInterconnect topology NUMAlink5Parallel file system space 1 PB (960 TB useable, GPFS)Parallel file system peak performance 24 GB/s

Kraken Job Mix - Annual

Kraken Job Mix – Jan 2011

14

816

2432

4048

5660

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

6312

725

551

1

1023

2047

4095

8191

8256

63

127

255

511

1023

2047

4095

8191

8256Number of Nodes

WallclockHours

Number of NodesCORE-Hours

HPC Ops Report Jan 2011

Kraken and Athena Utilization

%

Month

HPC Ops Report Jan 2011

Feb-10 Mar-10 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-110

10

20

30

40

50

60

70

80

90

100

87 84 82

92

90 87

96

93

96 95 9796

93 94

85

90

9398

95

95

9295 94 93

Kraken XT5 UtilizationAthena Utilization

9

Cycles Provided to TeraGrid

Upcoming Events / Work at NICS· Kraken upgrade to CLE 2.2 Update 3

– March 23rd, 8am to 5pm.– No changes required by users.– Should improve system stability.

· Annual power outage / electrical maintenance– Target is April 2nd, currently planned for 16 hours.