NICS RP Update

TeraGrid Round Table

March 10, 2011

Ryan BrabyNICS HPC Operations Group Lead

Management changes

· Patricia Kovatch is the Interim NICS Project Director.

· Ryan Braby is the new HPC Operations and Technology Integration Group Lead.– Started Jan 1st.– Over 12 years experience in HPC Systems

Administration and Integration.– Experienced with large IBM Power based systems,

BlueGene systems, Linux clusters, and Lustre.

Kraken XT5 Specifications

Compute processor type AMD 2.6 GHz Istanbul-6

Compute cores 112,896

Compute sockets 18,816

Compute nodes 9,408

Memory per node 16 GB (1.33 GB/core)

Total memory 147 TB

Peak system performance 1.17 PF

Interconnect topology 25 x 16 x 24 Torus/Seastar2+

Parallel file system space 3.3 PB (raw) 2.4 PB (usable)

Parallel file system peak performance 30 GB/s

Athena XT4 Specifications

Compute processor type AMD 2.3 GHz Barcelona-4

Compute cores 18,048

Compute sockets 4,512 quad-core

Compute nodes 4,512

Memory per node 4 GB (1 GB/core)

Total memory 17.6 TB

Peak system performance 0.166 PF

Interconnect topology 12 x 16 x 24 Torus/Seastar

Parallel file system space 100 TB (raw) 85 TB (usable)

Parallel file system peak performance 10 GB/s

Nautilus SGI UltraViolet Specs

Compute processor type Intel ~2.0 GHz Nehalem

Compute cores 1024

Compute sockets (nodes) 128 oct-core

Memory per core 4 GB

Total memory 4 TB (NUMA)

Accelerators 16 NVIDIA Fermi GPUs (8 active)

Peak system performance 8.2 TF

Interconnect topology NUMAlink5

Parallel file system space 1 PB (960 TB useable, GPFS)

Parallel file system peak performance 24 GB/s

Kraken Job Mix - Annual

Kraken Job Mix – Jan 2011

14

816

2432

4048

5660

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

63

127

255

511

1023

2047

4095

8191

8256

63

127

255

511

1023

2047

4095

8191

8256Number of Nodes

WallclockHours

Number of NodesCORE-Hours

HPC Ops Report Jan 2011

Kraken and Athena Utilization

%

Month

HPC Ops Report Jan 2011

Feb-10 Mar-10 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-110

10

20

30

40

50

60

70

80

90

100

8784 82

92

9087

96

93

96 95 9796

9394

85

90

9398

95

95

9295 94

93

Kraken XT5 Utilization

Athena Utilization

9

Cycles Provided to TeraGrid

Upcoming Events / Work at NICS

· Kraken upgrade to CLE 2.2 Update 3– March 23rd, 8am to 5pm.– No changes required by users.– Should improve system stability.

· Annual power outage / electrical maintenance– Target is April 2nd, currently planned for 16 hours.