SVG Upgrade 2012

AGENDA

•Upgraded SVG

• Motivation

• Hardware configuration

• Infiniband Network

• Environment

• Queues configuration

• Benchmarks

•CESGA Supercomputers in 2013

• Distribution of jobs

Hardware

Thin-nodes 8 HP SL230s Gen8 each with

2x Intel Xeon E5-2670, 8 cores each, 2.6GHz

64GB main memory DDR3-1600MHz

2TB SAS hard disk

2x1GbE

2xInfiniband FDR 56Gb

Peak Performance 332GFlops

Fat-nodes 2 HP DL560 Gen8 each with

4x Intel Xeon E5-4620, 8 cores each, 2.2GHz

512GB main memory DDR3-1600MHz

6 hard disks each 1TB

4x1GbE

Infiniband FDR 56Gb

10GbE

Peak Performance 563 GFlops

Total 24 Intel Sandy Bridge processors 192 cores 1,5 TB memory 28 TB disk Peak Performance 3788 GFLops

Total

24 Intel Sandy Bridge CPUs

192 cores

1,5 TB memory

28 TB disk

3788 Gflops peak performance

Motivation

Target

• Competitive solution for:

• Parallel MPI & OpenMP applications

• Memory Intensive

• Alternative for Finis Terrae

• Lower cost of operation and maintenance

• Finis Terrae II prototype

• To define new requirements

Infiniband network

Mellanox SX6036 switch

36 port FDR 56Gb/s

4Tb/s aggregated non-blocking BW

1 microseconds MPI latency

Dual connection:

High availability – same BW

Environment

Integrated in the SVG cluster:

Scientific Linux 6.3 (Red Hat)

Common /opt/cesga

Common /home, stores…

Same gateway: svg.cesga.es

Interactive use: compute –arch sandy

Jobs: qsub –arch sandy

Binary compatible - no need to recompile

Usage

svg.cesga.es

SSH

compute –arch amd qsub –arch amd

compute –arch sandy qsub –arch sandy

compute qsub

Configuration

Full production phase (November 2012)

• Only runs jobs with –sandy option

• General availability of applications specifically compiled

• Maximum wall-clock time 12 hours

• Maximum 2 jobs per node fat nodes

Under consideration near future:

Jobs without –sandy option

Higher wall-clock time

Queues Configuration

Exclusive nodes

To take advantage of Infiniband

To take advantage of Turboboost

Jobs not interferring each other

Maximum performance

Maximum 2 jobs on Fat nodes

32 cores nodes

Exclusive if required by the jobs (cores, memory)

Queues: Limits “module help sge”

Up to 112 cores (MPI)

Up to 32 cores shared memory (OpenMP)

Memory:

up to 64GB per core

up to 512GB for non MPI jobs

up to 1024GB per job

Scratch up to 1,7TB

Execution time: 12 hours

If needed ask for more resources in https://www.altausuarios.cesga.es/

Intel® Xeon® Processor E5-2600 Turbo Boost

1Max Turbo Boost frequency based on number of 100 MHz increments above marked frequency (+1 = 0.100 GHz, +2 = 0.200 GHz, +3 = 0.300 GHz, etc. )

CESGA Supercomputers 2013

Finis Terrae (FT)

Capability computing

Parallel jobs (>4 ... 1024 cores)

Huge memory (>4... 1024GB)

Huge parallel scratch (>50... 10,000GB)

Superordenador Virtual Gallego (SVG)

Throughput and Capacity computing

Sequential & parallel jobs up to 32 cores per node and 112 cores

MPI

Low-medium-large memory (up to 512GB!)

Medium single node scratch (<1000GB)

Customized clusters – Cloud services

Shared storage: /home

/store

Linux O. S.

Grid Engine Batch Scheduler

Other Improvements

VPN for home connection

Storage:

Do not use SFS from SVG

Use “store”

High availability front-ends:

svg.cesga.es