Strategies in Cluster-Design

Gerolf Ziegenhain, TU Kaiserslautern, Germany

Outline of This Talk● Look at the technologies once again● Provide more detail for making decisions● What to consider?● What should be avoided by all means?● Provide keywords / directions for further reading

● Less organized talk● Contains personal experience

Making Decisions● Strategic decisions:

– Do once, changes difficult, expensive● Setup relatively easy● Therefore know some numbers

(per person, group, university)– #jobs– Runtime of jobs– CPUs per job– Memory per job– Coupling of system: latency / bandwidth– Hdd storage (also consider final storage)

Buy or Build?● Buying

– Less work– More costs– You will have more than you want– Vendor may help in consulting

● Building yourself– More work– High learning effect– Less costs– You will have what you buy

Technological Overview

DHCP

NIS

Firewall

Queue

Syslog

Login1

Mirror

Mail

Boot

Admin Login2

NAS1

NAS3

NAS2

User1 User3

User2

Nodes

Components of a Cluster

Networking

A Word on Entropy● Managing 10 Workstations differs a lot from

managing a cluster● Entropy of cables

– Sort them immediately– Use colors– Use hook-and-loop-tape– Use printed labels

Choice of Hardware● Nodes● Networking● Overhead servers

Choosing Nodes

Example: Google

Example: Google● Stock hardware● Custom build low-tech cases● Modular approach● Components

– Mainboard, CPU, Memory– 2x HDD (Stripe)– UPS Battery

● Advantage:– Cheap– High learning effect

Example: BlueGene/P● PowerPC● Custom build

– Boards– Chips – Networking

● Advantage:– Scales very good

Buy a Rack● Common beowolf cluster● Buy ready-built 19”

pizza-boxes● Mounting in 19” rack

– Usually 42HE● Advantage

– Less work– High packing density

Use Ready-Built Desktops

Processors and Architectures?● Know your problem● What to know about your algorithms?

– How much memory?– Can the problem easily be decomposed?– What precision?

● Libraries– Do they exist for your problem (i.e. QM calculations)– Do they run on all architectures?

● Choices:– Architecture (usually AMD / Intel is a good choice)– #CPUs

Storage Management● Know your problem● Parameters to know

– How much HDD space?– What is the common bandwidth?

● Evaluating 100GB files in real-time?● Writing out 1TB files?

● Choices: – NAS (multiple?)– SAN– Distributed filesystem

Backup● RAID ≠ backup

– You still can kill your stuff byrm -rf /my_stuff

● Incremental backup – Critical user configuration– Configuration files– Complete overhead server installation

Networking

Types● Know your problem● Choices

– Bandwidth● Gbit < Infiniband● Gbit: channel bonding possible

– Latency time ● Gbit > SCI

– Scalability● Stacked network switches● Fat tree architecture

Switches● Important parameters

– Backbone speed ● throughput when all ports are under load?

– Can it be configured? ● Auto-Sensing ● IP ● ARP● ...

– Stackable?– (Uplink ports?)

Which #Cores/Node is Optimum?● Currently cheapest cost per core: 8 cores per node● Small systems (48 nodes)

– Doesn't matter because one switch is enough● Average systems

– Do you need all-to-all connections?– Different rings or change network topology– If you want to stick to single-switched networks:

current optimum is 16 cpu per node for this● Big system

– Go for fat tree network :)

Infrastructure Requirements● Cooling

– Each W burned in CPU heat⇒● Stable power supply

– Black out?– Fluctuations in voltage level

● Cheap power supplies will break on fluctuations

Notes about Power Consumption● Less power consumption

less heat ⇒ less defects(?)⇒

● Running costs per year can easily reach initial investment costs!– Do the math blade center could also pay off!⇒

● Do not switch on / off all nodes at once– Voltage peaks!

Decomposition of the Servers

Why Separate Login Nodes?● User interaction● May hang due to jobs ● Security

– Ssh ports open– May be hacked

● Configuration of user packages– System more on bleeding edge

Splitting Servers● Easily >10 overhead tasks● Why not in one big server?

– Security (one hole all broken)⇒– Stability– Maintenance

● Updates (what was done 3 years ago?)● Dependencies (how do software packages interfere?)● No plugin structure (no testing of different variants)

● Solution– Split the tasks >10 overhead servers⇒– Problem:

● Cost ● Hardware failures?

Combining Servers● Use XEN ● Host servers: 1...3 servers

– Hardware failure tolerant● Further advantages

– Extremely reduced costs– Complete rollback possible– Try different configurations

● Experiments are possible with limited budget– Clear separation of tasks

Administration

Administration Policies● Interaction with human beings

– Difficult social aspects– Good administrator is never realized (system works)

● Who has the root password?● Who will document what has been done?● Split the work, but communicate:

– Design decisions– Buying, writing grant proposals– Installation, bug fixing– Educating end-users

Administration Policies● User interaction

– Keep the users informed (mailing list)– Monitor system to cancel out problems before they

occur

Managing Different Groups● Impossible!● Each group has to provide at least one person

– Managing user education– Monitoring performance– Know the needs ( cluster design decisions)⇒

⇒ Sharing administrator not possible!● Sharing resources: possible & meaningful

What is the Critical Data?● What data has to be stored?

– User programs– Final data– May be put on RAID-Mirror

● What data can be exposed to potential loss?– Temporary files– May be put on RAID-Stripe

Compilation● Custom user programs / libraries● Where to install

– /usr/local/ (system-wide)– $HOME (per-user)

● Autotools provide possibility to install whole distribution in home-directory!

⇒ Depends on how often the code changes● Choosing a compiler

– GNU compilers are good & free– Special CPU instructions: buy a compiler

● Intel compiler● Portland compiler

Security● University networks are

– Insecure– Treasured victims

● Risks– Ssh password login– Open ports– Updating

● Keep up to date with serious bugs!– Users

● Therefore (attacks will happen on daily basis!)– Use firewall– Monitor system for odd behavior

Operating Systems

Which Operating System?● Different OS / distributions exist

– But widely compatible configuration– Way of doing stuff differs slightly in detail

● I.e. Directories / files– Watch out for licenses: BSD, GPL, ...

● OS: provide basic stable & secure functionality– Linux

● Debian● RedHat● SuSE (slow, costly, small community)

– FreeBSD (more secure, but ~older versions)– OpenBSD (most secure)

Updating or not?● Motivations

– Stability– Security– Features

● Possible solution: – Keep login servers and firewall up-to-date– Keep computation nodes stable (out-of-date)– Works only if nodes are in inner network

Rolling your own distribution● Possible solution for installation issues● Possibilities

– From scratch distribution– Modify existing distribution– Compiling only custom packages (/usr/local/bin)– Keep system hdd-images and clone them

Lesson Learned● Reproduceable?

– Making a distribution is exhausting● Documentation (wiki)

– Someday you have to handover– Or reinstall

● Keep a complete mirror – Packages may vanish

The Gentoo-Approach● Use source-packages● Autotools binary files⇒● Create special configuration files for dependencies

– In gentoo: portage (→ corvix: egatrop)– In bsd: ports

● Alternative– Linux from scratch

● Missing the configuration files ● Rely on autotools

– Arch linux● Websites are good sources for step-by-step

howtos

The Debian-Approach● Compile once, distribute binary packages● Create custom-packages with only one command● Advantage

– Extremely fast– Easier to maintain for big number of servers– Embedded devices use similar packages architecture

Our solution● Stable basis system:

– Debian overlays● Additional package source with custom packages

– Xen images of the installed debian-system ⇒ Even faster reinstallations

● Custom software– I.e. user demanded libraries– Compilation in ~

Other cluster distributions● Debian-Based / RedHat-Based exist

– I.e. RocksCluster, CentOS, PelicanHPC, Corvix, ...● Good source for howtos● Good as cheat-sheet● But

– HPC is inherently customized– Flexibility highest with customized installation– None of the distros solved a problem that we had

Thank you!

● Acknowledgements– Prof. Dr. rer. Nat. Herbert M. Urbassek,

TU Kaiserslautern, Germany