Practical Guidelines for Moab Stacks

© 2013 ADAPTIVE COMPUTING, INC. 1

Practical Guidelines for Highly Available Moab Stacks

Daniel Hardman, Chief Solutions Architect

@dhh1128 ~ http://codecraft.co ~ http://gplus.to/danielhardman ~ http://lnkd.in/z7PTAR

[email protected]

April 2013

© 2013 ADAPTIVE COMPUTING, INC. 2 © 2013 ADAPTIVE COMPUTING, INC. 2

The Goal of HA

…NOT! :-)


The real goals of HA

▪  Eliminate or reduce “downtime” for running jobs

▪  Eliminate or reduce “downtime” for new submissions

▪  Make failovers visible and manageable ▪  Satisfy regulatory requirements ▪  Preserve audit trail


HA is constrained by time, money

How much are you willing to spend to tolerate: ▪  A power outage? ▪  A software crash? ▪  A hacker from unit 61398 in Shanghai? ▪  The Chelyabinsk meteor? ▪  The Chicxulub meteor that wiped out the

dinosarus?


What is “downtime”?

0 – hardware failure

+3 min – usable, but very slow

-30 min – last checkpoint

+10 min – full restore


4 Basic Recipes

▪  Simple built-in HA ▪  Standard pairwise HA ▪  Shared pairwise HA ▪  Advanced HA


Recipe 1: simple, built-in HA


Simple, built-in HA

▪  hot ~ warm (daemons idle on fallback svr)

▪  Moab, TORQUE ▪  shared file system, synced clocks, two daemons,

last mod date on semaphore

▪  MAM ▪  DB replication, primary and fallback server


Sample deployment (simple, built-in HA)


Pros and cons (simple, built-in HA)

▪  Pros ▪  Fast and easy to set up ▪  Minimal learning curve

▪  Cons ▪  Doesn’t protect the solution DB, MWS, Viewpoint ▪  Depends on synchronized clocks, reliable

propagation of file metadata in shared fs ▪  Risk of false triggers ▪  Shared FS may be single point of failure,

depending on how it’s implemented


Recipe 2: standard, pairwise HA


Standard, pairwise HA

▪  Twin headnodes (all daemons) ▪  hot ~ cold (daemons inert on fallback svr) ▪  Heartbeat, redhat clustering ▪  Replicated FS (DRBD)


Sample deployment (standard, pairwise HA)


Pros and cons (standard, pairwise HA)

▪  Pros ▪  All services fail over the same way ▪  Heartbeat is robust, well understood ▪  FS can’t be a single point of failure

▪  Cons ▪  Some vulnerability to “split brain” scenario ▪  More learning curve ▪  More complexity than simple, built-in HA


Recipe 3: shared, pairwise HA


Shared, pairwise HA

▪  Twin headnodes (all daemons) ▪  hot ~ warm (some daemons inert, some

idle on fallback svr) ▪  Heartbeat, redhat clustering ▪  DB failover ▪  Shared FS (e.g., GFS2)


Sample deployment (shared, pairwise HA 1)


Sample deployment (shared, pairwise HA 2)


Pros and cons (shared, pairwise HA)

▪  Pros ▪  Solves “split brain” scenario ▪  May have slightly lower latency

▪  Cons ▪  Greater learning curve ▪  More complexity


Recipe 4: advanced HA


Advanced HA

▪  Each service (potentially) split onto dedicated box

▪  Daemons are paired and fail over with heartbeat, redhat clustering

▪  DB failover ▪  Replicated or shared FS


Advanced HA

This is less of a recipe, and more of a general pattern. Each unique server role has to have N-way redundancy. Complexity of config is high; we recommend involvement of professional services.


Pros and cons (advanced HA)

▪  Pros ▪  Can meet very aggressive SLAs ▪  Can be tailored and fine-tuned

▪  Cons ▪  Major implementation effort ▪  Requires sophisticated learning and monitoring


General Observations

▪  Important to audit ▪  Super-fast failover not a goal in our

recipes ▪  Security implications ▪  Not perf enhancer ▪  Not scalability enhancer ▪  Not DR


More Info

Whitepaper now available. Email me ([email protected]) for a copy, or download from /documents/ha-moab-cloud-hpc.pdf. Documentation for Hopper release includes a new HA task guide for simple, built-in HA configuration.

Date post:	19-Jul-2015
Category:	Technology
Upload:	insidehpc
View:	191 times
Download:	4 times

Practical Guidelines for Moab Stacks

Technology