Practical Guidelines for Moab Stacks

transcript

Practical Guidelines for Highly Available Moab Stacks

Daniel Hardman, Chief Solutions Architect

@dhh1128 ~ http://codecraft.co ~ http://gplus.to/danielhardman ~ http://lnkd.in/z7PTAR

dhardman@adaptivecomputing.com

April 2013

The Goal of HA

…NOT! :-)

The real goals of HA

▪  Eliminate or reduce “downtime” for running jobs

▪  Eliminate or reduce “downtime” for new submissions

▪  Make failovers visible and manageable ▪  Satisfy regulatory requirements ▪  Preserve audit trail

HA is constrained by time, money

How much are you willing to spend to tolerate: ▪  A power outage? ▪  A software crash? ▪  A hacker from unit 61398 in Shanghai? ▪  The Chelyabinsk meteor? ▪  The Chicxulub meteor that wiped out the

dinosarus?

What is “downtime”?

0 – hardware failure

+3 min – usable, but very slow

-30 min – last checkpoint

+10 min – full restore

4 Basic Recipes

▪  Simple built-in HA ▪  Standard pairwise HA ▪  Shared pairwise HA ▪  Advanced HA

Recipe 1: simple, built-in HA

Simple, built-in HA

▪  hot ~ warm (daemons idle on fallback svr)

▪  Moab, TORQUE ▪  shared file system, synced clocks, two daemons,

last mod date on semaphore

▪  MAM ▪  DB replication, primary and fallback server

Sample deployment (simple, built-in HA)

Pros and cons (simple, built-in HA)

▪  Pros ▪  Fast and easy to set up ▪  Minimal learning curve

▪  Cons ▪  Doesn’t protect the solution DB, MWS, Viewpoint ▪  Depends on synchronized clocks, reliable

propagation of file metadata in shared fs ▪  Risk of false triggers ▪  Shared FS may be single point of failure,

depending on how it’s implemented

Recipe 2: standard, pairwise HA

Standard, pairwise HA

▪  Twin headnodes (all daemons) ▪  hot ~ cold (daemons inert on fallback svr) ▪  Heartbeat, redhat clustering ▪  Replicated FS (DRBD)

Sample deployment (standard, pairwise HA)

Pros and cons (standard, pairwise HA)

▪  Pros ▪  All services fail over the same way ▪  Heartbeat is robust, well understood ▪  FS can’t be a single point of failure

▪  Cons ▪  Some vulnerability to “split brain” scenario ▪  More learning curve ▪  More complexity than simple, built-in HA

Recipe 3: shared, pairwise HA

Shared, pairwise HA

▪  Twin headnodes (all daemons) ▪  hot ~ warm (some daemons inert, some

idle on fallback svr) ▪  Heartbeat, redhat clustering ▪  DB failover ▪  Shared FS (e.g., GFS2)

Sample deployment (shared, pairwise HA 1)

Sample deployment (shared, pairwise HA 2)

Pros and cons (shared, pairwise HA)

▪  Pros ▪  Solves “split brain” scenario ▪  May have slightly lower latency

▪  Cons ▪  Greater learning curve ▪  More complexity

Recipe 4: advanced HA

Advanced HA

▪  Each service (potentially) split onto dedicated box

▪  Daemons are paired and fail over with heartbeat, redhat clustering

▪  DB failover ▪  Replicated or shared FS

Advanced HA

This is less of a recipe, and more of a general pattern. Each unique server role has to have N-way redundancy. Complexity of config is high; we recommend involvement of professional services.

Pros and cons (advanced HA)

▪  Pros ▪  Can meet very aggressive SLAs ▪  Can be tailored and fine-tuned

▪  Cons ▪  Major implementation effort ▪  Requires sophisticated learning and monitoring

General Observations

▪  Important to audit ▪  Super-fast failover not a goal in our

recipes ▪  Security implications ▪  Not perf enhancer ▪  Not scalability enhancer ▪  Not DR

More Info

Whitepaper now available. Email me (dhardman@adaptivecomputing.com) for a copy, or download from /documents/ha-moab-cloud-hpc.pdf. Documentation for Hopper release includes a new HA task guide for simple, built-in HA configuration.

Practical Guidelines for Moab Stacks

Technology