Algorithm-Based Master-Worker Model of Fault Tolerance in Time-Evolving...

transcript

Presented By:

Md. Mohsin Ali

April 24 2013

Algorithm-Based Master-Worker Model of Fault Tolerance in Time-Evolving

Applications

Authors: Md. Mohsin Ali and Peter E. Strazdins

Research School of Computer Science

The Australian National University

Canberra, ACT 0200, Australia

High Performance Computing High Performance Computing

application areas Atmosphere, Earth, Environmentapplication areas Atmosphere, Earth, Environment

Bioscience, Biotechnology, GeneticsBioscience, Biotechnology, Genetics

Chemistry, Molecular SciencesChemistry, Molecular Sciences

Computer Science, MathematicsComputer Science, Mathematics

Advanced graphics and virtual Advanced graphics and virtual

reality, etc.reality, etc.

Science and Engineering Industrial and Commercial Science and Engineering Industrial and Commercial

Introduction

Algorithm-Based Master-Worker Model of Fault Tolerance in Time-Evolving Applications | Md. Mohsin Ali |

High Performance Computing High Performance Computing

component hundreds of thousands of processing component hundreds of thousands of processing

elements to concurrently execute elements to concurrently execute

millions of threadsmillions of threads

Introduction

Size of HPC systems are becoming larger to meet Size of HPC systems are becoming larger to meet

current demandcurrent demand

Introduction

… … ButBut

System size ––– Probability (component failure)System size ––– Probability (component failure)

Introduction

… … ButBut

System size ––– Hardness of achieving parallelismSystem size ––– Hardness of achieving parallelism

Introduction

Frequency of FailureFrequency of Failure

*G. Gibson, B. Schroeder, J. Digney, 2007*G. Gibson, B. Schroeder, J. Digney, 2007

Introduction

Frequency of FailureFrequency of Failure

Reasons of FailureReasons of Failure

*G. Gibson, B. Schroeder, J. Digney, 2007*G. Gibson, B. Schroeder, J. Digney, 2007

Introduction

checkpoint/restartcheckpoint/restart

Ways of Failure Recovery

checkpoint/restartcheckpoint/restart

disadvantagesdisadvantages

I/O bottleneckI/O bottleneck

up to 25% of overhead in currentup to 25% of overhead in current

petascale systemspetascale systems

replicationreplication

need to keep data consistency need to keep data consistency

hard to find out proper places for replicationhard to find out proper places for replication

scalability is degradedscalability is degraded

message loggingmessage logging

same as replication but reduced message sizesame as replication but reduced message size

performance degradation caused by performance degradation caused by

synchronizationsynchronization

algorithm-basedalgorithm-based

advantagesadvantages

error detection, correction, and repeated error detection, correction, and repeated

computation are within the algorithm computation are within the algorithm

executing within a processing element (PE)executing within a processing element (PE)

errors are propagated on less number of PEerrors are propagated on less number of PE

““... partial di erential equations (PDEs) are the basis of all ff... partial di erential equations (PDEs) are the basis of all ff

physical theorems.physical theorems.””

Bernhard Riemann (1826-1866)Bernhard Riemann (1826-1866)

Motivation

Solution of PDEsSolution of PDEs

Time-evolving numerical methodsTime-evolving numerical methods

Motivation

Parallel version for complex PDEsParallel version for complex PDEs

Motivation

Even a single process failure postpone whole computation Even a single process failure postpone whole computation

Motivation

Even a single process failure postpone whole computation Even a single process failure postpone whole computation

More component on system causes more failure (More component on system causes more failure (and more complexity)and more complexity)

Motivation

Design and implementation of time-evolving applicationDesign and implementation of time-evolving application

tolerate process failuretolerate process failure

achieve high scalabilityachieve high scalability

To learn the usability of fault-tolerant semantics of FT-MPITo learn the usability of fault-tolerant semantics of FT-MPI

Challenges

How to detect processes failure?How to detect processes failure?

Challenges

How to determine which processes are failed?How to determine which processes are failed?

Challenges

How to recover failed processes?How to recover failed processes?

Challenges

How to recover How to recover ““lost state infolost state info”” of recovered processes? of recovered processes?

Challenges

How to continue time-step from the point of failure?How to continue time-step from the point of failure?

Challenges

How to retain scalability? How to retain scalability?

Challenges

MPI_ERR_OTHER semantics of FT-MPIMPI_ERR_OTHER semantics of FT-MPI

How to Tackle Challenges

Attribute catching mechanism of MPIAttribute catching mechanism of MPI

Creating new processes with the same rank as previousCreating new processes with the same rank as previous

> FT_MPI_CHECK_RECOVER and > FT_MPI_CHECK_RECOVER and

> MPI_Comm_dup semantics of FT-MPI > MPI_Comm_dup semantics of FT-MPI

Replaced byReplaced by

1D advection with periodic

boundary condition

““lost state infolost state info”” of recovered processes are recovered from of recovered processes are recovered from

master by FT-MPI process restart master by FT-MPI process restart

Every two-way exchange is going through master and save

“state info”on it

worker

boundary condition

““lost state infolost state info”” of recovered processes are recovered from of recovered processes are recovered from

master by FT-MPI process restartmaster by FT-MPI process restart

time-stepping is continued from one step backwardstime-stepping is continued from one step backwards36

Every two-way exchange is going through master and save

“state info”on it

worker

boundary condition

Sending from Master (Process 0) is FailedSending from Master (Process 0) is Failed

Sending from Worker (Process > 0) is FailedSending from Worker (Process > 0) is Failed

How to retain scalability?How to retain scalability?

Scalability is very low in this master-worker modelScalability is very low in this master-worker model

# cores 16 (total)# cores 16 (total)

# nodes 4 (total)# nodes 4 (total)

Memory 4 GB (each node)Memory 4 GB (each node)

standard GigE Switch standard GigE Switch

Overhead of FT-MPI over Open MPI

Scalability achieved for 16 cores (4 nodes) = 15% (very low)Scalability achieved for 16 cores (4 nodes) = 15% (very low)

Recovery time Recovery time

1 worker process failed = 1 sec1 worker process failed = 1 sec

4 worker processes failed = 2 sec4 worker processes failed = 2 sec

Scalability and Recovery Time

Checkpointing after each T time-steps on a specific node Checkpointing after each T time-steps on a specific node

Future Work

Checkpointin after each T time-steps on separate nodes Checkpointin after each T time-steps on separate nodes

Future Work

System size ––– Hardness of achieving parallelismSystem size ––– Hardness of achieving parallelism

Process failure detection by FT-MPIProcess failure detection by FT-MPI

Failed process restart by FT-MPI Failed process restart by FT-MPI

Algorithm-based fault tolerance technique for data recoveryAlgorithm-based fault tolerance technique for data recovery

Overhead of FT-MPI compared to Open MPI is lowOverhead of FT-MPI compared to Open MPI is low

Recovery time is lessRecovery time is less

Master-worker model is not so scalable, but can be used as a Master-worker model is not so scalable, but can be used as a

prototypeprototype

Conclusion

Thank You!Thank You!

53Algorithm-Based Master-Worker Model of Fault Tolerance in Time-Evolving Applications | Md. Mohsin Ali |

Algorithm-Based Master-Worker Model of Fault Tolerance in Time-Evolving...

Documents