Date post: | 14-Jan-2016 |
Category: |
Documents |
Upload: | penelope-arnold |
View: | 223 times |
Download: | 0 times |
www.cs.man.ac.uk/cnc
Performance Control in RealityGrid
Ken Mayes, Mikel Luján, Graham Riley,Rupert Ford, Len Freeman, John Gurd
!st RealityGrid Workshop, June 2003
Overview
Preamble• The case for Performance Control
Context• Malleable, component-based Grid
applications
RealityGrid Application• The LB3D Use Case
RealityGrid Performance Control System• Design and status report
Generalisation . . .
“Performance Analysis and Distributed Computing” . . .
We want specific computational performance and resource levels
We want these in an environment that is• Distributed (physically)• Heterogeneous (inherently)• Dynamic (by necessity)
We want them with a minimum effort
. . . Implies Performanceand Resource Control
There is no viable alternative apart from automating the process of achieving them• monitoring and analysis (sensing)• change when unsatisfactory (actuation)
This is a classical control system scenario, and we need to use established control system techniques to deal with it
Control Systems Overview
Control systems involve appropriate change of actuated variables informed by (timely) feedback of monitoring information:
actuator
feedback function
error
Summary
Traditional performance “control” amounts to “open loop” control, mediated by human “performance engineers”
Distributed environments introduce enough extra complexity that “closed loop” control becomes essential
Successful closed loop control demands accurate and rapid feedback data, thus affecting achievable control limits
Context
Grid applications will be distributed and, in some sense, component-based.
To deliver the power grid model, adaptation is key!• Elevate users above Grid resource and performance
details.
Our work is considering adaptation and its impact on performance:• Adaptation in component deployment,
- Initial model configuration and deployment on resources.- Flexibility in deployment of compositions of coupled
models.
• Adaptation via malleable components.- At run time, re-allocation of resources in response to
changes.
RealityGrid - LB3D Use Case
Display simulation time equal
LB3D
LB3D
Output rate 1Resolution 1
Output rate 2Resolution 2
Tracking a parameter search
UserChanges
dynamically
T=100
T=100
Params 1
Params 2
LB3D Use Case
User Display rates ~equal
LB3D
LB3D
Output rate 1Resolution 1
Output rate 2Resolution 2
Tracking a parameter study
Params 1
Params 2
“Malleable” LB3D - mechanisms
LB3D will respond to requests to change resources• Use more (or less) pes (& mem) on current system• Move from one system to another
- Via machine independent (parallel) checkpoint/restart
LB3D will output science data (for remote visualisation) at higher or lower rates
LB3D will (one day) respond to requests to continue running at higher (or lower) lattice resolution
Each of the above affects performance (e.g. timesteps per second rate)
Each has an associated cost . . .
Use Case - continued
User might be tracking many parameter set developments (one per LB3D instance)
Some will be uninteresting (for a while)• Lower output rate / resolution / terminate
Some will become interesting• Increase output rate / resolution
One aim: Re-distribute resources across all LB3D instances to maintain highest possible timestep rate
A General Grid Application
GenerateData
ProcessData
VisualiseData
Component 1 Component 2 Component 3
Computational Grid Resources
Applications and components exhibit phased behaviour
Life is Hierarchical
Can we use hierarchy to “divide and conquer” complex system problems?
Introduce “Performance Steerers” at component- and application-levels . . .
Performance Steerers - Design
CPS CPS CPS
Component 1 Component 2 Component 3
Computational Grid Resources
APS
Component Framework
ExternalResourceScheduler
Initial deployment+
Run-time adaptation
Full System
CPS
Component 2
ExternalResourceScheduler
APS
Loader
Predictor
Predictor
ComponentPerformanceRepository
ApplicationPerformanceRepository
ResourceUsage
Monitor
Performance Prediction
Role of APS• To distribute available resources to the
components in such a way that the predicted performance of components gives a minimum predicted execution time.
Role of CPS• To utilise allocated resources and
component performance effectors (actuators) so that the predicted execution time of the steered component is minimised.
Life is Repetitive
Many programs iterate more-or-less the same thing over-and-over again
We can take advantage of this• e.g. for load balance in weather prediction• and, possibly, for performance control
Application Progress Points
Assume that the execution of the application proceeds through phases. Phase boundaries are marked by Progress Points.
NB. Can take decisions about performance and take actions at the progress points:• Must be safe points
0 1 2 3 4 5 6 7
Ph 1 Ph 2 Ph 3 Ph 4 Ph 5 Ph 6 Ph 7
Component Progress Points
Information about progress points will be contained in some repository.
Application progress points
Component progress points
APS
CPS
Component
Time
Implementation:
APS
CPS
Comms I/f
Comms I/f
LB3DComponent
Interface
APS as daemon, CPS as library
CPS Progress
Component control
Procedurecalls
Sockets(rpc)
Implementation:
APS
Machine 1
CPS LB3D
MPIRUN
Component Loader
DUROC + RSL + GLOBUS + GRIDFTP
Machine 3
Component Loader
Machine 2
shme
m
socket
socket
shme
m
Start-up Process
1. GlobusRun, RSL script for Component loaders (one per machine in Grid) plus APS daemon.
2. Loaders connect to each other.3. LB3D started by Loader (via MPIRUN), calls
CPS (a library) at start-up.4. CPS connects to APS.5. LB3D calls CPS at each subsequent progress
point and CPS communicates with APS.6. Continue until LB3D completed (e.g. no. of
timesteps complete).
Status We have a prototype implementation (basic
mechanisms)First Experiment:• Every N tsteps, move LB3D between machines 2 and
3 – determined by APS.• At tstep mod N progress point in LB3D, APS tells CPS
(which tells the component) to checkpoint, CPS writes certain status information to the shmem area and then LB3D (and CPS) dies.
• Loader on machine it ran on communicates to loader on machine it is to run on. The restart file is Gridftp’d along with restart info. (e.g. tstep to shmem area of new loader).
• New LB3D is started and CPS manages the restart.• Continue until no more tsteps.
Status Preliminary performance results
np 4, data 64x64x64checkpoint file (XDR) size 32.8 MBaverage resident tstep time for cronus 3.384 (s.)average migration tstep time to cronus 43.718 (s.)average resident tstep time for centaur3 6.675 (s.)average migration tstep time to centaur3 55.280
(s.)
np 4, data 8x8x8checkpoint file (XDR) size 64 KBaverage resident tstep time for cronus 0.017 (s.)average migration tstep time to cronus 0.504 (s.)average resident tstep time for centaur3 0.061 (s.)average migration tstep time to centaur3 3.038 (s.)
cronus = SGI Origin 3400 and centaur3 = Sun TCF
Status We have a prototype implementation
Second Experiment:• Three platforms (2 SGI, 1 Solaris – Linux
coming soon)• Move LB3D at random between machines,
as determined by APS.• Has exposed Gridftp problems – worked
around.• Have timings for 2 machines, but not yet 3
• Now looking at ICENI framework.
Status
Developing implementation of performance repository• Berkeley Database (linked as library)
Survey of prediction and machine learning algorithms • runtime vs. off line analysis• accuracy of predictions• amount of history data required
Learning control theory and understand how to apply it to Performance Control.
Generalisation
LB3D as a component is an “easy” case• how does the above scheme generalise?
Are components necessary? Is hierarchy necessary?
• component < application < scheduler
What different kinds of component and/or application might there be?• redefinition of the “process”• a topic for my forthcoming leave of absence
Summary Aim:
Develop an architecture that enables us to investigate different mechanisms for Performance Control of malleable component based applications on the Grid
Main characteristic: • dynamic adaptation
Design/implementation tensions:• general vs. specific purpose• APS <-> CPS communication• APS <-> CPS ratio• performance prediction algorithms - accuracy vs.
execution time• APS/CPS overhead vs. application execution time
Work in development (first year in one sentence): • A Grid Framework for Malleable Component-based
Application Migration
Related Projects at Manchester
Met Office – FLUME - design of next generation Unified Model software• Coupled models
Tyndall Centre – SoftIAM - Climate Impact, Integrated assessment modelling• Coupling climate and economic models
Computational Markets • 1 RA and 1 PhD - positions being filled at present• UK e-Science funded (Imperial College led)
For more information check http://www.cs.man.ac.uk/cnc
http://www.realitygrid.org