Polish InfrastructurePolish Infrastructurefor Supporting Computational Sciencefor Supporting Computational Science
in the European Research Spacein the European Research Space
Component Approach to Distributed Component Approach to Distributed Multiscale SimulationsMultiscale Simulations
Katarzyna Rycerz(1,2), Marian Bubak(1,2)Katarzyna Rycerz(1,2), Marian Bubak(1,2)(1) Institute of Computer Science AGH, Mickiewicza 30, 30-059 Kraków, Poland(1) Institute of Computer Science AGH, Mickiewicza 30, 30-059 Kraków, Poland
(2) ACC Cyfronet AGH, ul. Nawojki 11, 30-950 Kraków, Poland(2) ACC Cyfronet AGH, ul. Nawojki 11, 30-950 Kraków, Poland
KU KDM, Zakopane, 18-19.03.2010
2
OutlineOutline
Requirements of multiscale simulationsMotivation for a component model for such simulationsHLA-based component model (idea, design challenges,
possible solutions)Experiment with Multiscale Multiphysics Scientific
Environment (MUSE)Possible integration with GridSpace VLSummary
3
Multiscale SimulationsMultiscale Simulations
Consists of modules of different scaleExamples – e.g. modelling:
reacting gas flows capillary growth colloidal dynamics stellar systems (e.g. Multiscale Multiphysics Scientific
Environment – MUSE used in this work) and many more ...
4
Multiscale Simulations - RequirementsMultiscale Simulations - Requirements
Actual connection of two or more models together obeying the law of physics (e.g. conservation law) advanced time management: ability to connect modules with
different time scales and internal time management strategies support for connecting models of different space scale
Composability and reusability of existing models of different scale finding existing models needed and connecting them either
together or to the new models ease of plugging in and unplugging them from the running
system standarized models’ connections + many users sharing their
models = more chances for general solutions
5
MotivationMotivation To wrap simulations into recombinant components that can be selected and
assembled in various combinations to satisfy requirements of multiscale simulations Need for a special component model that:
• provides machanisms specyfic for distributed multiscale simulations adaptation of one of the existing solutions for distributed
simulations – our choice – High Level Architecture (HLA)• supports long running simulations - setup and steering of components
should be possible also during runtime• gives a possibility to wrap legacy simulation kernels into components
Need for an infrastructure that facilitates cross-domain exchange of components among scientists need for support for the component model using Grid solutions (e-infrastructures) for crossing administrative domains
6
Related workRelated work
Model Couling Toolkit applies a message passing (MPI) style of communication between simulation models. oriented towards domain data decomposition of the simulated problem provides a support for advanced data transformations between different models J. Larson, R. Jacob, E. Ong ”The Model Coupling Toolkit: A New Fortran90 Toolkit for Building
Multiphysics Parallel Coupled Models.” 2005: Int. J. High Perf. Comp. App.,19(3), 277-292.
Multiscale Multiphysics Scientific Environment (MUSE) a software environment for astrophysical applications scripting approach (Python) is used to couple models together. models include: stellar evolution, hydrodynamics, stellar dynamics and radiative transfer sequential execution S. Portegies Zwart, S. McMillan, at al. A Multiphysics and Multiscale Software Environment for
Modeling Astrophysical Systems, New Astronomy, volume 14, issue 4, year 2009, pp. 369 - 378
The Multiscale Coupling Library and Environment (MUSCLE) provides a software framework to build simulations according to the complex automata theory introduces concept of kernels that communicate by unidirectional pipelines dedicated to pass a
specific kind of data from/to a kernel (asynchronous communication) J. Hegewald, M. Krafczyk, J. Tlke, A. G. Hoekstra, and B. Chopard. An agent-based coupling
platform for complex automata. ICCS, volume 5102 of Lecture Notes in Computer Science, pages 227233. Springer, 2008.
7
Why High Level Architecture (HLA) ?Why High Level Architecture (HLA) ?
Introduces the concept of simulation systems (federations) built from distributed elements (federates)
Supports joining models of different time scale - ability to connect simulations with different internal time management in one system
Supports data management (publish/subscribe mechanism) Separates actual simulation from communication between fedarates Partial support for interoperability and reusability (Simulation Object
Model (SOM), Federation Object Model (FOM), Base Object Model (BOM))
Well-known IEEE and OMT standard Reference implementation – HLA Runtime Infrastructure (HLA RTI) Open source implementations available – e.g. CERTI, ohla
8
HLA Component ModelHLA Component Model
Model differs from common models
(e.g. CCA) – no direct connections, no remote procedure call (RPC)
Components run concurrently and communicate using HLA mechanisms
Components use HLA facilities (e.g. time and data management)
Differs from original HLA mechanism: interactions can be dynamically
changed at runtime by a user change of state is triggered from
outside of any federate
BBAuses port provides port
CCA model
HLA model
ABB
federationtime managementdata managementother HLA mechanisms
join/resign C
set time policypublish/unpublishsubscribe/unsubscribe, etc.
9
HLA components design challengesHLA components design challenges
Transfer of control between many layers
requests from the Grid layer outside the component
simulation code layer HLA RTI layer.
The component should be able to efficiently process concurrently:
actual simulation that communicates with other simulation components via RTI layer
external requests of changing state of simulation in HLA RTI layer .
Simulation Code
CompoHLA library
HLA RTI
Component HLA
Component HLA
Grid platform (H2O)
External requests:start/stopjoin/resignset time policypublish/subscribe
Grid platform (H2O)
10
Preliminary solution - MPreliminary solution - Mechanism of HLA RTIechanism of HLA RTI CConcurrent oncurrent AAccess ccess CControlontrol
Use concurrent access exception handling available in HLA
Transparent to developer Synchronous mode - requests
processed as they come simulation is running in a
separate thread Dependent on implementation of
concurrency control in used HLA RTI
Concurrency difficult to handle effectively e.g starvation of requests that
causes overhead in simulation execution
Simulation Code
CompoHLA library
HLA RTI (concurrent access control)
Component HLA
Component HLA
Grid platform (H2O)
External requests
Grid platform (H2O)
11
Advanced Solution - Use Active Object PatternAdvanced Solution - Use Active Object Pattern
Requires to call a single routine in a simulation loop
Asynchronous mode - separates invocation from execution
Requests processed when scheduler is called from simulation loop
Independent on behavior of HLA implementation
Concurrency easy to handle JNI used for communication
between Simulation Code, Scheduler and CompoHLA library
Simulation Code
CompoHLA library
HLA RTI
Component HLA
Component HLA
Grid platform (H2O)
External requests
Grid platform (H2O)
Scheduler
Queue
12
Interactions between components in example Interactions between components in example experiment experiment
Modules taken from Multiscale Multiphysics Scientific Environment (MUSE)
Multiscale simulation of dense stellar systems Two modules of different time scale:
stellar evolution (macro scale) stellar dynamics - N-body simulation
(meso scale) Data management
mass of changed stars are sent from evolution (macro scale) to dynamics (meso scale)
no data is needed from dynamics to evolution
data flow affects whole dynamics simulation
Dynamics takes more steps than evolution to reach the same point of simulation time
Time management - Regulating federate (evolution) regulate the progress in time of constrained federate (dynamics)
The maximal point in time which the constrained federate can reach (LBTS) at certain moment is calculated dynamically according to the position of regulating federate on the time axis
LBTS - Other federates will not send messages before this time.
Federate may only advance time within this interval
Federate’s current logical time.
Federate’seffective logical time.
Federate may not publish messages within this interval
Federate’s current logical time.
t=0
Lookahead
Constrained federate(dynamics)
Regulating federate (evolution)
13
Usage example – MUSE applicationUsage example – MUSE application
H2O kernel
Grid side A
H2O kernel
Grid side B
Component user
Component Client
Asks chosen components to join into a simulation system (called federation in HLA terminology)
Asks chosen components to publish or subscribe to certain data objects (e.g. Stars)
Asks components to set their time policy
Dynamics HLAComponent
EvolutionHLAComponent
HLA federation
join federation
join federation
subscribe publishbe constrained be regulating
Dynamics HLAComponent
EvolutionHLAComponent
14
Usage example – MUSE applicationUsage example – MUSE application
H2O kernel
Grid side A
H2O kernel
Grid side B
Component user
Component Client
Asks components to start Alter the
publications/subscriptions/time policy during runtime
Dynamics HLAComponent
EvolutionHLAComponent
HLA federation
start startunpublish
Star data objectStar data object
15
Experiment ResultsExperiment Results Comparision of:
Concurrent execution, conservative approach of dynamics and evolution as HLA components
Sequential execution (MUSE) Timing of:
Request processing (through grid and component layer)
Request realisation (scheduler) H2O v2.1 as a Grid platform and HLA
CERTI v 3.2.4 – open source Experiment run on DAS3 grid nodes in:
Delft (MUSE sequential version and dynamics component)
Amsterdam UvA (evolution component) Leiden (component client) Amsterdam VU (RTIexec control
process)
Each grid node is a cluster of two 1-GHz Pentium-IIIs nodes connected with internal Myrinet-2000 network
10Gb ethernet used as the external network between Grid nodes
16
Possible Integration with GridSpace VLPossible Integration with GridSpace VL
Modules that can be reused: IDE for
Experiment Script Execution Engine Registry Scenario
Repository
Extensions needed: Support for HLA component descriptions that include events/objects
produced/consumed by a component Component Description Assember will guide the user in joining
component descriptions into simulation description (Federation Object Model–like files).
IDE for ExperimentScript
ComponentDescription Assembler
Repositoryviewer
Simulations’ descriptions (based on HLA FOM)
-objects and events exchanged between Components by HLA RTI
Execution engineRegistry
-models descriptions- data sources
Simulation ScenarioRepository
-experiments scripts-FOM-like simulation’s
descriptions
E-infrastructure
HLA ComponentSimulationModel A
HLA ComponentSimulationModel B
HLA ComponentSimulationModel C
Experiment script-operations on components
-definiton of I/O data sources
17
Future workFuture work
A description language for connecting HLA components: Currently used: HLA FOM - definition of structures of data objects
and events that need to be passed between HLA components Needs to contain more information especially related to modules’
scale. Needs to support different data types e.g. arrays often used in
legacy implementations of simulation models etc. Interactivity:
the support for components that are sources of data streams - often a long running simulations - produce partial results that should be streamed to the user before the simulation actually stops.
the ability to interpret commands given to HLA components in the interactive mode
18
SummarySummary
Presented HLA component model enables the user to dynamically compose/decompose distributed simulations from multiscale elements residing on the Grid
Architecture of the HLA component supports steering of interactions with other components during simulation runtime
The presented approach differs from that in original HLA, where all decisions about actual interactions are made by federates themselves.
The functionality of the prototype is shown on the example of multiscale simulation of a dense stellar system – MUSE environment.
Experiment results show that that grid and component layers do not introduce much overhead.
In the future we plan to fully integrate the HLA components with GridSpace Virtual Laboratory
19
ReferencesReferences
K. Rycerz, M. Bubak, and P. M. A. Sloot, Using HLA and Grid for Distributed Multiscale Simulations, in: R. Wyrzykowski, J. Dongarra, K. Karczewski, and J. Wasniewski (Eds.), Proceedings of 7-th International Conference, PPAM 2007, Gdansk, Poland, September 2007, LNCS 4967, Springer 2008, pp.780-787
K. Rycerz, M. Bubak and P.M.A. Sloot, Dynamic Interactions in HLA Component Model for Multiscale Simulations, ICCS, volume 5102 of Lecture Notes in Computer Science, pages 217-226. Springer, 2008.
K. Rycerz, M. Bubak, P. M. A. Sloot: HLA Component Based Environment For Distributed Multiscale Simulations In: T. Priol and M. Vanneschi (Eds.), From Grids to Service and Pervasive Computing, Springer, 2008, pp. 229-239
K. Rycerz, M. Bubak, P. M. A. Sloot : Collaborative Environment for HLA Component-Based Distributed Multiscale Simulations (in preparation)
Grid Space webpage http://gs.cyfronet.pl/ PL-Grid Project, http://www.plgrid.pl/en
29
Interactions between components in example Interactions between components in example experiment experiment
Modules taken from Multiscale Multiphysics Scientific Environment (MUSE)
Multiscale simulation of dense stellar systems Two modules of different time scale:
stellar evolution (macro scale) stellar dynamics - N-body simulation (meso
scale) Data management
mass of changed stars are sent from evolution (macro scale) to dynamics (meso scale)
no data is needed from dynamics to evolution data flow affects whole dynamics simulation
Dynamics takes more steps than evolution to reach the same point of simulation time
Time management - Regulating federate (evolution) regulate the progress in time of constrained federate (dynamics)
The maximal point in time which the constrained federate can reach (LBTS) at certain moment is calculated dynamically according to the position of regulating federate on the time axis
LBTS - Other federates will not send messages before this time.
Federate may only advance time within this interval
Federate’s current logical time.
Federate’seffective logical time.
Federate may not publish messages within this interval
Federate’s current logical time.
t=0
Lookahead
Constrained federate(dynamics)
Regulating federate (evolution)