April 19, 2023

Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark

CoreGRID Integration Workshop, Pisa

A. Iosup, D.H.J. Epema Jason Maassen, Rob van NieuwpoortPDS Group, EEMCS, TU Delft Computer Science Dept., VU Amsterdam

April 19, 20232

Collaboration context

• The Partners• WP6

Technische Universiteit Delft (TU Delft)Alexandru Iosup, Dick Epema

• WP3Vrije Universiteit Amsterdam (VUA)Jason Maassen, Rob van Nieuwpoort

• This work is the result of the collaboration of the Partners, within the CoreGRID framework.

April 19, 20233

Outline

• Introduction• The GrenchMark framework• Experience with GrenchMark • Extending GrenchMark• Conclusions

April 19, 20234

Introduction

• Synthetic worklods are the basis of grid schedulers comparison [our work], and of ...• Grid systems analysis• Functionality testing in grid environments• Comparing grid settings• Grid systems design and procurement• Many others …

• Synthetic grid workloads• Many different types of representative Grid applications

• Need a standard tool for synthetic grid workloads generation and submission

April 19, 20235

Representative Grid applications (1/3)

• Unitary applications• Just one scheduling unit • Examples:

Sequential, MPI, Java RMI, Ibis, …

• Composite applications• Composed of several unitary or composite

applications• Examples:

Parameter sweeps, chains of tasks, DAGs, workflows, …

April 19, 20236

Representative Grid applications (2/3)Unitary: synthetic

[I] I1 data elements from stdin

[O] O1 data elements to stdout

for each N steps (i) # superstep

[I] I2 data elements from stdin

for each M memory locations (j)

[M] get item j of size S

[P] compute C computation units per memory item (fmul), and store results into temp memory location

[M] put S values from temp to location j

[O] O2 data elements to stdout

[O] O3 data elements to stdout

• Also with I/O• Also parallel,

with communication

• Synthetic application model with CPU cycles, memory, and I/O requirements

April 19, 20237

Representative Grid applications (3/3)Unitary: Ibis jobs• No modeling, just real parallel

applications[ http://www.cs.vu.nl/ibis ]

• physical simulation• parallel rendering• computational mathematics• state space search• bioinformatics• data compression• grid methods• optimization

April 19, 20238

The KOALA scheduler

• Grid scheduler [ http://www.st.ewi.tudelft.nl/koala/ ]• Native processor and data co-allocation support

H.H. Mohamed and D.H.J. Epema, An Evaluation of the Close-to-Files Processor and Data Co-Allocation Policy in Multiclusters, IEEE Int'l Conference Cluster Computing, Sep 2004.

• By PDS group, EEMCS, PDS / Virtual Laboratory for e-Science

• Application-specific support• Ibis jobs• Multi-site MPI jobs• …

April 19, 20239

The GrenchMark framework

• What’s in a name?grid benchmark → help standardizing the testing procedures, but benchmarks are too early…

• GrenchMarkA systematic approach to testing Grid schedulers• A set of metrics for comparing schedulers• A set of representative Grid applications

• Both real and synthetic• Easy-to-use tools to create synthetic workloads• Flexible, portable, extensible

• Can also be used for testing other Grid components, …

April 19, 202310

The GrenchMark framework

April 19, 202311

GrenchMark status

• Already done in Python• Generator + Globus, KOALA generators + RSL printer• Submitter• Results analyzer (crude)

• Applications• Unitary, 3 types: sequential, MPI, Ibis (Java)• +35 different applications• Composite applications: DAG-based

• Ongoing work• Automated results analyzer• Better modeling capabilities

April 19, 202312

GrenchMark for testing KOALA…

• KOALA

• Non-fixed, fixed, semi-fixed jobs

• Test & evaluate• Status: pre-release, supposed stable• Workloads with different jobs requirements, inter-arrival

rates, co-allocated v. single site jobs…• Jobs success rate, KOALA’s overhead and bottlenecks

• Results• +5,000 jobs successfully run• 2 major bugs first day, +10 bugs overall (all fixed)• KOALA is officially released

(full credit to KOALA developers, 10x for testing with GrenchMark)

April 19, 202313

… and DAS-2’s functionality …

• Already done• Evaluate for KOALA + Globus + DAS-2

• jobs success rate, turnaround time, middleware overhead, types and sources of errors

• Results• 5 workloads• 500 jobs

A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema,

Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, CoreGRID IW, Nov 2005 [here, now].

April 19, 202314

… and for analyzing, testing, and comparing grids

• Grid system analysis• Performance testing / analysis• What-if analysis

• Functionality testing in grid environments• System functionality testing• Periodic testing

• Comparing grid settings• Single-site vs. co-allocated jobs• Unitary vs. composite jobs

A.Iosup, D.H.J.Epema, GrenchMark: A Framework for Analyzing, Testing, and Comparing Grids, 2005 (submitted).

April 19, 202315

Extending GrenchMark

Motto:Extending GrenchMark is easy!

1. Write your own Job Generators• a function with a predefined name in a Python module• auto-loaded

2. Write your own Unit Generators• a function with a predefined name in a Python module• auto-loaded

3. Interface with C/C++, Ruby, Perl, Java, …• define your own protocol

4. Write your own printers• a function with a predefined name in a Python module• auto-loaded

April 19, 202316

Conclusions and future work

• GrenchMark generates diverse workloads of Grid applicationseasy-to-use, flexible, portable, extensible, …

• Experienceused GrenchMark to test KOALA’s functionality and performance. used GrenchMark to test some DAS Grid functionality. +5,000 jobs generated and run … and counting.

• Ongoing collaboration within CoreGRIDwith U. Dortmund group (R. Yahyapour, B. Song, C. Ernemann, …)

• … advertisementHave specific Grid applications types you would like to test? Test with GrenchMark!

April 19, 202317

Thank you!

Questions? Remarks? Observations? All welcome!

Grenchmark http://grenchmark.st.ewi.tudelft.nl/ [ “grenchmark”]

Alexandru IOSUPTU Delft

A.Iosup@ewi.tudelft.nl[ “iosup”]

Also: H. Mohamed and W. Lammers (KOALA), P. Anita (admin) G. Wrzesinska, N. Drost, and M. den Burger (Ibis)