Computer Science 320

Load Balancing with Clusters

Mandelbrot on a Cluster

• Each node has a portion of the pixel data matrix

• After computation, each portion of the matrix sent to one process (process 0) which then writes the data to the image file

• The pixel data matrix is partitioned into K row slices, which are gathered after computation finishes

Row slicing among K = 4 processes

Process 0 Matrix After Gather

Execution Timeline

Data for Cluster Program// Start timing here// Initialize world and get world, size, and rank// Parse command line data here

static in xoffset;static int yoffset;

static in[][] matrix;static PJColorImage image;static Range[] ranges;static Range myRange;static in mylb;static int myub;

static IntegerBuf[] slices;static IntegerBuf mySlice;

static int[] hueTable;

Initializing the Data

// Same command line data as in SMP program go here

matrix = new int[height][];ranges = new Range(0, height – 1).subranges(size);myRange = ranges[rank];mylb = myRange.lb();myub = myRange.ub();

if (rank == 0) Arrays.allocate(matrix, width);else Arrays.allocate(matrix, myRange, width);

Process 0 gets the whole matrix; others get only their range

Set Up the Buffers and Compute!

slices = IntegerBuf.rowSliceBuffers(matrix, ranges);mySlice = slices[rank];

// Set up the hue table as in SMP program

for (int r = mylb; r <= myub; ++r) // Test for membership in Mandelbrot as before and // store the iteration count in the matrix // at position (r, c)

Gather the Data and Write It Outworld.gather(0, mySlice, slices);

if (rank == 0) // Same code as in SMP program for output to image file

Performance

Load Balancing

• For SMP situation, use a parallel for loop with a dynamic or guided schedule

• For a cluster, use a master-worker pattern

• The master node schedules agenda tasks for the other nodes

• A separate thread on the master runs a task as well

Master-Worker Pattern

One Master and K Workers

The Communication Sequence

• Master thread sends first K row slices to worker threads; master thread waits for results

• Master uses wildcard receive for results

• A range object arrives first, then the data

• Master sends range of next row to this worker thread

• Master sends a null message when last row is received

Master-worker execution timeline

Types of Messages and Tags

• Three types of messages: master sends a range, master receive a range, master receives pixel data

• Distinguish these by tagging them with message tags

• A receive with a tag will only match a send with the same tag

world.send(toRank, tab, buffer);

...

world.receive(fromRank, tag, buffer);

New Data for MandelbrotSetClu2// Message tagsstatic final int WORKER_MSG = 0;static final int MASTER_MSG = 1;static final int PIXEL_DATA_MSG = 2;

// Number of chunks the worker completedstatic in chunkCount;

Master and Worker Sectionsif (rank == 0) new ParallelTeam(2).exccute(new ParallelRegion(){ public void run(){ execute( new ParallelSection(){ public void run(){ masterSection(); } }, new ParallelSection(){ public void run(){ workerSection(); } }); } });else workerSection();

The Master Section: Sending the Dataprivate static void masterSection() throws Exception{ int worker; Range range; matrix = new int{height][width]; IntegerSchedule schedule = IntegerSchedule.runTime(); schedule.start(size, new Range(0, height – 1); int activeWorkers = size; for (int worker = 0; worker < size; ++worker){ range = schedule.next(worker); world.send(worker, WORKER_MSG, ObjectBuf.buffer(range)); if (range == null) --activeWorkers; ...

The Master Section: Receiving Resultsprivate static void masterSection() throws Exception{ ... ...

while (activeWorkers > 0){ ObjectItemBuf<Range> rangeBuf = ObjectBuf.buffer(); CommStatus status = world.receive(null, MASTER_MSG, rangeBuf); worker = status.fromRank; range = rangeBuf.item; world.receive(worker, PIXEL_DATA_MSG, IntegerBuf.rowSLiceBuffer(matrix, range)); range = schedule.next(worker); world.send(worker, WORKER_MSG, ObjectBuf.buffer(range)); if (range == null) --activeWorkers; }}

The Worker Section: Doing the Rowsprivate static void workerSection() throws Exception{ int[][] slice = null; for (;;){ ObjectItemBuf<Range> rangeBuf = ObjectBuf.buffer(); world.receive(0, WORKER_MSG, rangeBuf); Range range = rangeBuf.item; if (range == null) break; int lb = range.lb(); int ub = range.ub(); int len = range.length(); ++chunkCount; if (slice == null || slice.length < len) slice = new int[len][width]; for (int r = lb; r <= ub; ++r){ int slice_r = slice[r – lb]; // yahdah, yahdah, yahdah

...

The Worker Section: Send Resultsprivate static void workerSection() throws Exception{ int[][] slice = null; for (;;){ ...

for (int r = lb; r <= ub; ++r){ int slice_r = slice[r – lb]; // yahdah, yahdah, yahdah

world.send(0, MASTER_MSG, rangeBuf);

world.send(0, PIXEL_DATA_MSG, IntegerBuf.rowSliceBuffer(slice, new Range(0, len – 1))); }}

Running the Program$ java –Dpj.schedule=“dynamic(10) . . .

Running the Program$ java –Dpj.schedule=“dynamic(10) . . .