Dynamic Load Balancing for VORPAL

Viktor Przebinda

Center for Integrated Plasma Studies

What is VORPAL?

• A parallel simulation package that models laser - plasma interactions using the PIC or fluid model.

• Built on an object oriented framework.

• Over 1,300 pages (68,000 lines) of C++ code.

Goal: Implement Dynamic Load Balancing for VORPAL

Dynamic: Automatically adjusts decomposition appropriately at runtime.

Fully compatible with all simulations.Efficient: Minimizes overhead.User friendly: Requires no special

configuration.

Grids define how regions are to be updated.

extended region

physical region

One or two cells wide

Extended regions simplify parallel synchronization.

extended region

physical region

extended region

physical region

CPU 1CPU 0

Extended regions contain data in physical regions of other CPUs.

Allocated region provides simple over allocation scheme.

extended region

physical region

Allocated Region

Working with current framework introduces restrictions.

• Boundaries can be adjusted by at most one cell in each direction.

• Decomposition description is restricted to prevent irregular domains.

Direction 0

Direction 1

Direction 0

Direction 1

ImpossibleNormal

Two strategies exist to accommodate decomposition adjustment.

• Over allocate all fields to accommodate future decomposition adjustment.

• Resize and copy all field data as needed for each decomposition adjustment.

Over Allocation

Region in use overflow

Disadvantage:Decreases cache hits, resulting in lower efficiency

Advantage:Minimal overhead.

overflow

Over allocation introduces minimal overhead.

Region in use overflow

Over Allocation in direction of maximum stride is optimal.

Region in use

overflow

Resizing

New memory block requested,values are copied over

Disadvantage:Large overhead

Advantage:Does not affect efficiency.

Optimal performance is achieved through both methods

New memory block requested,values are copied over

Memory is over allocated to prevent future resizes.

Decomposition adjustment occurs at end of update

Update Load Balancing

CPU time measured each iteration

DLB is achieved in eight steps.

1. All nodes send processing times to node zero.

2. Node zero decides whether to perform an adjustment.

3. Node zero constructs an adjustment description and sends to all other nodes.

4. All nodes apply the adjustment, reconfigure grid.

5. All fields resize as necessary.

6. Field iteraters are rebuilt

7. All messengers are rebuilt

8. Fields and particles are synchronized.

1. Processing times are collected to aid in adjustment decisions.

• Each node measures the virtual time it took to perform its last update. Virtual time excludes time spent blocked on I/O.

• This amount is sent to node zero.

• The process waits for reply from node zero.

Time

2. Adjustment decision made based on idle time.

• Given the cost of performing a load balance, VORPAL only does so only if any node during the last time step was idle for more than 10% of the time.

Time

3. Adjustment is computed to eliminate bottlenecks

• Using a greedy algorithm, node zero constructs adjustment information for all processors.

• This is sent to all nodes.

Time

Boundaries are shrunk around processors with high load.

CPU1

CPU2 CPU3

CPU Loadlowest

highest

Direction 0

Direction 1CPU0

4. Decomposition object is modified on each processor

• Each node applies the adjustment.

• The local grid is adjusted to match the new size.

• The allocated region is modified if it cannot support the new size.

Time

5. Fields resize if allocated region has changed.

• All fields check the allocated region to see if it has grown.

• If so, the field allocates additional memory to accommodate 25 more cells in the direction of growth.

Time

6. Outdated information in Field iteraters is rebuilt.

• Pointers to specific memory locations may have changed if a resize was performed.

• Physical and extended regions may have changed size.

Time

7. Outdated message passing objects are rebuilt.

• Intersecting regions may have changed so the must be reconstructed.

Time

8. New boundaries must be synchronized with neighbors.

• Field data on physical boundaries is sent to neighboring processors and extended regions are filled from neighbors.

• Particles that may have crossed outside the boundary of the current node are sent to neighboring nodes.

• Unfortunately, since there is nothing to do while synchronization takes place, an enormous overhead is seen at this step.

Time

When to use load balancing.

• When running a PIC simulations.

• When plasma concentration is expected to change.

• When decomposition is along the zero direction.

• When a large number of time steps are used.

Case study: DLB can beat best static decomposition by 23%.

CPU0 CPU1

Particles loaded into right region

Boundary at midpoint

Sliding window moves particles left to CPU0

CPU0 CPU1

Particles shifted into left region

Standard run shows large differences in CPU use.Update time per processor without load balancing

0

100

200

300

400

500

600

1 55 109 163 217 271 325 379 433 487 541 595 649 703 757 811 865 919 973

Time (Msec)

Time Step

CPU0

CPU1

Load Balancing ensures uptimes on each node are equal.

Update time per processor with load balancing

0

100

200

300

400

500

600

1 55 109 163 217 271 325 379 433 487 541 595 649 703 757 811 865 919 973

Time step

Time (Msec)

CPU0

CPU1

Conclusion

• Load balancing performs desired functions.

• Overhead involved in message passing is quite significant, somewhat limiting usefulness.