Date post: | 03-Jun-2018 |
Category: |
Documents |
Upload: | ing-eduardo-p-hdez |
View: | 221 times |
Download: | 0 times |
of 45
8/12/2019 Collective Comm
1/45
Collective Communication
8/12/2019 Collective Comm
2/45
Collective Communication
Collective communication is defined ascommunication that involves a group ofprocesses
More restrictive than point to point Data sent is same as the data received, i.e.
type, amount All processes involved make one call, no tag to
match operation Processes involved can return only when
operation completes blocking communication only
Standard Mode only
8/12/2019 Collective Comm
3/45
Collective Functions
Barrier synchronization across all group members Broadcast from one member to all members of a group Gather data from all group members to one member Scatter data from one member to all members of a group A variation on Gather where all members of the group receive
the result. (allgather) Scatter/Gather data from all members to all members of a
group (also called complete exchange or all-to-all) (alltoall) Global reduction operations such as sum, max, min, or user-
defined functions, where the result is returned to all groupmembers and a variation where the result is returned to onlyone member
A combined reduction and scatter operation Scan across all members of a group (also called prefix)
8/12/2019 Collective Comm
4/45
Collective Functions
8/12/2019 Collective Comm
5/45
Collective Functions
8/12/2019 Collective Comm
6/45
Collective Functions MPI_BARRIER
blocks the caller until all group membershave called it
returns at any process only after all groupmembers have entered the call
C int MPI_Barrier(MPI_Comm comm ) Input Parameter
comm: communicator (handle)
Fortran MPI_BARRIER(COMM, IERROR) INTEGER COMM, IERROR
8/12/2019 Collective Comm
7/45
Collective Functions MPI_BCAST
broadcasts a message from the process with rank root to allprocesses of the group, itself included
C int MPI_Bcast(void* buffer, int count, MPI_Datatype datatype, int
root, MPI_Comm comm ) Input Parameters
count: number of entries in buffer (integer) datatype: data type of buffer (handle)
root: rank of broadcast root (integer)
comm: communicator (handle)
Input / Output Parameter buffer: starting address of buffer (choice)
Fortran
MPI_BCAST(BUFFER, COUNT, DATATYPE, ROOT, COMM, IERROR) BUFFER(*) INTEGER COUNT, DATATYPE, ROOT, COMM, IERROR
8/12/2019 Collective Comm
8/45
Collective Functions MPI_BCAST
A
AAA A
8/12/2019 Collective Comm
9/45
Collective Functions MPI_GATHER
Each process (root process included) sends the contents of itssend buffer to the root process.
The root process receives the messages and stores them inrank order
C
int MPI_Gather(void* sendbuf, int sendcount, MPI_Datatypesendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype,int root, MPI_Comm comm)
Input Parameters sendbuf: starting address of send buffer (choice)
sendcount: number of elements in send buffer (integer) sendtype: data type of send buffer elements (handle)
recvcount: number of elements for any single receive (integer,significant only at root)
recvtype: data type of recv buffer elements (significant only at root)(handle)
root: rank of receiving process (integer)
comm: communicator (handle)
8/12/2019 Collective Comm
10/45
Collective Functions MPI_GATHER
Output Parameter
recvbuf: address of receive buffer (choice,significant only at root)
Fortran MPI_GATHER(SENDBUF, SENDCOUNT,
SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,ROOT, COMM, IERROR)
SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE,RECVCOUNT, RECVTYPE, ROOT, COMM,IERROR
8/12/2019 Collective Comm
11/45
Collective Functions MPI_GATHER
BA C D
DCA
A B C D
B
8/12/2019 Collective Comm
12/45
Collective Functions MPI_SCATTER
MPI_SCATTER is the inverse operation to MPI_GATHER C
int MPI_Scatter(void* sendbuf, int sendcount,MPI_Datatype sendtype, void* recvbuf, int recvcount,MPI_Datatype recvtype, int root, MPI_Comm comm)
Input Parameters sendbuf: address of send buffer (choice, significant only at
root) sendcount: number of elements sent to each process (integer,
significant only at root) sendtype: data type of send buffer elements (significant only
at root) (handle) recvcount: number of elements in receive buffer (integer) recvtype: data type of receive buffer elements (handle) root: rank of sending process (integer) comm: communicator (handle)
8/12/2019 Collective Comm
13/45
Collective Functions MPI_SCATTER
Output Parameter recvbuf: address of receive buffer (choice)
Fortran MPI_SCATTER(SENDBUF, SENDCOUNT,
SENDTYPE, RECVBUF, RECVCOUNT,RECVTYPE, ROOT, COMM, IERROR)
SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE,RECVCOUNT, RECVTYPE, ROOT, COMM,IERROR
8/12/2019 Collective Comm
14/45
Collective Functions MPI_SCATTER
A B C D
DCA
A B C D
B
8/12/2019 Collective Comm
15/45
Collective Functions
MPI_ALLGATHER
MPI_ALLGATHER can be thought of as MPI_GATHER, butwhere all processes receive the result, instead of just the root.
The jth block of data sent from each process is received byevery process and placed in the jth block of the buffer recvbuf.
C
int MPI_Allgather(void* sendbuf, int sendcount, MPI_Datatypesendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype,MPI_Comm comm)
Input Parameters sendbuf: starting address of send buffer (choice)
sendcount: number of elements in send buffer (integer) sendtype: data type of send buffer elements (handle)
recvcount: number of elements received from any process (integer)
recvtype: data type of receive buffer elements (handle)
comm: communicator (handle)
8/12/2019 Collective Comm
16/45
Collective Functions
MPI_ALLGATHER
Output Parameter
recvbuf: address of receive buffer (choice)
Fortran
MPI_ALLGATHER(SENDBUF, SENDCOUNT,SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE,COMM, IERROR)
SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE,RECVCOUNT, RECVTYPE, COMM, IERROR
8/12/2019 Collective Comm
17/45
Collective Functions
MPI_ALLGATHER
BA C D
DCA
A B C D
B
A B C D A B C DA B C D
MPI_ALLGATHER
8/12/2019 Collective Comm
18/45
Collective Functions MPI_ALLTOALL
Extension of MPI_ALLGATHER to the case where eachprocess sends distinct data to each of the receivers.The jth block sent from process i is received by process
j and is placed in the ith block of recvbuf C
int MPI_Alltoall(void* sendbuf, int sendcount,MPI_Datatype sendtype, void* recvbuf, int recvcount,MPI_Datatype recvtype, MPI_Comm comm)
Input Parameters sendbuf: starting address of send buffer (choice) sendcount: number of elements sent to each process (integer) sendtype: data type of send buffer elements (handle)
recvcount: number of elements received from any process(integer) recvtype: data type of receive buffer elements (handle) comm: communicator (handle)
8/12/2019 Collective Comm
19/45
Collective Functions MPI_ALLTOALL
Output Parameter recvbuf: address of receive buffer (choice)
Fortran MPI_ALLTOALL(SENDBUF, SENDCOUNT,
SENDTYPE, RECVBUF, RECVCOUNT,RECVTYPE, COMM, IERROR)
SENDBUF(*), RECVBUF(*)
INTEGER SENDCOUNT, SENDTYPE,RECVCOUNT, RECVTYPE, COMM,IERROR
8/12/2019 Collective Comm
20/45
Collective Functions MPI_ALLTOALL
E F G HA B C D I J K L M N O P
A B C D
A E I M
E F G H
B F J N
I J K L
C G K O
M N O P
D H L P
Rank 0 Rank 1 Rank 2 Rank 3
MPI_ALLTOALL
8/12/2019 Collective Comm
21/45
Collective Functions MPI_REDUCE
MPI_REDUCE combines the elements provided in the inputbuffer (sendbuf) of each process in the group, using theoperation op, and returns the combined value in the outputbuffer (recvbuf) of the process with rank root
C int MPI_Reduce(void* sendbuf, void* recvbuf, int count,
MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) Input Parameters
sendbuf: address of send buffer (choice)
count: number of elements in send buffer (integer)
datatype: data type of elements of send buffer (handle) op: reduce operation (handle)
root: rank of root process (integer)
comm: communicator (handle)
Output Parameter recvbuf: address of receive buffer (choice, significant only at root)
8/12/2019 Collective Comm
22/45
Collective Functions MPI_REDUCE
Fortran MPI_REDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, ROOT, COMM,
IERROR) SENDBUF(*), RECVBUF(*) INTEGER COUNT, DATATYPE, OP, ROOT, COMM, IERROR
Predefined Reduce Operations [ MPI_MAX] maximum [ MPI_MIN] minimum [ MPI_SUM] sum
[ MPI_PROD] product
[ MPI_LAND] logical and [ MPI_BAND] bit-wise and [ MPI_LOR] logical or
[ MPI_BOR] bit-wise or
[ MPI_LXOR] logical xor [ MPI_BXOR] bit-wise xor [ MPI_MAXLOC] max value and location (return the max and an integer,
which is the rank storing the max value)
[ MPI_MINLOC] min value and location
8/12/2019 Collective Comm
23/45
Collective Functions MPI_REDUCE
E F G HA B C D I J K L M N O P
A B C D E F G H I J K L M N O P
AoEoIoM
Rank 0 Rank 1 Rank 2 Rank 3
In this case, root = 1
if count = 2, therewill be BoFoJoN inthe 2nd element ofthe array
8/12/2019 Collective Comm
24/45
Collective Functions
MPI_ALLREDUCE
Variants of the reduce operations wherethe result is returned to all processes inthe group
The all-reduce operations can be
implemented as a reduce, followed by abroadcast. However, a directimplementation can lead to betterperformance.
C int MPI_Allreduce(void* sendbuf, void* recvbuf,
int count, MPI_Datatype datatype, MPI_Op op,MPI_Comm comm)
8/12/2019 Collective Comm
25/45
Collective Functions
MPI_ALLREDUCE
Input Parameters sendbuf: starting address of send buffer (choice) count: number of elements in send buffer (integer) datatype: data type of elements of send buffer
(handle) op: operation (handle) comm: communicator (handle)
Output Parameter recvbuf: starting address of receive buffer (choice)
Fortran
MPI_ALLREDUCE(SENDBUF, RECVBUF, COUNT,DATATYPE, OP, COMM, IERROR) SENDBUF(*), RECVBUF(*) INTEGER COUNT, DATATYPE, OP, COMM, IERROR
8/12/2019 Collective Comm
26/45
Collective Functions
MPI_ALLREDUCE
E F G HA B C D I J K L M N O P
A B C D E F G H I J K L M N O P
AoEoIoM
Rank 0 Rank 1 Rank 2 Rank 3
8/12/2019 Collective Comm
27/45
Collective Functions
MPI_REDUCE_SCATTER
Variants of each of the reduce operations where the result isscattered to all processes in the group on return.
MPI_REDUCE_SCATTER first does an element-wise reductionon vector of count=irecvcounts[i] elements in the sendbuffer defined by sendbuf, count and datatype.
Next, the resulting vector of results is split into n disjointsegments, where n is the number of members in the group.Segment i contains recvcounts[i] elements.
The ith segment is sent to process i and stored in the receivebuffer defined by recvbuf, recvcounts[i] and datatype.
The MPI_REDUCE_SCATTER routine is functionally equivalentto: A MPI_REDUCE operation function with count equal to thesum of recvcounts[i] followed by MPI_SCATTERV with
sendcounts equal to recvcounts. However, a directimplementation may run faster.
8/12/2019 Collective Comm
28/45
Collective Functions
MPI_REDUCE_SCATTER
C int MPI_Reduce_scatter(void* sendbuf, void* recvbuf, int
*recvcounts, MPI_Datatype datatype, MPI_Op op, MPI_Commcomm)
Input Parameters sendbuf: starting address of send buffer (choice) recvcounts: integer array specifying the number of elements in result
distributed to each process. Array must be identical on all calling processes. datatype: data type of elements of input buffer (handle) op: operation (handle) comm: communicator (handle)
Output Parameter recvbuf: starting address of receive buffer (choice)
Fortran MPI_REDUCE_SCATTER(SENDBUF, RECVBUF, RECVCOUNTS,
DATATYPE, OP, COMM, IERROR) SENDBUF(*), RECVBUF(*) INTEGER RECVCOUNTS(*), DATATYPE, OP, COMM, IERROR
8/12/2019 Collective Comm
29/45
Collective Functions
MPI_REDUCE_SCATTER
A B C DRank 0recvcounts = 1
E F G H
I J K L
M N O P
Rank 1
recvcounts = 2
Rank 2recvcounts = 0
Rank 3recvcounts = 1
AoEoIoM
A B C D
E F G H
I J K L
M N O P
BoFoJoN
CoGoKoO
DoHoLoP
8/12/2019 Collective Comm
30/45
Collective Functions MPI_SCAN
MPI_SCAN is used to perform a prefix reductionon data distributed across the group. Theoperation returns, in the receive buffer of theprocess with rank i, the reduction of the values in
the send buffers of processes with ranks 0,...,i(inclusive). The type of operations supported,their semantics, and the constraints on send andreceive buffers are as for MPI_REDUCE.
C
int MPI_Scan(void* sendbuf, void* recvbuf, intcount, MPI_Datatype datatype, MPI_Op op,MPI_Comm comm )
8/12/2019 Collective Comm
31/45
Collective Functions MPI_SCAN
Input Parameters sendbuf: starting address of send buffer (choice) count: number of elements in input buffer (integer) datatype: data type of elements of input buffer
(handle) op: operation (handle) comm: communicator (handle)
Output Parameter recvbuf: starting address of receive buffer (choice)
Fortran MPI_SCAN(SENDBUF, RECVBUF, COUNT, DATATYPE,
OP, COMM, IERROR) SENDBUF(*), RECVBUF(*) INTEGER COUNT, DATATYPE, OP, COMM, IERROR
8/12/2019 Collective Comm
32/45
Collective Functions MPI_SCAN
A B C DRank 0
E F G H
I J K L
M N O P
Rank 1
Rank 2
Rank 3
AoEoIoM
A B C D
E F G H
I J K L
M N O P
AoEoI
AoE
A
8/12/2019 Collective Comm
33/45
Example MPI_BCAST
To demonstrate how to useMPI_BCAST to distribute an array toother process
8/12/2019 Collective Comm
34/45
Example MPI_BCAST (C)
/* // root broadcast the array to all processes */
#include #include
#define SIZE 10
main( int argc, char** argv)
{ int my_rank; // the rank of each proc int array[SIZE]; int root = 0; // the rank of root int i; MPI_Comm comm = MPI_COMM_WORLD;
MPI_Init(&argc, &argv); MPI_Comm_rank(comm, &my_rank);
if (my_rank == 0) { for (i = 0; i < SIZE; i ++) { array[i] = i; } }
8/12/2019 Collective Comm
35/45
Example MPI_BCAST (C)
else { for (i = 0; i < SIZE; i ++) {
array[i] = 0; } }
printf("Proc %d: (Before Broadcast) ", my_rank); for (i = 0; i < SIZE; i ++)
{ printf("%d ", array[i]); } printf("\n");
MPI_Bcast(array, SIZE, MPI_INT, root, comm);
printf("Proc %d: (After Broadcast) ", my_rank); for (i = 0; i < SIZE; i ++) { printf("%d ", array[i]); } printf("\n");
MPI_Finalize(); return 0; }
8/12/2019 Collective Comm
36/45
Example MPI_BCAST (Fortran)
C /* C * root broadcast the array to all processes C */
PROGRAM main INCLUDE 'mpif.h'
PARAMETER (SIZE = 10) INTEGER my_rank, ierr, root, i INTEGER array(SIZE) INTEGER comm INTEGER arraysize
root = 0 comm = MPI_COMM_WORLD arraysize = SIZE
8/12/2019 Collective Comm
37/45
Example MPI_BCAST (Fortran)
CALL MPI_INIT(ierr) CALL MPI_COMM_RANK(comm, my_rank, ierr)
IF (my_rank.EQ.0) THEN DO i = 1, SIZE array(i) = i
END DO ELSE DO i = 1, SIZE array(i) = 0 END DO END IF
WRITE(6, *) "Proc ", my_rank, ": (Before Broadcast)", (array(i), i=1, SIZE) CALL MPI_Bcast(array, arraysize, MPI_INTEGER, root, comm, ierr)
WRITE(6, *) "Proc ", my_rank, ": (After Broadcast)", (array(i), i=1, SIZE)
call MPI_FINALIZE(ierr) end
8/12/2019 Collective Comm
38/45
Case Study 1 MPI_SCATTER and
MPI_REDUCE
Master distributes (scatters) anarray across processes. Processesadd their elements, then combine
sum in master through a reductionoperation.
Step 1 Proc 0 initializes a 16 integers array
Proc 0: {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,12, 13, 14, 15, 16}
8/12/2019 Collective Comm
39/45
Case Study 1 MPI_SCATTER and
MPI_REDUCE
Step 2
Scatter array among all processes
Proc 0: {1, 2, 3, 4}
Proc 1: {5, 6, 7, 8}
Proc 2: {9, 10, 11, 12}
Proc 3: {13, 14, 15, 16}
Step 3 Each process does some calculations
8/12/2019 Collective Comm
40/45
Case Study 1 MPI_SCATTER and
MPI_REDUCE
Step 4 Reduce to Proc 0 Proc 0: Total Sum
C mpi_scatter_reduce01.c
Compilation: mpicc mpi_scatter_reduce01.c o mpi_scatter_reduce01
Run: mpirun np 4 mpi_scatter_reduce01
Fortran mpi_scatter_reduce01.f
Compilation: mpif77 mpi_scatter_reduce01.f o mpi_scatter_reduce01
Run: mpirun np 4 mpi_scatter_reduce01
8/12/2019 Collective Comm
41/45
Case Study 2 MPI_GATHER
Matrix Multiplication
760
686
612
538
20161912188174
20151911187173
20141910186172
2013199185171
20
19
18
17
161284
151173
141062
13951
Algorithm:
{4x4 matrix A} x {4x1 vector x} = product
Each process stores a row of A and a single entry ofx
Use 4 gather operations to place a full copy of x ineach process, then perform multiplications
8/12/2019 Collective Comm
42/45
Case Study 2 MPI_GATHER
Matrix Multiplication
Step 1: Initialization Proc 0: {1 5 9 13}, {17} Proc 1: {2 6 10 14}, {18} Proc 2: {3 7 11 15}, {19} Proc 3: {4 8 12 16}, {20}
Step 2: Perform 4 times MPI_GATHER to gather the column
matrix to each process Proc0: {1 5 9 13}, {17 18 19 20}
Proc1: {2 6 10 14}, {17 18 19 20} Proc2: {3 7 11 15}, {17 18 19 20} Proc3: {4 8 12 16}, {17 18 19 20}
8/12/2019 Collective Comm
43/45
Case Study 2 MPI_GATHER
Matrix Multiplication
Step 3:
Perform multiplication
Proc 0: 1x17+5x18+9x19+13x20=538
Proc 1: 2x17+6x18+10x19+14x20=612 Proc 2: 3x17+7x18+11x19+15x20=686
Proc 3: 4x17+8x18+12x19+16x20=760
Step 4:
Gather all processinner product into master
process and display the result
8/12/2019 Collective Comm
44/45
Case Study 2 MPI_GATHER
Matrix Multiplication
C mpi_gather01.c Compilation:
mpicc mpi_gather01.c o mpi_gather01
Run: mpirun np 4 mpi_gather01
Fortran mpi_gather01.f Compilation:
mpif77 mpi_gather01.f o mpi_gather01 Run:
mpirun np 4 mpi_gather01
8/12/2019 Collective Comm
45/45
END