04/10/25Parallel and Distributed Programming1 Shared-memory Parallel Programming Taura Lab M1 Yuuki...

04/10/25 Parallel and Distributed Programming 1

Shared-memory Parallel Programming

Taura Lab M1Yuuki Horita


Agenda Introduction Sample Sequential Program Multi-thread programming OpenMP Summary




Parallel Programming Model

Message Passing Model Talked by Imatake-kun just now

Shared Memory Model Memory is shared with all process

elements Multiprocessor (SMP, SunFire, …) DSM (Distributed Shared Memory)

Process elements can communicate each other through the shared memory


Shared Memory Model

PE PE PE……

Memory


Shared Memory Model

Simplicity not necessary to think about the location

of the computation data Fast communication (Multiprocessor)

not necessary to use networks in process communication

Dynamic load sharing the same reason as simplicity


Shared Memory Parallel Programming Multi-thread programming

Pthreads OpenMP

Parallel Programming model for shared memory multiprocessor




Sample Sequential Program

…loop{

for (i=0; i<N; i++){ for (j=0; j<N; j++){ a[i][j] = 0.2 * (a[i][j-1] + a[i][j+1] + a[i-1][j] + a[i+1][j] + a[i][j]); } }

}…

FDM (Finite Difference Method)


Parallelization Procedure

Sequential Computation

Decomposition

Tasks

Assignment

Process Elements

Orchestration

Mapping

Processors


Parallelize the Sequential Program

Decomposition

a task

…loop{

for (i=0; i<N; i++){ for (j=0; j<N; j++){

a[i][j] = 0.2 * (a[i][j-1] + a[i][j+1] + a[i-1][j] + a[i+1][j] + a[i][j]); } }

}…



Assignment

PE

PE

PE

PE

Divide the tasks equally among process elements



Orchestration

PE

PE

PE

PE

need to communicate and to synchronize



Mapping

PE

PE

PE

PE

Multiprocessor




Multi-thread Programming

A process element is a thread cf. a process

Memory is shared among all threads generated from the same process

Threads can communicate with each other through shared memory


Fork-Join Model

Fork

Join

Parallelized Section

Serialized Section

Serialized SectionProgram starts (Main Thread)

Main Thread creates new threads

Other threads join Main Thread

Main Thread continues processing

Main Thread


Libraries for Thread Programming

Pthreads (C/C++) pthread_create() pthread_join()

Java Thread Thread Class / Runnable Interface


Pthreads API (fork/join) pthread_t // thread variable pthread_create (

pthread_t *thread, // thread variable pthread_attr_t *attr, // thread attributes void *(*func)(void *), // start function void *arg // arguments of the function )

pthread_join ( pthread_t thread, // thread variable void **thread_return // the return value of the thread)


Pthreads Parallel Programming#include …

void do_sequentially (void){ /* sequential execution */}

main (){ … do_sequentially(); // want to parallelize …}


Pthreads Parallel Programming#include …#include <pthread.h>

void do_in_parallel (void){ /* parallel execution */}

main (){ pthread_t tid; … pthread_create(&tid, NULL, (void *)do_in_parallel, NULL); do_in_parallel(); pthread_join(tid); …}


Exclusive Access Control

int sum = 0;

thread_A(){ sum++;}

thread_B(){ sum++;}

ThreadA ThreadB

a ← read sum

write a → sum

a = a + 1

a ← read sum

write a → sum

a = a + 1

0

0

1

1

sum = 0

sum = 1sum = 1


Pthreads API (Exclusive Access Control)

Variable pthread_mutex_t

Initialization Function pthread_mutex_init( pthread_mutex_t *mutex, pthread_mutexattr_t *mutexattr )

Lock Function pthread_mutex_lock(pthread_mutex_t *mutex) pthread_mutex_unlock(pthread_mutex_t *mutex)


Exclusive Access Control

int sum = 0;pthread_mutex_t mutex;pthread_mutex_init(&mutex, 0)

thread_A(){ pthread_mutex_lock(&mutex); sum++; pthread_mutex_unlock(&mutex);}

thread_B(){ pthread_mutex_lock(&mutex); sum++; pthread_mutex_unlock(&mutex);}

acquire lock

sum ++

release lock

sum ++

acquire lock

acquire lock

release lock

ThreadA ThreadB


Pthreads API (Condition Variable) Variable

pthread_cond_t Initialization Function

pthread_cond_init( pthread_cond_t *cond, pthread_condattr_t *condattr )

Condition Function pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex) pthread_cond_broadcast(pthread_cond_t *cond) pthread_cond_signal(pthread_cond_t *cond);


Condition Wait

acquire lock

release lock

pthread_mutex_lock(&mutex)while( condition is not satisfied ){ pthread_cond_wait(&cond, &mutex);}pthread_mutex_unlock(&mutex);

Is condition satisfied?

release lock

sleep

pthread_cond_broadcastpthread_cond_signal

pthread_mutex_lock(&mutex)update_condition();pthread_cond_broadcast(&cond);pthread_mutex_unlock(&mutex);

ThreadA

ThreadB


Synchronization Synchronization in the sample program

n = 0;…pthread_mutex_lock(&mutex);n++;while ( n < nthreads ){ pthread_cond_wait(&cond, &mutex);}pthread_cond_broadcast(&cond);pthread_mutex_unlock(&mutex);


Characteristics of Pthreads

troublesome to describe exclusive access control and synchronization

likely to be deadlocked still hard to parallelize a given

sequential program




What’s OpenMP? specification for a set of compiler

directives, library routines, and environment variables that can be used to specify shared memory parallelism in Fortran and C/C++ programs Fortran ver1.0 API – Oct.1997 C/C++ ver1.0 API – Oct. 1998


Background of OpenMP spread of shared memory multiprocessors need for common directives in shared

memory multiprocessors Each vendors had provided a different set of

directives need for simpler and more flexible interface

for developing parallel applications Pthread is hard for developers to describe

parallel applications


OpenMP API

Directives Libraries Environment Variables


Directives

C/C++

Fortran

#pragma omp directive_name …

!$OMP directive_name …

If user’s compiler doesn’t support openMP, the directive sentences are ignored and therefore the program can be executed as a sequential program.


Parallel Region the part parallelized by some threads

#pragma omp parallel{ /* parallel region */}

create some threads at the beginning of the parallel region

join at the end of the parallel region


Parallel Region (thread)

the number of thread omp_get_num_threads() :

get current # of threads omp_set_num_threads(int nthreads) :

set # of threads to nthreads $OMP_NUM_THREADS

thread ID (0 ～ # of threads-1)

omp_get_thread_num() : get thread ID


Work Sharing Construction

specify the task assignment inside parallel region for

sharing iterations among threads sections

sharing sections among threads single

executing only by one thread


Example of Work Sharing

for (i=0; i<N; i++){ for (j=0; j<N; j++){ a[i][j] = 0.2 * (a[i][j-1] + a[i][j+1] + a[i-1][j] + a[i+1][j] + a[i][j]); }}

omp_set_num_threads(4);

#pragma omp parallel#pragma omp forfor (i=0; i<N; i++){ for (j=0; j<N; j++){ a[i][j] = 0.2 * (a[i][j-1] + a[i][j+1] + a[i-1][j] + a[i+1][j] + a[i][j]); }}


#pragma omp parallel forfor (i=0; i<N; i++){ for (j=0; j<N; j++){ a[i][j] = 0.2 * (a[i][j-1] + a[i][j+1] + a[i-1][j] + a[i+1][j] + a[i][j]); }}

Memory access conflict at i and j makes the computation slow


Data Scoping Attributes specify the data scoping at parallel

construction or work sharing construction shared( var_list )

var_list is shared among threads private( var_list )

var_list is private reduction (operator : var_list )

ex) #pragma omp for reduction (+: sum) var_list is private in construction and reflected

after the construction


Example of Data Scoping Attributes


#pragma omp parallel for private(i, j)for (i=0; i<N; i++){ for (j=0; j<N; j++){ a[i][j] = 0.2 * (a[i][j-1] + a[i][j+1] + a[i-1][j] + a[i+1][j] + a[i][j]); }}


Synchronization barrier

wait until all threads reach this line #pragma omp barrier

critical execute exclusively #pragma omp critical [(name)] { … }

atomic update a scalar variable atomically #pragma omp atomic

……


Synchronization (Pthreads/OpenMP)

Synchronization in the sample program

<Pthreads>

pthread_mutex_lock(&mutex);n++;while ( n < nthreads ){ pthread_cond_wait(&cond, &mutex);}pthread_cond_broadcast(&cond);pthread_mutex_unlock(&mutex);

<OpenMP>

#pragma omp barrier


Summary of OpenMP

Incremental parallelization of sequential programs

Portability Easier to implement parallel

application than Pthreads and MPI




Message Passing Model / Shared Memory Model

Message Passing Shared Memory

Architecture any SMP or DSM

Programming difficult easier

Performance good better (SMP)worse (DSM)

Cost less expensive very expensiveSunFire15K $4,140,830


Thank you!

Date post:	13-Jan-2016
Category:	Documents
Upload:	ethan-weaver
View:	219 times
Download:	0 times

04/10/25Parallel and Distributed Programming1 Shared-memory Parallel Programming Taura Lab M1 Yuuki...

Documents