1

Mohsan JameelDepartment of Computing

NUST School of Electrical Engineering and Computer Science

2

OutlineI. Introduction to OpenMP

II. OpenMP Programming Model

III. OpenMP Directives

IV. OpenMP Clauses

V. Run-Time Library Routine

VI. Environment Variables

VII. Summary

3

What is OpenMPApplication program interface (API) that is used

to explicitly direct multi-threaded, shared memory parallelism

Consists of:Compiler directives Run time routinesEnvironment variables

• Specification maintained by the OpenMP, Architecture Review Board (http://www.openmp.org)

• Version 3.0 has been released May 2008

4

What OpenMP is NotNot Automatic parallelization

User explicitly specifies parallel execution Compiler does not ignore user directives even if

wrong

Not just loop level parallelism Functionality to enable coarse grained parallelism

Not meant for distributed memory parallel systems

Not necessarily implemented identically by all vendors

Not Guaranteed to make the most efficient use of shared memory

5

History of OpenMP In the early 90's, vendors of shared-memory machines supplied

similar, directive-based, Fortran programming extensions: The user would augment a serial Fortran program with directives

specifying which loops were to be parallelized. First attempt at a standard was the draft for ANSI X3H5 in 1994. It

was never adopted, largely due to waning interest as distributed memory machines became popular.

The OpenMP standard specification started in the spring of 1997, taking over where ANSI X3H5had left off, as newer shared memory machine architectures started to become prevalent

6

Goal of OpenMPStandardization :

Provide a standard among a variety of shared memory architectures/platforms

Lean and mean : Establish a simple and limited set of directives for

programming shared memory machines.

Ease of Use : Provide capability to incrementally parallelize a

serial program Provide the capability to implement both coarse-

grain and fine-grain parallelism

Portability : Support Fortran (77, 90, and 95), C, and C++

7

OutlineI. Introduction to OpenMP

II. OpenMP Programming Model

III. OpenMP Directives

IV. OpenMP Clauses

V. Run-Time Library Routine

VI. Environment Variables

VII. Summary

8

OpenMP Programming ModelThread Based Parallelism

Explicit Parallelism

Compiler Directive Based

Dynamic Threads

Nested Parallelism Support

Task parallelism support (OpenMP specification 3.0)

9

Shared Memory Model

10

Execution Model

ID=0

ID=1,2,3…N-1

11

TerminologyOpenMP Team=: Master + workers

A parallel region is block of code executed by all threads simultaneously. Master thread always has thread ID=0 Thread adjustment is done before entering parallel

region. An “if” clause can be used with parallel construct,

incase the condition evaluate to FALSE, parallel region is avoided and code run serially

Work-sharing construct is responsible for dividing work among the threads in parallel region

12

Example OpenMP Code Structure

13

Components of OpenMP

14

I. Introduction to OpenMP

II. OpenMP Programming Model

III. OpenMP Directives

IV. OpenMP Clauses

V. Run-Time Library Routine

VI. Environment Variables

VII. Summary

15

Go to helloworld.c

16

C/C++ Parallel Region Example

!$OMP PARALLEL write (*,*) “Hello”

!$OMP END PARALLEL

Hello world from thread = 0Number of threads = 3

Hello world from thread = 1 Hello world from thread = 2

thread 0 thread 1 thread 2

17

OpenMP Directives

18

OpenMP ScopingStatic Extent:

The code textually enclosed between beginning and end of structure block

The static extent does not span other routines

Orphaned Directive:An OpenMP directive appear independently

Dynamic Extent: It include extent of both static extent and

orphaned directives

19

OpenMP Parallel RegionsA block of code that will be executed by multiple

threads

Properties

- Fork-Join Model

- Number of threads won’t change inside a parallel region

- SPMD execution within region

- Enclosed block of code must be structured, no branching into or out of block

Format

#pragma omp parallel clause1 clause2 …

20

OpenMP ThreadsHow many threads?

Use of the omp_set_threads() library functionSetting of the OMP_NUM_THREADS environment

variable Implementation default

Dynamic Threads :By default, the same number of threads are used to

execute each parallel regionTwo methods for enabling dynamic threads

1 Use of the omp_set_dynamic() library function2 Setting of the OMP_DYNAMIC environment variable

21

OpenMP Work-sharing constructs

Data parallelism Functional parallelism Serialize a section

22

Example: Count3s in an array

Lets assume we have an array of N integers.

We want to find how many 3s are in the array.

We needa for loop if statement, anda count variable

Lets look at its serial and parallel version

23

Serial: Count3s in an arrayint count, n=100;int array[n]; // initialize array

for(i=0;i<length;i++){

if (array[i]==3)count++;

}

24

Work-sharing construct: “for loop”

“for loop” work-sharing construct is thought of as data parallelism construct.

25

Parallelize 1st attempt: Count3s in an array

int count, n=100;int array[n]; // initialize array

#pragma omp parallel for default(none) shared(n,array,count) private(i)

for(i=0;i<length;i++){

if (array[i]==3)count++;

}

26

Work-sharing construct:Example of “for loop”

#pragma omp parallel for default(none) shared(n,a,b,c) private(i)

for (i=0;i<n;i++)

{

c[i] = a[i] + b[i];

}

27

Work-sharing construct: “section”

“Section” work-sharing construct is thought of as functional parallelism construct.

28

Parallelize 2nd attempt: Count3s in an array

• Say we also want to count 4s in same array.• Now we have two different function i.e. count 3 and count 4.

int count, n=100;int array[n]; // initialize array#pragma omp parallel sections default(none) shared(n,array,count3,count4)

private(i)

#pragma omp parallel sectionfor(i=0;i<length;i++){

if (array[i]==3)count3++;

}#pragma omp parallel sectionfor(i=0;i<length;i++){

if (array[i]==4)count4++;

}

No date race condition in this example. WHY?

29

#pragma omp parallel sections default(none) shared(a,b,c,d,e,n) private(i)

{#pragma omp section{

printf("Thread %d executes 1st loop \n”,omp_get_thread_num());

for(i=0;i<n;i++)a[i]=3*b[i];

}#pragma omp section{

printf("Thread %d executes 1st loop \n”,omp_get_thread_num());

for(i=0;i<n;i++) e[i]=2*c[i]+d[i];

}}final_sum=sum(a,n) + sum(e,n);printf("FINAL_SUM is %d\n",final_sum)

Work-sharing construct:Example 1 of “section”

30

Work-sharing construct:Example 2 of “section” 1/2

31

Work-sharing construct:Example 2 of “section” 2/2

32

Work-sharing construct:Example of “single”

In parallel region “single block” is used to specify that this block is executed only by one thread in the team of threads.

Lets look at an example

33

I. Introduction to OpenMP

II. OpenMP Programming Model

III. OpenMP Directives

IV. OpenMP Clauses

V. Run-Time Library Routine

VI. Environment Variables

VII. Summary

34

OpenMP Clauses: Data sharing 1/2

shared(list)shared clause is used to specify which data is

shared among thread.All threads can read and write to this shared

variable.By default all variables are shared.

private(list)private variable are local to thread.Typical example of private variable is loop counter,

since each thread has its own loop counter initialized at entry point.

35

A private variable is defined between entry and exit point of parallel region.

A private variable within parallel region has no scope out side of it

firstprivate and lastprivate clauses are used to increase scope of variable beyond parallel region.

firstprivate: All variables in the list are initialized with the original value that object had before entering parallel region

lastprivate: The thread that executes the last iteration or section updates the value of object in list.

OpenMP Clauses: Data sharing 2/2

36

Example: firstprivate and lastprivate

int main(){int C, B , A= 10;

/*--- Start of parallel region ---*/#pragma omp parallel for default(none) firstprivate(A)

lastprivate(B) private(i) for (i=0;i<n;i++){

…B = i + A; …

}/*--- End of parallel region ---*/

C=B;}

37

OpenMP Clauses: nowait

nowait clause is used to avoid implicit synchronization at end of work-sharing directive

38

OpenMP Clause: schedule schedule clause is supported in loop construct only.

Used to control the manner in which loop iterations are distributed over the threads.

Syntax: schedule(kind[,chunk_size)

Types: static[,chunk]: distribute iterations in blocks of size “chunk

over the threads in a round-robin fashion dynamic[,chunk]: fixed portions of work; size is controlled by

the value chunk, when thread finishes its portion it starts with next portion.

guided[,chunk]: same as “dynamic”, but size of the portion of work decreases exponentially.

runtime[,chunk]: iteration scheduling scheme is set at runtime thought environment variable OMP_SCHEDULE

39

The Experiment with schedule clause

40

OpenMP Critical construct

int main(){int sum, n=5;

int a[5]={1,2,3,4,5};/*--- Start of parallel region ---*/#pragma omp parallel for default(none) shared(sum,a,n) private(i) for (i=0;i<n;i++){ sum += a[i]; }/*--- End of parallel region ---*/printf(“sum of vector a =%d”,sum);}

Example summation of a vector

race condition

41

OpenMP Critical constructint main(){

int sum, local_sum, n=5; int a[5]={1,2,3,4,5};/*--- Start of parallel region ---*/#pragma omp parallel default(none) shared(sum,a,n) private(local_sum,i) {

#pragma omp forfor (i=0;i<n;i++){

local_sum += a[i]; }

#pragma omp critical {

sum+=local_sum}

}/*--- End of parallel region ---*/printf(“sum of vector a =%d”,sum);}

42

Parallelize 3rd attempt: Count3s in an array

int count, n=100;int array[n]; // initialize array

#pragma omp parallel default(none) shared(n,array,count) private(i,local_count)

{#pragma omp parallel for for(i=0;i<length;i++){

if (array[i]==3)local_count ++;

}#pragma omp critical

{ count+=local_count}

} /*--- End of Parallel region ---*/

43

OpenMP Clause: reduction

int main(){

int sum, n=5;

int a[5]={1,2,3,4,5};

/*--- Start of parallel region ---*/

#pragma omp parallel for default(none) shared(a,n) private(i)\

reduction(+:sum)

for (i=0;i<n;i++)

{

sum += a[i];

}

/*--- End of parallel region ---*/

printf(“sum of vector a =%d”,sum);

}

• OpenMP provides a reduction clause which is used with for loop and section directives.

• reduction variable must be shared among threads

• race condition is avoided implicitly.

44

Parallelize 4th attempt: Count3s in an array

int count, n=100;int array[n]; // initialize array

#pragma omp parallel for default(none) shared(n,array) private(i) \

for(i=0;i<length;i++){

if (array[i]==3)count++;

} /*--- End of Parallel region ---*/

reduction(+:count)

45

Tasking in OpenMP

Tasking in OpenMPIn OpenMP 3.0 the concept of tasks has been

added to the OpenMP execution model

The Task model is useful is case where the number of parallel pieces and the work involved in each piece varies and/or unknown

Before inclusion of the Task model OpenMP was not suited for unstructured problem

Tasks are often set up within a single construct in a manager-worker model.

46

Task Parallelism Approach 1/2 Threads line up as workers, go through the queue of work

to be done, and do a task

Threads do not wait, as in loop parallelism, rather go back to queue and do more tasks.

Each task is executed serially by work thread that encounter that task in queue.

Load balancing occur as short and long task are done as threads become available.

47

48

Task Parallelism Approach 2/2

49

Example: Task parallelism

50

Best Practices Optimize barrier use

Avoid ordered construct

Avoid large critical regions

Maximize parallel regions

Avoid multiple use of parallel regions

Address poor load balance

51

I. Introduction to OpenMP

II. OpenMP Programming Model

III. OpenMP Directives

IV. OpenMP Clauses

V. Run-Time Library Routine

VI. Environment Variables

VII. Summary

52

List of runtime library routineRuntime library routine are provided in omp.h

header file void omp_set_num_threads(int num); int omp_get_num_threads(); int omp_get_max_threads(); int omp_get_thread_num(); int omp_get_thread_limit(); int omp_get_num_procs(); double omp_get_wtime(); int omp_in_parallel(); // return 0 false and non-zero true Few more

53

More list of runtime library routineThese routine are new with OpenMP 3.0

54

I. Introduction to OpenMP

II. OpenMP Programming Model

III. OpenMP Directives

IV. OpenMP Clauses

V. Run-Time Library Routine

VI. Environment Variables

VII. Summary

55

Environment VariableOMP_NUM_THREAD

OMP_DYANMIC

OMP_THREAD_LIMIT

OMP_STACKSIZE

56

I. Introduction to OpenMP

II. OpenMP Programming Model

III. OpenMP Directives

IV. OpenMP Clauses

V. Run-Time Library Routine

VI. Environment Variables

VII. Summary

57

SummaryOpenMP provides small but yet powerful programming

model

Compilers with OpenMP support are widely available

OpenMP is a directive based shared memory programming model

OpenMP API is a general purpose parallel programming API with emphasis on the ability to parallelize existing programs

Scalable parallel programs can be written by using parallel regions

Work-sharing constructs enable efficient parallelization of computationally intensive portions of program

58

Thank Youand

Exercise Session