Computer Architecture

Parallel ProcessingOla Flygt

Växjö Universityhttp://w3.msi.vxu.se/users/ofl/

Ola.Flygt@msi.vxu.se+46 470 70 86 49

Outline

Basic conceptsTypes and levels of parallelismClassification of parallel

architectureBasic parallel techniquesRelationships between languages

and parallel architecture

CH03

Basic concepts

The concept of programordered set of instructions

(programmer’s view)executable file (operating system’s

view)

The concept of process

OS view, process relates to executionProcess creation

setting up the process descriptionallocating an address spaceloading the program into the allocated address

space, andpassing the process description to the scheduler

process statesready to runrunningwait

Process spawning (independent processes)

A

CB

D E

The concept of thread

smaller chunks of code (lightweight)threads are created within and

belong to processfor parallel thread processing,

scheduling is performed on a per-thread basis

finer-grain, less overhead on switching from thread to thread

Single-thread process or multi-thread (dependent)Thread tree

Process

Threads

Three basic methods for creating and terminating

threads1. unsynchronized creation and

unsynchronized terminationcalling library functions: CREATE_THREAD,

START_THREAD

2. unsynchronized creation and synchronized terminationFORK and JOIN

3. synchronized creation and synchronized terminationCOBEGIN and COEND

Processes and threads in languages

Black box view: T: thread

T2 T0 T1 T1 T2 T0 . . . Tn

FORK

JOIN

FORK

JOIN

COBEGIN

COEND

. . .

The concepts of concurrent execution (N-

client 1-server)

Non pre-emptive Pre-emptive

Time-shared Prioritized

PrioritySever SeverSever

Client ClientClient

Parallel execution

N-client N-server modelSynchronous or Asynchronous

SeverClient

Concurrent and Parallel Programming Languages

Classification of programming languages

Types and levels of parallelism

Available and utilized parallelismavailable: in program or in the problem

solutionsutilized: during execution

Types of available parallelismfunctional

arises from the logic of a problem solution

dataarises from data structures

Available levels Utilized levels

User (program) level User level

Procedure level Process level

Loop level Thread level

Instruction level Instruction level

2

1

1.Exploited by architectures2.Exploited by means of operating systems

Available and utilized levels of functional

parallelism

Utilization of functional parallelism

Available parallelism can be utilized by architecture,

instruction-level parallel architectures compilers

parallel optimizing compileroperating system

multitasking

Concurrent execution models

User level --- Multiprogramming, time sharing

Process level --- MultitaskingThread level --- Multi-threading

Utilization of data parallelism

By using data-parallel architecture

Convert into functional parallelism (i.e. loop constructs)

Classification of parallel architectures

Flynn’s classificationSISDSIMDMISD (Multiple Instruction Single

Date)MIMD

Part III Part II Part IV

Data-parallel architectures Function-parallel architectures

Instruction-level

PAs

Thread-level

PAs

Process-levelPAs

ILPS MIMDs

Vectorarchitecture

Associative

architecturearchitectureand neural

SIMDs Systolic Pipelinedprocessors processorsVLIWs Superscalar Distributed

memory

(multi-computer)

Sharedmemory(multi-MIMD

DPs

Parallel architectures

processors)

Basic parallel technique

Pipelining (time)a number of functional units are employed in

sequence to perform a single computationa number of steps for each computation

Replication (space)a number of functional units perform multiply computation simultaneously

more processorsmore memorymore I/O

more computers

Relation between basic techniques and

architectures

Relationships between languages and parallel

architecture SPMD (Single Procedure Multiple data)

Loop: split into N threads that works on different invocations of the same loop

threads can execute the same code at different speeds synchronize the parallel threads at the end of the loop

barrier synchronization use MIMD

Data-parallel languages DAP Fortran

C = A + B (A, B and C are arrays) use SIMD

Other types like Vector machines or Systolic arrays does not require any specific language support.

Synchronization mechanismsTest_and_set

Semaphore

ConditionalCritical region

Monitor

Rendezvous

Send/receivemessage

Broadcast Shiftnet_receive

(processor form)

Remote procedure calls

Net_send

Using Semaphores to handle mutual execution

P1

Criticalregion

V(S)

S Semaphore

Shareddata

structureV(S)

P(S)

P2

Criticalregion

P(S)

BusyWait

Parallel distributed computing

Ada used rendezvous concepts which combines

feature of RPC and monitorsPVM (Parallel Virtual Machine)

to support workstation clustersMPI (Message-Passing Interface)

programming interface for parallel computers

Summary of forms of

parallelism