Lecture 1TTH 03:30AM-04:45PM

Dr. Jianjun Huhttp://mleg.cse.sc.edu/edu/csc

e569/

CSCE569 Parallel Computing

University of South CarolinaDepartment of Computer Science and Engineering

CSCE569 Course InformationMeet time: TTH 03:30AM-04:45PM Swearingen

2A21

4 HomeworkUse CSE turn-in system to submit your Homework

(https://dropbox.cse.sc.edu)Deadline policy

1 Midterm Exam (conceptual understanding)1 Final Project (deliverable to your future

employer!)TeamworkImplementation project/research project

TA: No TA.

CSCE569 Course InformationTextbook and references

Parallel Programming: for Multicore and Cluster Systems By:Thomas Rauber (Author), Gudula Rünger (Author) Publisher: Springer; 1st Edition. edition (March 10, 2010)

Good reference book: Parallel Programming in C with MPI and OpenMP by Michael J. Quinn

Most important information sources: Slides.Grading policy

4 homeworks, 1 midterm, 1 final project, in-class participation

About Your InstructorDr. Jianjun Hu (jianjunh@cse.sc.edu)

Office hours: TTH 2:30-3:20PM or Drop by any time

Office Phone#: 803-7777304 3A66 SWNGBackground:

Mechanical Engineering/CADMachine learning/Computational

intelligence/Genetic Algorithms/Genetic Programming (PhD)

Bioinformatics and Genomics (Postdoc)Multi-disciplinary just as parallel computing

app.

OutlineMotivationModern scientific methodEvolution of supercomputingModern parallel computersSeeking concurrencyData clustering case studyProgramming parallel computers

Why You are Here?

Solve BIG problems

Use Supercomputers

Write parallel programs

Why Faster Computers?Solve compute-intensive problems

fasterMake infeasible problems feasibleReduce design time

Solve larger problems in same amount of timeImprove answer’s precisionReduce design time

Gain competitive advantage

Why Parallel Computing?The massively parallel architecture of

GPUs, coming from its graphics heritage, is now delivering transformative results for scientists and researchers all over the world. For some of the world’s most challenging problems in medical research, drug discovery, weather modeling, and seismic exploration – computation is the ultimate tool. Without it, research would still be confined to trial and error-based physical experiments and observation.

What problems need Parallel Computing?

Parallel Computing in the Real-world

EngineeringScienceBusinessGameCloud-computing

What This course can do for You?

Understanding of parallel computer architectures

Developing parallel programs for both clusters and shared memory multi-core system MPI/OpenMP

Know basics of CUDA programmingLearn to do performance analysis of parallel

programs

DefinitionsParallel computing

Using parallel computer to solve single problems faster

Parallel computerMultiple-processor/core system

supporting parallel programmingParallel programming

Programming in a language that supports concurrency explicitly

Classical Science

Nature

Observation

Theory PhysicalExperimentation

Modern Scientific Method

Nature

Observation

Theory PhysicalExperimentation

NumericalSimulation

Evolution of SupercomputingWorld War II

Hand-computed artillery tablesNeed to speed computationsENIAC

Cold WarNuclear weapon designIntelligence gatheringCode-breaking

SupercomputerGeneral-purpose computerSolves individual problems at high speeds,

compared with contemporary systemsTypically costs $10 million or moreTraditionally found in government labs

Commercial SupercomputingStarted in capital-intensive industries

Petroleum explorationAutomobile manufacturing

Other companies followed suitPharmaceutical designConsumer products

CPUs 1 Million Times FasterFaster clock speedsGreater system concurrency

Multiple functional unitsConcurrent instruction executionSpeculative instruction execution

Systems 1 Billion Times FasterProcessors are 1 million times fasterCombine thousands of processorsParallel computer

Multiple processorsSupports parallel programming

Parallel computing = Using a parallel computer to execute a program faster

Beowulf ConceptNASA (Sterling and Becker)Commodity processorsCommodity interconnectLinux operating systemMessage Passing Interface (MPI) libraryHigh performance/$ for certain applications

Computing speed of supercomputers

Projected Computing speed of supercomputers

Top 10 Supercomputers 2010.11

GPU

What you can useHardware

Multicore chips (2011: mostly 2 cores and 4 cores, but doubling) (cores=processors)

Servers (often 2 or 4 multicores sharing memory)

Clusters (often several, to tens, and many more servers not sharing memory)

Supercomputer at USC CEC

Supercomputers at USC CEC

76 Compute Nodes w/ dual 3.4 GHz 64 Nodes: Dual CPU

Supercomputers at USC CECSGI Altix 4700 Shared-memory systemHardware 128 Itanium Cores @ 1.6 GHz/ 8MB

Cache 256 GB RAM 8TB storage NUMAlink Interconnect FabricSoftware SUSE10 w/SGI PROPACK Intel C/C++ and Fortran Compilers VASP PBSPro scheduling software Message Passing Toolkit Intel Math Kernel Library GNU Scientific Library Boost library

Some historical machines

Earth Simulator was #1

Some interesting hardware

Nvidia Cell Processor

Sicortex – “Teraflops from Milliwatts”

http://www.gizmag.com/mit-cycling-human-powered-computation/8503/

http://www.sicortex.com/products/sc648

GPU-based supercomputing+CUDA

Topic1: Hardware Architecture of parallel computing system

Topic2: Programming/SoftwareCommon parallel computing methodsPBS- job scheduling systemMPI: The Message Passing Interface

Low level “lowest common denominator” language that the world has stuck with for nearly 20 years

Can get performance, but can be a hindrance as well

Pthread for multi-core shared memory parallel programming

CUDA GPU programmingMapReduce Google style high-

performance computing

Why MPI?MPI = “Message Passing Interface”Standard specification for message-passing

librariesLibraries available on virtually all parallel

computersFree libraries also available for networks of

workstations or commodity clusters

Why OpenMP?OpenMP an application programming

interface (API) for shared-memory systemsSupports higher performance parallel

programming of symmetrical multiprocessors

Topic3: PerformanceSingle processor speeds for now no longer

growing.Moore’s law still allows for more real estate per

core (transistors double/nearly every two years)http://www.intel.com/technology/mooreslaw/index.htm

People want performance but hard to getSlowdowns seen before speedupsFlops (floating point ops / second)

Gigaflops (109), Teraflops (1012), Petaflops(1015)

Summary (1/2)High performance computing

U.S. governmentCapital-intensive industriesMany companies and research labs

Parallel computersCommercial systemsCommodity-based systems

Summary (2/2)Power of CPUs keeps growing

exponentiallyParallel programming environments

changing very slowlyTwo standards have emerged

MPI library, for processes that do not share memory

OpenMP directives, for processes that do share memory

Places to LookBest current news:

http://www.hpcwire.com/Huge Conference:

http://sc09.supercomputing.org/http://www.interactivesupercomputing.com

Top500.org