University of Tartu, Institute of Computer Science

Introduction to Scientific Computing

MTAT.08.025

eero.vainikko@ut.ee

Spring 2017

2 Practical information

Lectures: Liivi 2 - 122 MON 10:15Computer classes: Liivi 2 - 205, MON 12:15 Amnir Hadachi <am-

nir.hadachi@ut.ee>3 ECTS

Lectures: 16h; Computer Classes: 16h; Independent work: 46hFinal grade forms from :

1. Active partitipation at lectures 10%

2. Stand-up quiz(es) 10%

3. Computer class activities 50%

4. Exam 30%

Course homepage (http://courses.cs.ut.ee/2017/isc)

3 Introduction 1.1 Syllabus

1 Introduction1.1 SyllabusLectures:

• Python for Scientific Computing NumPy, SciPy

• Scientific Computing - an Overview

• Floating point numbers, how to deal with roundoff errors

• Large problems in Linear Algebra, condition number

• Memory hierarchies and making use of it

• Numerical integration and differentiation

• Numerical solution of differential and integral equations

• Fast Fourier Transform

4 Introduction 1.1 Syllabus

Computer Classes (preliminary plan)

1. Python for Scientific Computing; Fibonacci numbers; Collatz conjecture

2. Discretization and round-off errors

3. NumPy arrays, matrices; LU-Factorization with Gauss Elimination Method(GEM)

4. UT HPC server; LU-Factorization and GEM on HPC cluster

5. Floating point numbers

6. Fractals

7. Fourier series and Fast Fourier Transform

8. Discrete-time models and ordinary differential equations

5 Introduction 1.2 Literature

1.2 Literature

General Scientific Computing:

1. RH Landau, A First Course in Scientific Computing. Symbolic, Graphic, andNumeric Modeling Using Maple, Java, Mathematica, and Fortran90. PrincentonUniversity Press, 2005.

2. LR Scott, T Clark, B Bagheri. Scientific Parallel Computing. Princenton Uni-versity Press, 2005.

3. MT Heath, Scientific Computing; ISBN: 007112229X, McGraw-Hill Compa-nies, 2001.

4. JW Demmel, Applied Numerical Linear Algebra; ISBN: 0898713897, Societyfor Industrial & Applied Mathematics, Paperback, 1997.

6 Introduction 1.2 Literature

Python for SC:

1. Hans Petter Langtangen, A Primer on Scientific Programming withPython, Springer, 2009. Book webpage (http://vefur.simula.no/intro-programming/).

2. Hans Petter Langtangen, Python Scripting for Computational Science. ThirdEdition, Springer 2008. Web-site for the book (http://folk.uio.no/hpl/scripting/).

3. Neeme Kahusk, Sissejuhatus Pythonisse (http://www.cl.ut.ee/inimesed/nkahusk/sissejuhatus-pythonisse/).

4. Travis E. Oliphant, Guide to NumPy (http://www.tramy.us), TrelgolPublishing 2006.

5. Help documentation inside sage – used during Computer Classes

7 Introduction 1.3 Scripting vs programming

1.3 Scripting vs programming

1.3.1 What is a script?

• Very high-level, often short, programwritten in a high-level scripting language

• Scripting languages:

– Unix shells,

– Tcl,

– Perl,

– Python,

– Ruby,

– Scheme,

– Rexx,

– JavaScript,

– VisualBasic,

– ...

8 Introduction 1.3 Scripting vs programming

1.3.2 Characteristics of a script

• Glue other programs together

• Extensive text processing

• File and directory manipulation

• Often special-purpose code

• Many small interacting scripts may yield a big system

• Perhaps a special-purpose GUI on top

• Portable across Unix, Windows, Mac

• Interpreted program (no compilation+linking)

9 Introduction 1.3 Scripting vs programming

1.3.3 Why not stick to Java, C/C++ or Fortran?

Features of Perl and Python compared with Java, C/C++ and Fortran:

• shorter, more high-level programs

• much faster software development

• more convenient programming

• you feel more productive

• no variable declarations ,but lots of consistency checks at run time

• lots of standardized libraries and tools

10 Introduction 1.4 Scripts yield short code

1.4 Scripts yield short code

Consider reading real numbers from a file, where each line can contain an arbitrarynumber of real numbers:� �1.1 9 5.2

1.762543E-02

0 0.01 0.001 9 3 7�Python solution:� �

F = open(filename, ’r’)

n = F.read().split()�Perl solution:� �

open F, $filename;

$s = join "", <F>;

@n = split ’ ’, $s;�

11 Introduction 1.5 Performance issues

Ruby solution:� �n = IO.readlines(filename).join.split�

...Doing this in C++ or Java1 requires at least a loop, and in Fortran and C quitesome code lines are necessary

1.5 Performance issues

1.5.1 Scripts can be slow

• Perl and Python scripts are first compiled to byte-code

• The byte-code is then interpreted

1True in case of older Java implementations. With Java 8 and Java Streames that come with it, forexample, plain Java 8 code does the same:Files.lines(filePath).flatMap( Pattern.compile("\\s+")::splitAsStream );

12 Introduction 1.5 Performance issues

• Text processing is usually as fast as in C

• Loops over large data structures might be very slow

� �for i in range(len(A)):

A[i] = ...�• Fortran, C and C++ compilers are good at optimizing such loops at compile time

and produce very efficient assembly code (e.g. 100 times faster)

• Fortunately, long loops in scripts can easily be migrated to Fortran or C (orspecial libraries like numpy!)

13 Introduction 1.5 Performance issues

1.5.2 Scripts may be fast enough

Read 100 000 (x,y) data from file and write (x,f(y)) out again

• Pure Python: 4s

• Pure Perl: 3s

• Pure Tcl: 11s

• Pure C (fscanf/fprintf): 1s

• Pure C++ (iostream): 3.6s

• Pure C++ (buffered streams): 2.5s

• Numerical Python modules: 2.2s (!)

• Remark: in practice, 100 000 data points are written and read in binary format,resulting in much smaller differences

14 Introduction 1.5 Performance issues

1.5.3 When scripting is convenient

• The application’s main task is to connect together existing components

• The application includes a graphical user interface

• The application performs extensive string/text manipulation

• The design of the application code is expected to change significantly

• CPU-time intensive parts can be migrated to C/C++ or Fortran

• The application can be made short if it operates heavily on list or hash structures

• The application is supposed to communicate with Web servers

• The application should run without modifications on Unix, Windows, and Mac-intosh computers, also when a GUI is included

15 Introduction 1.5 Performance issues

1.5.4 When to use C, C++, Java, Fortran

• Does the application implement complicated algorithms and data structures?

• Does the application manipulate large datasets so that execution speed iscritical?

• Are the application’s functions well-defined and changing slowly?

• Will type-safe languages be an advantage, e.g., in large development teams?

At this course we will be using python and corresponding libraries for Scientific Com-puting (numpy, scipy through sage)

NOTE: is Python really a scripting language?

• Object-oriented language!

• Cython

• PyPy – jit (just in time) compiler

16 What is Scientific Computing? 2.1 Introduction to Scientific Computing

2 What is Scientific Computing?2.1 Introduction to Scientific Computing

• Scientific computing – subject on crossroads of

• physics, chemistry, [social, engineering,...] sciences

– problems typically translated into

∗ linear algebraic problems

∗ sometimes combinatorial problems

• a computational scientist needs knowledge of some aspects of

– numerical analysis

– linear algebra

– discrete mathematics

17 What is Scientific Computing? 2.1 Introduction to Scientific Computing

• An efficient implementation needs some understanding of

– computer architecture

∗ both on the CPU level

∗ on the level of parallel computing

– some specific skills of software management

Scientific Computing – field of study concerned with constructing math-ematical models and numerical solution techniques and using computersto analyse and solve scientific and engineering problems

• typically – application of computer simulation and other forms of computationto problems in various scientific disciplines.

18 What is Scientific Computing? 2.1 Introduction to Scientific Computing

Main purpose of Scientific Computing:

• mirroring

• predictionof real world processes’

• characteristics

– behaviour

– development

Example of Computational SimulationASTROPHYSICS: what happens with collision of two black holes in the universe?Situation which is

• impossible to observe in the nature,

• test in a lab

• estimate barely theoretically

19 What is Scientific Computing? 2.1 Introduction to Scientific Computing

Computer simulation CAN HELPBut what is needed for simulation?

• adequate mathematical model (Einstein’s general relativity theory)

• algorithm for numerical solution of the equations

• enough big computer for actual realisation of the algorithms

Frequently: need for simulation of situations that could be performed experimantally,but simulation on computers is needed because:

• HIGH COST OF THE REAL EXPERIMENT. Examples:

– car crash-tests

– simulation of gas explosions

– nuclear explosion simulation

– behaviour of ships in Ocean waves

20 What is Scientific Computing? 2.1 Introduction to Scientific Computing

– airplane aerodynamics

– strength calculations in big constructions (for example oil-platforms)

– oil-field simulations

• TIME FACTOR. Some examples:

– Climate change predictions

– Geological development of the Earth (including oil-fields)

– Glacier flow model

– Weather prediction

• SCALE OF THE PROBLEM. Some examples:

– modeling chemical reactions on the molecular level

– development of biological ecosystems

21 What is Scientific Computing? 2.1 Introduction to Scientific Computing

• PROCESSES THAT CAN NOT BE INTERVENED Some examples:

– human heart model

– global economy model

22 What is Scientific Computing? 2.2 Specifics of computational problems

2.2 Specifics of computational problems

Usually, computer simulation consists of:

1. Creation of mathematical model – usually in a form of equations –physical properties and dependencies of the subject

2. Algorithm creation for numerical solution of the equations

3. Application of the algorithms in computer software

4. Using the created software on a computer in a particular simulationprocess

5. Visualizing the results in an understandable way using computer graph-ics, for example

6. Integration of the results and repetition/redesign of arbitrary given stepabove

23 What is Scientific Computing? 2.2 Specifics of computational problems

Most often:

• algorithm

– written down in an intuitive way

– and/or using special modeling software

• computer program written, based on the algorithm

• testing

• iterating

Explorative nature of Scientific Computing!

24 What is Scientific Computing? 2.3 Mathematical model

2.3 Mathematical model

GENERAL STRATEGY:REPLACE A DIFFICULT PROBLEM WITH A MORE SIMPLE ONE

• – which has the same solution

• – or at least approximate solution

– but still reflecting the most important features of the problem

25 What is Scientific Computing? 2.3 Mathematical model

SOME EXAMPLES OF SUCH TECHNIQUES:

• Replacing infinite spaces with finite ones (in maths sense)

• Infinite processes replacement with finite ones

– replacing integrals with finite sums

– derivatives replaced by finite differences

• Replacing differential equations with algebraic equations

• Nonlinear equations replaced by linear equations

• Replacing higher order systems with lower order ones

• Replacing complicated functions with more simple ones (like polynomials)

• Arbitrary structured matrix replacement with more simple structured matrices

26 What is Scientific Computing? 2.3 Mathematical model

AT THIS COURSE WE TRY TO GIVE:

an overview of some methods and analysis for development of reliable andefficient software for Scientific Computing

Reliability means here both the reliability of the software as well as adequatyof the results – how much can one rely on the achieved results:

• Is the solution acceptable at all? is it a real solution? (extraneous solution?,instability of the solution? etc); Does the solution algorithm guarantee a solutionat all?

• How big is the calculated solution’s deviation from the real solution? How wellthe simulation reflects the real world?

Another aspect: software reliability

27 What is Scientific Computing? 2.3 Mathematical model

Efficiency expressed on various levels of the solution process

• speed

• amount of used resources

Resources can be:

– Time

– Cost

– Number of CPU cycles

– Number of processes

– Amount of RAM

– Human labour

28 What is Scientific Computing? 2.3 Mathematical model

General formula:

minapproximation error

timeEven more generally: – minimise time of the solution

Efficient method requires:

(i) good discretisation

(ii) good computer implementation

(iii) depends on computer architecture (processor speed, RAM size, mem-ory bus speed, availability of cache, number of cache levels and otherproperties )