GuruPrevails Machine Learning Course

Basic Math

Examples given in Octave code

Victor K Miclovich

vicmiclovich@{ugandasoft, gmail}.com

December 5, 2009

2

Preface

I would like to welcome you to this tutorial courtesy of GuruPrevails. Wewant to make Machine learning a really wonderful experience and I hopethese tutorials shall give you great satisfaction.Good luck and have fun!

Victor K MiclovichDecember, 2009

Uganda

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1 Introduction 51.1 Starting Octave . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.1.1 Using Octave like a calculator . . . . . . . . . . . . . . 51.1.2 Inbuilt functions . . . . . . . . . . . . . . . . . . . . . 61.1.3 Some constants to take note of. . . . . . . . . . . . . . 6

2 Linear Algebra basics 72.1 Matrix(-ces) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.1 Representing a matrix . . . . . . . . . . . . . . . . . . 72.1.2 Matrix operations... at least some of them . . . . . . . 8

2.2 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3 Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3.1 What is a vector? . . . . . . . . . . . . . . . . . . . . 102.3.2 What is a vector space? . . . . . . . . . . . . . . . . . 102.3.3 The real definition! . . . . . . . . . . . . . . . . . . . . 10

3 Probability Theory 133.1 Simple terminology . . . . . . . . . . . . . . . . . . . . . . . . 13

3.1.1 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.1.2 Probability space or Sample space . . . . . . . . . . . 133.1.3 Probability . . . . . . . . . . . . . . . . . . . . . . . . 133.1.4 Review of basic Probability concepts . . . . . . . . . . 133.1.5 Law of Total Probability . . . . . . . . . . . . . . . . . 143.1.6 Baye’s Theorem . . . . . . . . . . . . . . . . . . . . . 14

3.2 Where Probability occurs in Octave . . . . . . . . . . . . . . 143.3 Functions that are probabilistic in nature . . . . . . . . . . . 15

4 What next? 174.1 Alternatives that we are going to use . . . . . . . . . . . . . . 17

4.1.1 What makes Python different from Octave/MatLab? . 17

3

4 CONTENTS

Chapter 1

Introduction

In this section I briefly demonstrate the power of Octave in solving mathe-matical problems.Octave is supported by many operating systems: commonly in MicrosoftWindows, Linux and Mac OS... In this tutorial, I will target an audiencewith a strong UNIX1. So get ready...

1.1 Starting Octave

1. Start your terminal2

2. Type Octave3

3. Start hacking ; you should see a prompt something like thisoctave:1>

1.1.1 Using Octave like a calculator

Octave can be used like a calculator; this just means you could type math-ematical expressions and have a result come back to you straight away.

Addition

octave:1> 3 + 4=>7This is a simple example... and almost any kind of operand4 will work.

1Don’t worry if you don’t have strong UNIX background... similar things can get donein Windows

2Windows users should look for Octave under the Programs menu of their machine;install Octave if you don’t have it.

3Windows users will probably click under the programs menu... and a terminal shouldpop out.

4In the above example, the operands are 3 and 4

5

6 CHAPTER 1. INTRODUCTION

Subtraction

octave:1> 5 - 2=>3

Multiplication

octave:1> 5 * 2=>10

1.1.2 Inbuilt functions

There are many inbuilt functions in Octave. Examples include sin, cos, tan,sqrt, etc.octave:1>sqrt(4)=>2octave:2>sin(pi / 2)=>1octave:3>sin(90)=>0.89405

1.1.3 Some constants to take note of.

octave:1>pi=>3.1416octave:2>e2.7183There are many other constants in science and math that even I may notknow about because that’s not my specialty... physicists for example willhave their known set of constants, chemists have theirs... Mathematicianslike I have our own special set of constants that we use...The next sections of this tutorial are going to be on more specific topics.Enjoy reading :)

5trigonometric functions usually take radians an input... e.g. 90◦ is π2

Chapter 2

Linear Algebra basics

2.1 Matrix(-ces)

2.1.1 Representing a matrix

A matrix is just an array of numbers. These numbers could be real numbers,integers or some other system of numbers or numerics.Mathematical representation of an m× n matrix

Am,n =

a1,1 a1,2 · · · a1,n

a2,1 a2,2 · · · a2,n...

.... . .

...am,1 am,2 · · · am,n

In Octave I show several types of matricesRow matrix

octave:1>A = [3,4,5]=>3 4 5

Column matrix

octave:2>A = [3; 4; 5]=>345A 2 × 3 matrixoctave:3>A = [1, 2, 3; 4, 5, 6;]=>

7

8 CHAPTER 2. LINEAR ALGEBRA BASICS

1 2 34 5 6or octave:3>A = [ [1, 2, 3]; [4, 5, 6]]=>1 2 34 5 6

A 3 × 3 matrix

octave:4>A = [ [1, 2, 3]; [4, 5, 6]; [7, 8, 9] ]=>1 2 34 5 67 8 9

2.1.2 Matrix operations... at least some of them

I am going to use the 3× 3 matrix A used in the previous sub-section.

Adding two matrices

octave:1># given two matrices A and some Boctave:2>A = [ [1,2,3]; [4, 5, 6]; [7, 8, 9] ];octave:3># a semi-colon at the end of the statement above preventsOctave from print A octave:4>B = [ [10, 11, 12]; [13,14,15]; [16,17,18]];octave:5>A + B=>11 13 1517 19 2123 25 27

Please, don’t get confused by my use of #; this is put inside your programto help you read and understand the logic of your program.

It is good practice to keep commenting your programs... any program-ming language’s decoder will understand that it does not have to execute anyinstructions in the line that has the ’#’. One can use a ’%’ symbol to com-ment. I’ve just decided to use ’#’ symbol at the start of that line.Other programming languages have other ways in which comments areplaced in their code.

Scalar products

2.1. MATRIX(-CES) 9

octave:1> newA = 2 * A;octave:2># remember to always give A some values. You can testother values...octave:3># A is a variable; a place in computer memory where westor the values octave:4># in this case those values are entries/elementsof an arrayoctave:5> newA=>2 4 68 10 1214 16 18

Take note of my use of newA; this is called variable... and the linenewA = 2 * A computes a value and puts it under that name... so I canuse newA anywhere in my program. Please read the Octave manual to getother concepts in programming using the Octave language. Don’t forget toPRACTICE

Finding a transpose of matrix/vectorA definition of transpose is in this footnote1

octave:1> A = [3,2,3; 1,3,6];octave:2> A transpose = A’;octave:3> A transpose=>3 12 33 6

DeterminantsDeterminants are usually calculated for square matrices n × n; you mightturn up with some nasty errors if you work with singular or m× n matriceswhere m 6= n

octave:1>A = [2,3,4];[3,15,20];[7,8,1];octave:1>det(A)=> -203.00

1Mathematical definition of a Transpose of a matrix: Given an m × n matrix, it’stranspose is just that matrix with rows interchanged for columns... an n×m;

[A]Tij = [A]ji

10 CHAPTER 2. LINEAR ALGEBRA BASICS

2.2 Assignment

You are going to test the following functions; study the math and octavecode too!

1. Given ~v = (3, 2, 3), ~u = (4, 6, 7), Find ~v.~uv = [3 2 3];u=[4 6 7];dot(v,u)

2. Using the above vectors, get the cross product, ~v×~u [Hint: cross(x,y)gives ~x× ~y]

3. Think of other vectors... try out 2 dimensional, 3-D, 4-D and n-dimensional vectors. Also, take note of my exclusion of a commai.e. v = [3 2 3] is the same as v = [3, 2, 3]

2.3 Vector Spaces

2.3.1 What is a vector?

A vector is a line segment whose length is its magnitude and whose orien-tation in space is its direction.

2.3.2 What is a vector space?

A vector space is a mathematical structure formed by a collection of vectors.

2.3.3 The real definition!

Let V be a set on which addition and scalar multiplications are defined2. Ifthat is the case, then for all objects3 ~u, ~v, and ~w in V and all scalers c andk then V is called a vector spaces and the objects in V are called vectorsBelow are some axioms we look at:

1. ~u + ~v is in V 4

2. c~u is in V 5

3. ~u + ~v = ~v + ~u

4. u + v = v + u2if ~u and ~v are objects in V and c is a scalar, then we can in some way, define ~v + ~u

and c~u or c~v.3we use objects here because vector has 2 things its describing: orientation and

length/size... which makes the term sound more ”important”.4closed under addition5closed under scalar multiplication

2.3. VECTOR SPACES 11

5. ~u + (~v + ~w) = (~u + ~v) + ~w

6. zero vector is a special vector in V space.

7. For every ~u in V there’s another object in V , denoted −~u and calledthe negative of ~u such that ~u + (−~u) = 0 or ~u− ~u = 0

8. c(~u + ~v) = c~u + c~v

9. (c + k)~u = c~u + k~v

10. c(k~u) = (ck)~u

11. 1× ~u = ~u

From your knowledge of the previous sections, implement these axioms forthe following vectors... if this isn’t enough you can set your questions :)

(a)~v = (2, 7)

octave:1> v = [-13; -25]Use the constants c = 3, 5, 6, k = 3, 5, 6 And test using the Octaveinterpreter6

6another name for the terminal that you see on running Octave

12 CHAPTER 2. LINEAR ALGEBRA BASICS

Chapter 3

Probability Theory

3.1 Simple terminology

3.1.1 Events

An event is an outcome or occurrence that has a probability assigned to it.

3.1.2 Probability space or Sample space

These are all the possible outcomes of an experiment or series of events. Wecan represent this mathematically using an S

3.1.3 Probability

To find the probability of an event A

P (A) =n(A)n(S)

At times we may need to find the complement of some probability A′, thatevent that does not occur.

P (A′) = 1− P (A)

3.1.4 Review of basic Probability concepts

To find the probability of getting event A or B,

P (A ∪B) = P (A) + P (B)− P (A ∩B)

For mutually exclusive events P (A∩B) is zero! So that means you just addup the probabilities of events A and B occurring! Conditional probabil-ities are measure the probability of one event occurring relative to anotheroccurring.

P (A|B) =P (A ∩B)

P (B)

13

14 CHAPTER 3. PROBABILITY THEORY

This can easily be written as:

P (A ∩B) = P (A|B)× P (B)

For some extra credit and work... please re-familiarize yourselves with theprobability tree... I won’t do that here...Probability trees make calculations of conditional probabilities easy... con-ditional probabilities occur a lot with decision making algorithms, recursivedepth search and many other algorithms you are gonna encounter in yourMachine learning careerer.

3.1.5 Law of Total Probability

If you have two events A and B, then

P (B) = P (B ∩A) + P (B ∩A′)= P (A)P (B|A) + P (A′)P (B|A′)

This has an interesting attribute1.

3.1.6 Baye’s Theorem

If you have n mutually exclusive and exhaustive events A1, A2, through toAn, and B is another event, we have then:

P (A|B) =P (A)P (B|A)

P (A)P (B|A) + P (A′)P (B|A′)

The Baye’s Theorem is used in computing as a way of filtering emails anddetecting spam... most training sets use Baye’s theorem; and this teaches a”program”/machine to easily detect commonly used words in Junk mail orSPAM; and so we can easily classify and cluster emails as correctly addressedand due to you and the ones that head off to your SPAM box.

3.2 Where Probability occurs in Octave

Probability in computers is in many causes referred to as ”psuedo-random”...calculations that are made by a bunch of transistors and crystals2, and othercomplicated electronics stuff that is set to compute something that is seem-ingly ”random” and that is what we use to mimic an un-planned-for event– an accident of chance!So... these are just a few cases where probability is used...

1The equation above is the denominator of Baye’s Theorem2timers

3.3. FUNCTIONS THAT ARE PROBABILISTIC IN NATURE 15

3.3 Functions that are probabilistic in nature

octave:1>rand()The rand() function returns a matrix with random elements uniformly dis-tributed on the interval (0, 1).

16 CHAPTER 3. PROBABILITY THEORY

Chapter 4

What next?

During the next few weeks we shall be looking at probability distributionsand have them implement in Octave... this is a rather intensive part of thiscourse... that of being able to convert normal mathematics into understand-ing!I shall however post as many resources as I can... please try to look throughboth the code and references that I leave.The next tutorial shall be about generation of vectors! Watch this space!

4.1 Alternatives that we are going to use

I have written this section to encourage you all to get in times with Python.Python some fabulous support for many algorithms and scientific computing.

4.1.1 What makes Python different from Octave/MatLab?

• It is an object oriented interpreted programming language

• It has extensive support

• It is relatively fast; but if more speed is need extra programming canbe done in C and/or C++

• There’s a lot more fun stuff... like libraries that let you play with theweb; you can implement features like web scrappers, SPAM filters,signal analysis, training algorithms, etc.

During the course, we shall look at python and more math in detail.

17