PROBABILITY AND STATISTICS FOR ENGINEERING

Hossein Sameti

Department of Computer EngineeringSharif University of Technology

Let X represent a Binomial r.v ,Then from

for large n. In this context, two approximations are extremely useful.

2

1

2

1

.)(21

k

kk

knkk

kkn qp

kn

kPkXkP

.)()(

) " and between is of sOccurrence ("2

1

2

1

211 1

21

k

kk

knkk

kkkkkk qp

kn

XPXXXP

kkAP

The Normal Approximation (Demoivre-Laplace Theorem)

If hen for k in the neighborhood of np, we can approximate

(for example, when with p held fixed) And we have:

where

n

npq

.2

1 2/)( 2 npqnpkknk enpq

qpkn

,21

21 2/

2/)(

21

22

1

22

1

dyedxenpq

kXkP yx

x

npqnpxk

k

. , 22

11 npq

npkxnpq

npkx

As we know,

If and are within with approximation:

where

We can express this formula in terms of the normalized integral

that has been tabulated extensively.

2

1

2

1

)(21

k

kk

knkk

kkn qp

kn

kPkXkP

1k 2k , , npqnpnpqnp

,21

21 2/

2/)(

21

22

1

22

1

dyedxenpq

kXkP yx

x

npqnpxk

k

)(21)(

0

2/2

xerfdyexerfx y

. , 22

11 npq

npkxnpq

npkx

x erf(x) x erf(x) x erf(x) x erf(x)

0.05

0.10 0.15 0.20 0.25

0.30 0.35 0.40 0.45 0.50

0.55 0.60 0.65 0.70 0.75

0.01994 0.03983 0.05962 0.07926 0.09871 0.11791 0.13683 0.15542 0.17364 0.19146 0.20884 0.22575 0.24215 0.25804 0.27337

0.80 0.85 0.90 0.95 1.00

1.05 1.10 1.15 1.20 1.25

1.30 1.35 1.40 1.45 1.50

0.28814 0.30234 0.31594 0.32894 0.34134 0.35314 0.36433 0.37493 0.38493 0.39435 0.40320 0.41149 0.41924 0.42647 0.43319

1.55

1.60 1.65 1.70 1.75

1.80 1.85 1.90 1.95 2.00

2.05 2.10 2.15 2.20 2.25

0.43943 0.44520 0.45053 0.45543 0.45994 0.46407 0.46784 0.47128 0.47441 0.47726 0.47982 0.48214 0.48422 0.48610 0.48778

2.30

2.35 2.40 2.45 2.50

2.55 2.60 2.65 2.70 2.75

2.80 2.85 2.90 2.95 3.00

0.48928 0.49061 0.49180 0.49286 0.49379 0.49461 0.49534 0.49597 0.49653 0.49702 0.49744 0.49781 0.49813 0.49841 0.49865

A fair coin is tossed 5,000 times. Find the probability that the number of heads is between 2,475 to 2,525.

We need

Since n is large we can use the normal approximation. so that and

and So the approximation is valid for and

Example

Solution

).525,2475,2( XP

,21

p 500,2np .35npq

,465,2 npqnp475,21 k .525,22 k

,535,2 npqnp

Here,

Using the table,

Example - continued

2

1

2

.21 2/

21

x

x

y dyekXkP

.75 ,

75 2

21

1

npq

npkxnpq

npkx

516.075erf2

|)(|erf)(erf )(erf)(erf525,2475,2

12

12

xxxxXP

. 258.0)7.0(erf

The Poisson Approximation

For large n, the Gaussian approximation of a binomial r.v is valid only if p is fixed, i.e., only if and

What if is small, or if it does not increase with n?- for example when , such that is a fixed

number.

1np .1npq

np0p ,n np

The Poisson Theorem

If

Then

,0p ,n np

ek

ppkkn

nkPk

knkn !

)1(!)!(

!)( 𝑛→ ∞→

The Poisson Approximation

Consider random arrivals such as telephone calls over a line.

n : total number of calls in the interval as we have Suppose Δ : a small interval of duration Let be the probability of k calls in the interval Δ We have shown that

).,0( TT n

.Tn 0 T

1 2 n

𝑃𝑛 (𝑘 )=𝑃 (𝑘𝑖𝑛∆ )=(𝑛𝑘)𝑝𝑘𝑞𝑛−𝑘𝑤h𝑒𝑟𝑒𝑝= ∆𝑇

The Poisson Approximation

p: probability of a particular call occurring in Δ: as

Normal approximation is invalid here because is not valid.

: probability of obtaining k calls (in any order) in an interval of duration Δ ,

.T

p

0p .T

T

Tnp

knkn pp

kknnkP

)1(!)!(

!)(

0 T

1 2 n

The Poisson Approximation

Thus,

.)/1()/1(

!112111

)/1(!)()1()1()(

k

nk

knk

kn

nn

knk

nn

nnpk

npn

knnnkP

e

kkP

k

nnppn !)(lim

,0 ,the Poisson p.m.f

Suppose - two million lottery tickets are issued - with 100 winning tickets among them.

a) If a person purchases 100 tickets, what is the probability of winning?

Example: Winning a Lottery

Solution

.105102

100 ticketsof no. Total

tickets winningof no. Total 56

pThe probability of buying a winning ticket

X: number of winning tickets

n: number of purchased tickets ,

P: an approximate Poisson distribution with parameter

So, The Probability of winning is:

Winning a Lottery - continued

.005.0105100 5 np

,!

)(k

ekXPk

.005.01)0(1)1( eXPXP

b) How many tickets should one buy to be 95% confident of having a winning ticket?

we need

But or

Thus one needs to buy about 60,000 tickets to be 95% confident of having a winning ticket!

Winning a Lottery - continued

Solution

.95.0)1( XP

.320ln implies 95.01)1( eXP

3105 5 nnp .000,60n

A space craft has 100,000 components The probability of any one component being defective is The mission will be in danger if five or more components become defective. Find the probability of such an event.

n is large and p is small Poisson Approximation with parameter

Example: Danger in Space Mission

Solution

n

).0(102 5 p

2102000,100 5 np

.052.032

342211

!1

!1)4(1)5(

2

4

0

24

0

e

ke

keXPXP

k

kk

k

Conditional Probability Density Function

, )( )( xXPxFX

.0)( ,)(

)()|(

BPBP

BAPBAP

.)(

)( |)( )|(BP

BxXPBxXPBxFX

.0)()(

)( )( )|(

,1)()(

)( )( )|(

BPP

BPBXPBF

BPBP

BPBXPBF

X

X

Conditional Probability Density Function

Further,

Since for

),|()|( )(

)( )|)((

12

2121

BxFBxFBP

BxXxPBxXxP

XX

,12 xx

. )()()( 2112 xXxxXxX

,

)|()|(dx

BxdFBxf XX

2

1

21 .)|(|)(x

x X dxBxfBxXxP

x

XX duBufBxF

.)|()|(

Toss a coin and X(T)=0, X(H)=1. Suppose Determine

has the following form. We need for all x. For so that and

Example

Solution

}.{HB).|( BxFX

)(xFX

)|( BxFX

,)( ,0 xXx ,)( BxX.0)|( BxFX

)(xFX

x

(a)

q1

1

( | )XF x B

x

(b)

1

1

For so that

For and

Example - continued

, )( ,10 TxXx

HTBxX )( .0)|( and BxFX

,)( ,1 xXx

}{ )( BBBxX 1)()()|( and

BPBPBxFX

( | )XF x B

x

1

1

Given suppose Find

We will first determine

For so that

For so that

Example

Solution

),(xFX .)( aXB ).|( BxfX

).|( BxFX

. )|(

aXPaXxXPBxFX

xXaXxXax ,

.

)()()|(

aFxF

aXPxXPBxF

X

XX

)( , aXaXxXax .1)|( BxFX

Thus

and hence

Example - continued

, ,1

, ,)()(

)|(ax

axaFxF

BxFX

X

X

otherwise. ,0

,,)()(

)|()|( axaFxf

BxFdxdBxf

X

X

XX

)|( BxFX

)(xFX

xa

1

(a)

)|( BxfX

)(xfX

xa(b)

Let B represent the event with For a given determine and

Example

Solution

.

)()()( )(

)( )()(

|)( )|(

aFbFbXaxXP

bXaPbXaxXP

BxXPBxF

XX

X

bXa )( .ab),(xFX )|( BxFX ).|( BxfX

For we have and hence

For we have and hence

For we have so that

Thus,

Example - continued

,bxa })({ )( )( xXabXaxX

.)()()()(

)()()()|(

aFbFaFxF

aFbFxXaPBxF

XX

XX

XXX

,bx bXabXaxX )( )( )( .1)|( BxFX

,ax ,)( )( bXaxX.0)|( BxFX

otherwise.,0

,,)()(

)()|( bxa

aFbFxf

BxfXX

X

X

)|( BxfX

)(xfX

xa b

Conditional p.d.f & Bayes’ Theorem First, we extend the conditional probability results to random variables: We know that If is a partition of S and B is an arbitrary event,

then:

By setting we obtain:

)()|()()|()( 11 nn APABPAPABPBP

],,,[ 21 nAAAU

Son partition a form ,

)()|()()|()(

)()|()()|()(

Hence

)()|()()|()(

1

11

11

11

n

nn

nn

nn

AA

APAxfAPAxfxf

APAxFAPAxFxF

APAxXPAPAxXPxXP

}{ xXB

Conditional p.d.f & Bayes’ Theorem

Using:

We obtain:

For ,

.)(

)()|()|(BP

APABPBAP

)(

| )(

|)| AP

xFAxF

APxXP

AxXPxXAP

21 )( xXxB

).()(

)|()(

)()()|()|(

)(|

|

2

1

2

1

12

12

21

2121

APdxxf

dxAxfAP

xFxFAxFAxF

APxXxP

AxXxPxXxAP

x

x X

x

x X

XX

XX

Conditional p.d.f & Bayes’ Theorem

Let so that in the limit as

or

we also get

or

,0 , , 21 xxxx ,0

).()(

)|(|)(|lim

0AP

xfAxf

xXAPxXxAPX

X

.)(

)()|()|(| APxfxXAPAxf X

AX

,)()|()|()(1

dxxfxXAPdxAxfAP XX

dxxfxXAPAP X )()|()(

(Total Probability Theorem)

Bayes’ Theorem (continuous version)

using total probability theorem in

We get the desired result

,)(

)()|()|(| APxfxXAPAxf X

AX

.)()|(

)()|()|(|

dxxfxXAP

xfxXAPAxfX

XAX

probability of obtaining a head in a toss. For a given coin, a-priori p can possess any value in (0,1). : A uniform in the absence of any additional information After tossing the coin n times, k heads are observed. How can we update this is new information?

Let A= “k heads in n specific tosses”. Since these tosses result in a specific sequence,

and using Total Probability Theorem we get

Example: Coin Tossing Problem Revisited

Solution

)(HPp

)( pfP

)(pfP

,)|( knkqppPAP

)( pfP

p0 1

.)!1(

!)!()1()()|()(1

0

1

0

nkkndpppdppfpPAPAP knk

P

The a-posteriori p.d.f represents the updated information given the event A,

Using

This is a beta distribution. We can use this a-posteriori p.d.f to make further predictions. For example, in the light of the above experiment, what can we say about the

probability of a head occurring in the next (n+1)th toss?

Example - continued

| ( | )P Af p A

,)(

)()|()|(| APxfxXAPAxf X

AX

|( | ) ( )( | )

( )( 1)! , 0 1 ( , ).

( )! !

PP A

k n k

P A P p f pf p AP A

n p q p n kn k k

)|(| Apf AP

p10

Let B= “head occurring in the (n+1)th toss, given that k heads have occurred in n previous tosses”.

Clearly From Total Probability Theorem,

Using (1) in (2), we get:

Thus, if n =10, and k = 6, then

which is more realistic compared to p = 0.5.

Example - continued

,)|( ppPBP

1

0 .)|()|()( dpApfpPBPBP P

,58.0127)( BP

1

0

( 1)! 1( ) .( )! ! 2

k n kn kP B p p q dpn k k n