Multiple Discrete Random Variables

Introduction

Consider the choice of a student at random from a population. We wish to know student’s height, weight, blood pressure, pulse rate, etc.

The mapping from sample space of students to measurements of height and weight, would be H(si) = hi, W(si) = wi, of the student selected.

The table is a two-dimensional array that lists the probability P[H = hi and W =wj].

Introduction The information can also be

displayed in a three-dimensional format.

We will study dependencies between the multiple RV. For example: “Can we predict a person’s height from his weight?”

These probabilities were termed join probabilities. The height and weight could be represented as 2 x 1 random vector.

Jointly Distributed RVsConsider two discrete RVs X and Y. They represent the functions that map an outcome of an experiment si to a value in the plane.

for all

The exp. consists of the simultaneous tossing of a penny and a nickel.

Two random variable that are defined on the same sample space S are said to be jointly distributed.

Jointly Distributed RVs There are four vectors that comprise the sample space

The values of the random vector (multiple random variables) are denoted either by (x,y) a point in the plane or [x y]T a 2D vector.• The size of the sample space for discrete RV can be • Finite• Countably infinite.

• If X can take on 2 values Nx = 2, and Y can take on 2 values NY =2, the total number of elements in SX,Y is NXNY = 4.

Jointly Distributed RVs Generally, if SX = {x1, x2,…,xNx} and SY = {y1, y2,…,yNy},

then the random vector can take on values in

The notation A × B, denotes a Cartesian product set.

The joint PMF (bivariate PMF) as

Properties of joint PMF

Property 1. Range of values of joint PMF

Property 2. Sum of values of joint PMF

Similarly for a countably infinite sample space.

For two fair coins that do not interact as they are tossed we might assign pX,Y[i,j] = ¼.

The procedure to determine the joint PMF from the probabilities defined on S

The procedure depends on whether the RV mapping is one-to-one or many-to-one. For a one-to-one mapping from S to SX,Y we have

It is assumed that sk is the only solution to X(s) = xi and Y(s) = yj.

For a many-to-one transformation the joint PMF is found as

Two dice toss with different colored dice A red die and a blue die are tossed. The die that yields the larger

number of dots is chosen. If both dice display the same number of dots, the red die is chosen. The numerical outcome of the experiment is defined to be 0 if the

blue die is chosen and 1 if the red die is chosen, along with its corresponding number of dots.

What is pX,Y[1,3] for example?

Two dice toss with different colored diceTo determine the desired value of the PMF, we assume that each outcome in S is equally likely and therefore is equal to 1/36.

Since there are three outcomes that map into (1,3).

In general, we can use the joint PMF, to find probability of event A defined on SX,Y = SX × SY.

Marginal PMFs and CDFs

If pX,Y[x,y] is known, then marginal probabilities pX[xi] and pY[yi] can be determined. Consider an event of interest A on countably infinite sample space .

Let A = {xk} × SY. Then,

with i = k only

with j = k only

Example: Two coin toss

A penny (RV X) and a nickel (RV Y) are tossed and the outcomes are mapped into a 1 for a head and a 0 for a tail. Consider the joint PMF

The marginal PMFs are given as

=1

=1

Joint PMF cannot be determined from marginal PMFs

It is not possible in general to obtain joint PMF from marginal PMFs. Consider the following joint PMF

The marginal PMFs are the same as the ones before. There are an infinite number of joint PMFs that have the same

marginal PMFs.joint PMF marginal PMFs marginal PMFs joint PMF

Joint cumulative distribution function A joint cumulative distribution function (CDF) can be defined

for a random vector as and can be found explicitly by summing the joint PMFs as

The PMF can be recovered as

Properties of Cumulative distribution functions

The marginal CDFs can be easily found from the joint CDF as

Property 1. Range of values

Property 2. Values of “endpoints”

Property 3. Monotonically increasingMonotonically increases as x and/or y increases.

Property 4. “Right” continuous• The joint CDF takes the value after the jump.

Independence of Multiple RV Consider the experiment of tossing a coin and then a die.

The outcome of the coin X = {0,1} and The outcome of a die Y = {1,2,3,4,5,6}

hence the probability of the random vector (X,Y) taking on a value Y = yi does not depend on X = xi.

X and Y are independent random variables if all the joint events on SX,Y are independent.

The probability of joint events may be reduced to probabilities of “marginal events”.

If A = {xi} and B = {yj}, then

and

are independent

Independence of Multiple RVThe converse it true.

If the joint PMF factors, then X and Y are independent.

Example: Two coin toss – independenceAssume we toss a penny and nickel. If all outcomes are equivalently the joint PMF is given by

marginal probability

Independence of Multiple RV

Example: Two coin toss – dependenceConsider the same experiment but with a joint PMF given by

Then pX,Y[0,0] = 1/8 ≠ (1/4)(3/8) = pX[0]pY[0] and hence X and Y cannot be independent.

If two random variables are not independent, they are said to be dependent.

Independence of Multiple RV Example: Two coin toss – dependent but fair coins

Consider the same experiment again but with joint PMF given by

Since pX,Y[0,0] = 3/8 ≠(1/2)(1/2), X and Y are dependent.

But, by examining the marginal PMFs we see P[heads] = ½ (fair??), we might conclude that the RVs were independent. This is incorrect.

If the RVs are independent, the joint CDF factors as well.

Transformations of Multiple Random Variables

The PMF of Y = g(x) if the PMF of X is known is given by

In the case of two discrete RVs X and Y that are transformed into W = g(X, Y) and Z = h(X, Y), we have

Sometimes we wish to determine the PMF of Z = h(X, Y) only. Then we can use auxiliary RV W = X, so that pZ is the marginal PMF and can be found form the formula above as

Example: Independent Poisson RVs

Assume that the joint PMF is give as the product of the marginal PMFs, and each PMF is Poisson PMF.

Consider the transformation

We need to determine all (k,l) so that

But xk, yl and wi, zj can be replaced by k,l and i,j each with 0,1,….

Example: Independent Poisson RVsApply the given transformation we get

Solving for (k, l) for the given (i, j), we have

We must have l ≥ 0 so that l = j – i ≥ 0.

discrete unit step

Use the discrete unit step sequence to avoid mistakes

The discrete unit step sequence was introduced to designate the region of w-z plane over which pW,Z[i,j] is nonzero.

The transformation will generally change the region over which the new joint PMF is nonzero.

A common mistake is to disregard this region and assert that the joint PMF is nonzero over i = 0,1,…; j = 0,1,….

To avoid possible errors unit steps are applied

Example: Independent Poisson RVsTo find the PMF of Z = X + Y from the joint PMF obtained earlier we set W = X so we have SW = SX = {0,1,…} and

Since u[i] = 1 for i = 0,1,… and u[j - i] = 1 for i = 0,1,…,j and u[j - i] = 0 for i > j, we drop u[i]u[j – i] multipliers.Note that Z can take on values j = 0,1,… since Z = X + Y.

Connection to characteristic function Generally the formula for the PMF of the sum of any two

discrete RV X and Y, dependent or independent is given by

If the RV are independent, then since the joint PMF must factor, we have the result

This summation is a discrete convolution. Taking the Fourier transformation (defined with a +j) of both sides produces

Example: Independent Poisson RVs using CF approach

We showed that if X ~ Pois(λ), then

Thus using the above Fourier property we have

But the CF in the braces is that of a Poisson RV and corresponds to

• The use of CF for determination of PMF for a sum of independent RV has considerably simplified the derivation.

• In summary, if X and Y are independent RV with integer values, then the PMF of Z = X + Y is given by

Transformation of a fine sample space

It is possible to obtain the PMF of Z = g(X,Y) by a direct calculation if the sample SX,Y is finite.

To first obtain the transformed joint PMF pw,z we 1. determine the finite sample space SZ.2. determine which sample points (xi,yj) in SX,Y map into each

3. sum the probabilities of those (xi,yj) sample points to yield pz[zk].

Mathematically this is equivalent to

Direct computation of PMF for transformed RV

Consider the transformation of the RV (X,Y) into the scalar RV Z = X2 + Y2. The joint PMF is given by

To find the PMF for Z first note that (X,Y) takes on the values (i,j) = (0,0),(1,0),(0,1),(1,1). Therefore, Z must take on the values

zk = i2 + j2 = 0,1,2.

Then

Expected Values

If Z = g(X, Y), then by definition its expected value

Or using a more direct approach

EX. Expected value of a sum of random variables: Z = g(X,Y) = X + Y

Expected value of a product of RV

If Z = g(X, Y) = XY, then

If X and Y are independent, then since the joint PMF factors, we have

More generally,

Variance of a sum of RVs

Consider the calculation of var(X + Y). Then, letting Z = g(X,Y) = ((X + Y) – EX,Y[(X + Y)])2,

we have

The last term is called the covariance and defined as

or alternatively

Joint moments

The questions of interestIf the outcomes of one RV is a given value, what can we say

about the outcome of the other RV?

There is clearly a relationship between height and weight.

Joint moments

To quantify these relationships we form the product XY, which can take on the values +1, -1, and ±1 for the joint PMF above.

To determine the value of XY on the average we define the joint moment as EX,Y[XY].

For the case (a)

Joint moments

In previous example EX[X] = EY[Y] = 0.

If means aren’t zero, the joint moments will depend on the values of the means.

To nullify this effect it is convenient to use the joint central moments.

That will produce the desired +1 for the joint PMF above.

Independence implies zero covariance but zero covariance does not imply independence

For the joint PMF the covariance is zero since

Consider the joint PMF which assigns equal probability ½ to each of the four points point.

and thus

However, X and Y are dependent because pX,Y[1,0] = 1/4 but pX[1]pY[0]=(1/4)(1/2) = 1/8.

The joint k-lth moment

More generally the joint k-lth moment is defined as

For k = 1,2,…; l =1,2,…, when it exists. The joint k-lth moment central moment is defined as

For k = 1,2,…; l =1,2,…, when it exists.

Prediction of a RV outcome

The covariance between two RVs is useful for predicting Y based on knowledge of the outcome of X.

We seek a predictor Y that is linear in X or

The constants a and b are to be chosen so that “on the average” the observed value of aX + b is close to the observed value of Y.

The solution is given by

Prediction of a RV outcomeThe the optimal linear prediction of Y given the outcome X = x is

Example: Predicting one RV outcome from knowledge of second RV outcome.

We found from marginals

Regression line

Prediction of a RV outcome

We could also have predicted X from Y = y by interchanging X and Y.

If cov(X,Y) = 0, then or X = x provides no information to predict Y i.e. X and Y are independent.

If the cov is zero, the RV can still be dependent.

A standardized random variable A standardized RV is defined to be for which the mean is zero

and the variance is one.

Example: if X ~ Pois(λ), then• Let’s find the best linear prediction of the standardized Y based

on a standardized Xs = xs.

then

and therefore

Previous example continued

For the previous example we have

then

and

so

Correlation coefficient

The factor that scales xs to produce YS is called correlation coefficient (CC)

When X and Y have ρX,Y ≠0 , then X and Y are said to be correlated.

If covariance is zero and hence ρX,Y =0 then the RVs are said to be uncorrelated.

Property Property: Correlation coefficient is always less than or equal to

one in magnitude or Proof: For RVs V and W the Cauchy-Schwarz inequality says that

With equality if and only if W = cV for c a constant. Thus letting V = X – EX[X] and W = Y – EY[Y], we have

• Equality will hold if and only if W = cV or equivalently if Y – EY[Y] = c(X – EX[X]), which is easily shown to imply that

Correlation does not imply a causal relationship between RVs

A frequent misapplication of probability is to assert that two quantities that are correlated (ρX,Y ≠0 ) are such because one causes the other.

Incidence of prostate cancer per 1000 individuals older than age 55 versus height.

Correlation between two variables only indicates an association i.e. if one increases, then so does the other (or vice versa).

Joint Characteristic Functions

For the RV X and Y it is defined as

• It is seen to be the 2D Fourier transform of the two-dimensional sequence pX,Y[k,l].

• The joint moments are given by the formula

• Assuming both RV take on integer values, it is evaluated using as

Joint Characteristic Functions

The import application is to finding the PMF for the sum of

independent RVs X and Y then the joint characteristic function

factors due to the property EX,Y[g(X)h(Y)] = EX[g(X)]EY[h(Y)].

Joint Characteristic Functions If the joint CF factors, then X and Y are independent RVs. Consider the transformed RV W = g(X) and Z = h(Y),

where X and Y are independent. Prove that W and Z are independent as well. The joint CF of the transformed RVs is

But we have that

Real-world Example: Assessing Health Risks

Obesity is found to be associated with many life-threatening illnesses, especially diabetes.

The definition of an obese person is given by the BMI as

Where W and H is weight and height in inches.25 < BMI < 30 overweight

30 < BMI obese

We will use a hypothetical population of college students. For this population we would like to know the probability of obese persons (i.e. 30 < BMI ).

Real-world Example: Assessing Health Risks

Computer simulation of Random Vectors If X and Y are independent, then we generate

a realization of X according to pX[xi] and a realization Y, according to pY[yj]

Concatenating the realizations together we form the vector of random variables.

j = 0 j = 1

i = 0 1/8 1/8

i = 1 1/4 1/2

Computer simulation of Random Vectors

Once the realization are available we can estimate the joint PMF and marginal PMFs

• And the joint moments estimated as

Practice problems Two coins are tossed in succession with a head being mapped

into a +1 and a tail mapped into a -1. If a RV is defined as (X,Y) with X representing the mapping of the first toss and Y representing the mapping of the second toss, draw the mapping. Also, what is SX,Y? (Hint: see slides 4,5).

Two dice are tossed. The number of dots observed on the dice are added together to form the random variable X and also difference to form Y. Determine the possible outcomes of the random vector (X,Y) and plot them in the plane. How many possible outcomes are there?

Is a valid joint PMF? For a given joint PMF is given find marginal probability

53

Practice problems Find a formula for var(X - Y) similar to

What can you say about the relationship between var(X + Y) if X and Y are uncorrelated? Find the covariance for the joint PMF given in the table

How do you know the value that you obtained is correct?

Homework

1

2) The values of a joint PMF are given below. Determine the marginal PMFs

3)

55

Homework4) Prove that the minimum mean square error of the optimal linear predictor is given by