Chapter 5: JOINT PROBABILITYDISTRIBUTIONS
Part 2: Covariance and CorrelationSection 5-2
Consider the joint probability distribution fXY (x, y).
Is there a relationship between X and Y ? If so,what kind?
If you’re given information on X , does it giveyou information on the distribution of Y ? (Thinkof a conditional distribution). Or are they inde-pendent?
1
Below is a different joint probability distribu-tion for X and Y .
Does there seem to be a relationship between Xand Y ? Are they independent?
If you’re given information on X , does it giveyou information on the distribution of Y ?
How would you describe the relationship?
Is it stronger than the relationship on the pre-vious page? Do you know MORE about Y fora given X?
2
Below is a joint probability distribution for anindependent X and Y .
↑This picture is the give-awaythat they’re independent.
Does there seem to be a relationship between Xand Y ?
If you’re given information on X , does it giveyou information on the distribution of Y ?
3
Covariance
When two random variables are being consid-ered simultaneously, it is useful to describe howthey relate to each other, or how they vary to-gether.
A common measure of the relationship betweentwo random variables is the covariance.
•CovarianceThe covariance between the random variablesX and Y , denoted as cov(X, Y ), or σXY , is
σXY = E[(X − E(X))(Y − E(Y ))]
= E[(X − µX)(Y − µY )]
= E(XY )− E(X)E(Y )
= E(XY )− µXµY4
To calculate covariance, we need to find the ex-pected value of a function of X and Y . Thisis done similarly to how it was done in the uni-variate case...
For X, Y discrete,E[h(x, y)] =
∑x∑y h(x, y)fXY (x, y)
For X, Y continuous,E[h(x, y)] =
∫ ∫h(x, y)fXY (x, y)dxdy
—————————————————————
Covariance (i.e. σXY ) is an expected value ofa function of X and Y over the (X, Y ) space,if X and Y are continuous we can write
σXY =
∫ ∞−∞
∫ ∞−∞
(x−µX)(y−µY )fXY (x, y) dx dy
To compute covariance, you’ll probably use...
σXY = E(XY )− E(X)E(Y )
5
When does the covariance have a positive value?
In the integration we’re conceptually putting‘weight’ on values of (x− µX)(y − µY ).
What regions of (X, Y ) space has...(x− µX)(y − µY ) > 0?
• Both X and Y are above their means.
• Both X and Y are below their means.
• ⇒ Values along a line of positive slope.
A distribution that puts high probability on theseregions will have a positive covariance.
6
When does the covariance have a negativevalue?
In the integration we’re conceptually putting‘weight’ on values of (x− µX)(y − µY ).
What regions of (X, Y ) space has...(x− µX)(y − µY ) < 0?
•X is above its mean, and Y is below its mean.
• Y is above its mean, andX is below its mean.
• ⇒ Values along a line of negative slope.
A distribution that puts high probability on theseregions will have a negative covariance.
7
Covariance is a measure of the linear relationshipbetween X and Y .
If there is a non-linear relationship between Xand Y (such as a quadratic relationship), thecovariance may not be sensitive to this.
—————————————————————
When does the covariance have a zero value?
This can happen in a number of situations, butthere’s one situation that is of large interest...when X and Y are independent...
When X and Y are independent, σXY = 0.
8
If X and Y are independent, then...
σXY =
∫ ∞−∞
∫ ∞−∞
(x− µX)(y − µY )fXY (x,y) dx dy
=
∫ ∞−∞
∫ ∞−∞
(x− µX)(y − µY )fX(x)fY (y) dx dy
=
(∫ ∞−∞
(x− µX)fX(x)dx
)·(∫ ∞−∞
(y − µY )fY (y)dy
)=
(∫ ∞−∞
xfX(x)dx− µX)·(∫ ∞−∞
yfY (y)dy − µY)
= (E(X)− µX) · (E(Y )− µY )
= (µX − µX) · (µY − µY )
= 0
This does NOT mean... If covariance=0, thenX and Y are independent.
We can find cases to the contrary of the abovestatement, like when there is a strong quadraticrelationship between X and Y (so they’re notindependent), but you can still get σXY = 0.
Remember that covariance specifically looks fora linear relationship.
9
When X and Y are independent, σXY = 0.
For this distribution showing independence,there is equal weight along the positive and neg-ative diagonals.
10
A couple comments...
• You can also define covariance for discrete Xand Y :
σXY = E[(X − µX)(Y − µY )]
=∑x∑y(x− µX)(y − µY )fXY (x, y)
• And recall that you can get the expectedvalue of any function of X and Y :
E[h(X, Y )] =∫∞−∞
∫∞−∞ h(x, y)fXY (x, y) dx dy
or
E[h(X, Y )] =∑x∑y h(x, y)fXY (x, y)
11
Correlation
Covariance is a measure of the linear relationshipbetween two variables, but perhaps a more com-mon and more easily interpretable measure iscorrelation.
•CorrelationThe correlation (or correlation coefficient) be-tween random variables X and Y , denotedas ρXY , is
ρXY =cov(X, Y )√V (X)V (Y )
=σXYσXσY
.
Notice that the numerator is the covariance,but it’s now been scaled according to thestandard deviation of X and Y (which areboth > 0), we’re just scaling the covariance.
NOTE: Covariance and correlation will havethe same sign (positive or negative).
12
Correlation lies in [−1, 1], in other words,
−1 ≤ ρXY ≤ +1
Correlation is a unitless (or dimensionless)quantity.
Correlation...
• −1 ≤ ρXY ≤ +1
• If X and Y have a strong positive linear re-lation ρXY is near +1.
• If X and Y have a strong negative linearrelation ρXY is near −1.
• If X and Y have a non-zero correlation, theyare said to be correlated.
• Correlation is a measure of linear relation-ship.
• If X and Y are independent, ρXY = 0.
13
• Example: Recall the particle movement model
An article describes a model for the move-ment of a particle. Assume that a particlemoves within the region A bounded by the xaxis, the line x = 1, and the line y = x. Let(X, Y ) denote the position of the particle ata given time. The joint density of X and Yis given by
fXY (x, y) = 8xy for (x, y) ∈ A
a) Find cov(X, Y )
ANS: Earlier, we found E(X) = 45 ...
14
15
• Example: Book problem 5-43 p. 179.
The joint probability distribution is
x -1 0 0 1y 0 -1 1 0fXY 0.25 0.25 0.25 0.25
Show that the correlation between X and Yis zero, but X and Y are not independent.
16
17