Lesson 3: expectation, variance, covariance and correlation

The probability distribution of a discrete random variable lists every value it can take, with its probability: for home goals, ten values and ten probabilities. A few numbers summarise it, and samples estimate them: the expectation, a probability-weighted average; the variance and standard deviation, its spread; and, for two variables, the covariance and correlation, how they move together.

1. The centre of a distribution: expectation

How many goals does a home side score in a match, on average? Lesson 2 set out the distribution of home goals in the 1,900 Premier League matches of 2019/20 to 2023/24. Draw one of those matches at random and let \(X\) be the home side's goals: as in lesson 2, the capital \(X\) is the random variable and the lower-case \(x\) a value it takes. Then \(P(X = x)\) is the share of the 1,900 matches with \(x\) home goals: \(P(X = 1) = 609/1{,}900 = 0.321\). We want one number for where this distribution is centred.

The expectation weights each value by its probability and adds up:

In words: expectation = the sum, over every value, of the value times its probability.

Draw a match at random again and again, putting it back each time. By the law of large numbers of lesson 4, the share of draws with each value settles at its probability. Hence the average of the goals drawn, each value times its share, settles at \(E[X]\): the expectation is the long-run average.

Each probability here is a number of matches over 1,900, so the sum can be done in whole numbers, with no rounding: multiply each value by its matches, add, and divide by 1,900.

The expectation need not be a value \(X\) can take: no home side scores 1.563 goals.

A 0/1 variable turns a share into a mean, which lessons 4 and 5 use. Let \(W = 1\) if the home side wins and \(W = 0\) otherwise, with \(P(W = 1) = p\). Then

The mean of a 0/1 variable is the probability of a 1. The home side won 838 of the 1,900 matches, so the mean of \(W\) is the share of home wins, 0.441. For a continuous variable, such as a birthweight, a density replaces the probabilities and an integral replaces the sum; the meaning is unchanged.

We write \(\mu\) for \(E[X]\), or \(\mu_X\) when two variables must be told apart; expectation, expected value and population mean are three names for it. Books also write \(\mathbb{E}(X)\); the choice of letters is cosmetic. The population mean is a fixed feature of the distribution: 1.563 here. A sample mean is computed from data and changes from sample to sample: five samples of 10 matches drawn at random from the 1,900 gave 2.0, 2.2, 1.3, 1.0 and 1.1 goals (lesson 4 studies this variation).

The same weighting gives the expectation of any function \(g\) of \(X\):

In words: the average in the population of g(X). The variance below is the choice \(g(x) = (x - \mu)^2\); the covariance in section 4 weights a function of two variables by their joint probabilities.

2. How far values lie from the centre: variance and standard deviation

The mean gives the centre of the distribution and does not measure how far the values lie from it. A mean of 1.563 goals does not show whether scores of 0 or 5 are rare or routine: a side that scored 1 goal in 44% of its matches and 2 in the rest would also average 1.56. Our home sides scored none in 23% of matches and 5 or more in 3%. We need a number for the spread.

A first idea is the average deviation from the mean, where a deviation \(x - \mu\) is how far a value lies above the mean. The probabilities add to 1 and \(\sum_x x\,P(X = x) = \mu\), so for every distribution

Deviations above the mean cancel those below it, so their average says nothing about spread. We square them before averaging instead. A square is never negative, so nothing cancels, and a value twice as far from the mean counts four times as much. Distances without their signs, \(|x - \mu|\), would also stop the cancelling, but squares obey a rule that they do not: the variance of a sum follows from the two variances and the covariance (section 6), and under random sampling that rule gives the variance of the mean of \(n\) observations, \(\sigma^2/n\). The variance is the average squared distance from the mean, the choice \(g(x) = (x - \mu)^2\):

In words: variance = the probability-weighted average of the squared distances from the mean.

The table works it out for home goals, with deviations from the unrounded mean, 2,970/1,900, and each probability as matches/1,900.

The terms add to \(\operatorname{Var}(X) = 1.806\), in goals squared, since each is a squared number of goals.

The standard deviation

The square root returns the variance to goals, the units the data are measured in, and gives the standard deviation:

In words: standard deviation = square root of the variance.

It reads as a typical distance from the mean, but "typical" needs care: it is the square root of the average squared distance, not the average distance. The average distance, \(E[\,|X - \mu|\,]\), called the mean absolute deviation, is 1.082 goals. Squaring gives large distances more weight, so the standard deviation is the larger; both are in goals and on the same scale.

A shortcut avoids the deviations:

In words: variance = the mean of the squares minus the square of the mean.

For a 0/1 variable, \(W^2 = W\) (as \(0^2 = 0\) and \(1^2 = 1\)), so \(E[W^2] = p\) and

which is \(0.441 \times 0.559 = 0.2465\) for home wins. It is largest, 0.25, at \(p = 0.5\), where the outcome is least predictable: a guess is wrong half the time.

3. Changing units: the rules for means and variances

The same quantity can be recorded in different units, ounces or grams, dollars or cents, and its mean and variance change with the units by fixed rules. A change of units or of the zero point turns \(X\) into \(a + bX\): a weight in grams is the weight in ounces times 28.3495, so \(a = 0\) and \(b = 28.3495\). Weighting \(a + bx\) by the probabilities and adding gives \(a\), as the probabilities add to 1, plus \(b\mu\):

In words: shift and rescale the variable, and its mean is shifted and rescaled in the same way.

Each deviation becomes \((a + bx) - (a + b\mu) = b(x - \mu)\): the shift cancels and the deviation is multiplied by \(b\). So every squared deviation, and their average, is multiplied by \(b^2\):

In words: adding a constant leaves the variance alone; multiplying by b multiplies the variance by b squared and the standard deviation by |b|.

The rules hold for the mean and standard deviation of data too, since the values and their mean move together. The 1,388 birthweights of lesson 6, a sample of births, have a mean of 118.70 ounces and a standard deviation of 20.35 ounces. In grams these are \(118.70 \times 28.3495 = 3{,}365\) and \(20.35 \times 28.3495 = 577\), and the variance is \(28.3495^2 = 803.7\) times its value in ounces squared.

For two random variables, the expectation of a sum is the sum of the expectations, the additive property of expectations:

In words: the mean of a sum = the sum of the means.

No step of the derivation uses how \(X\) and \(Y\) are related, so the rule holds whether or not they are: total goals average \(1.563 + 1.310 = 2.873\) a match. It extends to any number of variables; lesson 4 uses it for the sample mean. The variance of a sum is not this simple: it depends on whether the variables move together.

4. Do two variables move together? Covariance

Economics asks whether two variables move together: whether pay is higher for workers with more education, or birthweight lower when the mother smoked. The covariance is one number for this: the expectation of the product of the two deviations from their means, each pair of values weighted by its joint probability:

In words: covariance = the probability-weighted average of the product of the two deviations from their means.

The product is positive when both variables are above their means or both below, and negative when one is above and the other below. So the covariance is positive when the variables move together and negative when they move in opposite directions. Lesson 9 draws this as four quadrants around the point of means.

Take the table of two 0/1 variables from lesson 2. Their means are their probabilities of a 1, \(\mu_X = 0.3\) and \(\mu_Y = 0.5\), and each pair of values contributes its product of deviations times its joint probability:

The two pairs in which \(X\) and \(Y\) are equal have positive products and contribute \(0.06 + 0.07 = 0.13\); the two in which they differ contribute \(-0.035 - 0.045 = -0.08\). The positive part is the larger, so the covariance is positive. Expanding the product, as for the variance shortcut, gives

In words: covariance = the mean of the product minus the product of the means.

Here \(XY = 1\) only when both are 1, so \(E[XY] = 0.2\), and \(0.2 - 0.3 \times 0.5 = 0.05\): the same answer.

A covariance is in the units of one variable times those of the other, since each product multiplies the two deviations. For the 526 US workers of the WAGE1 data (1976), the covariance of the hourly wage and years of education, computed from the data as in section 7, is 4.151 dollars an hour times years. Its size cannot be read on its own, since it changes with the units: with the wage in cents every wage deviation is 100 times larger, and the same workers give 415.1. A variable's covariance with itself is its variance: put \(Y = X\) in the definitions to convince yourself.

5. Correlation: covariance without units

To measure how closely two variables move together on a scale free of their units, divide the covariance by both standard deviations:

In words: correlation = covariance divided by the product of the two standard deviations.

The numerator and the denominator are both in the units of \(X\) times those of \(Y\), so the units cancel, and a change of units leaves the correlation alone: with the wage in cents, the covariance and the standard deviation of the wage both rise 100-fold. For the workers, whose sample standard deviations are 3.693 dollars an hour and 2.769 years, the sample correlation, \(r\) (section 7), is

and \(415.1/(369.3 \times 2.769) = 0.406\) in cents. For the 0/1 table, \(\sigma_X = \sqrt{0.21} = 0.458\) and \(\sigma_Y = \sqrt{0.25} = 0.5\), so the correlation is \(0.05/(0.458 \times 0.5) = 0.218\).

If the points lie on a straight line, \(Y = a + bX\) with \(b\) not zero, each deviation of \(Y\) is \(b\) times that of \(X\) (section 3). So \(\operatorname{Cov}(X, Y) = b\operatorname{Var}(X)\), \(\sigma_Y = |b|\,\sigma_X\) and the correlation is \(b/|b|\): 1 for a rising line, -1 for a falling one. In standard units, the average squared distance between the second variable and \(\rho\) times the first is \(1 - \rho^2\), which cannot be negative. Hence a correlation lies between -1 and 1, closer to 0 the more the points scatter about a line.

Figure 1 has four real pairs of variables, with the correlation computed from the data in each panel's title. The 20 clubs are the whole league in 2023/24, not a sample. In the last panel each bubble is one score, its area the number of matches that ended that way.

Each number measures the strength of a straight-line association: the clubs' points rise almost in step with goal difference, while wages, though higher on average with more education, are widely spread at every level. A correlation is not a slope: for the clubs it is 0.96, while the least-squares line of lesson 9 rises 0.62 points per goal of goal difference, a number whose size depends on the units. Nor is a correlation a cause: all four measure association.

The two correlations of -0.15 come from different pictures. The births stand in columns, since 1,176 of the mothers reported no cigarettes and smokers reported round numbers; match scores sit on a grid of whole numbers. A correlation reduces a scatter to one number and leaves its shape out, so look at the scatter as well.

Zero correlation is not independence

If \(X\) and \(Y\) are independent, every joint probability is the product of the two marginal ones, so the double sum for \(E[XY]\) splits into two single sums:

and the covariance is zero: independence implies zero covariance, and so zero correlation. The reverse fails. Let \(X\) be -1, 0 or 1 with probability 1/3 each, and \(Y = X^2\) (Figure 2).

Then \(E[X] = 0\), \(E[Y] = 2/3\) and \(E[XY] = E[X^3]\), which is \((-1 + 0 + 1)/3 = 0\), so \(\operatorname{Cov}(X, Y) = 0 - 0 \times 2/3 = 0\). Yet \(Y\) is determined by \(X\): given \(X = 0\), \(Y\) is 0 for certain, although \(P(Y = 0) = 1/3\). The products of deviations, \((x - 0)(y - 2/3)\), are \(-1/3\), 0 and \(+1/3\) at \(x = -1\), 0 and 1: \(Y\) falls on the left and rises on the right, and the two halves cancel. A covariance, and so a correlation, measures straight-line association only; a relationship that bends can leave it at zero.

6. The variance of a sum

Total goals are home goals plus away goals, and a sample mean is a sum divided by \(n\), so we need the variance of a sum. For two random variables it is

In words: variance of a sum = the sum of the variances plus twice the covariance.

Intuitively, variables that move together reinforce each other's deviations, and their sum varies more; variables that move in opposite directions partly offset, and it varies less. The matches are the second case. Take them, as in sections 1 and 2, as the distribution, so that every variance and covariance divides by 1,900:

which is also the variance of the 1,900 totals worked out directly. Unrelated home and away goals would have given \(1.806 + 1.534 = 3.340\). The covariance is negative, so total goals vary less: a standard deviation of 1.68 goals rather than 1.83.

If \(X\) and \(Y\) are uncorrelated, and in particular if they are independent, the variances add. With more variables, the variance of a sum is the sum of the variances plus twice the covariance of every pair, and independence again makes every covariance zero. Lesson 6 uses two rules for the standard error, one from section 3 and one from the variance of a sum:

• A constant comes out of a variance squared: \(\operatorname{Var}(aX) = a^2\operatorname{Var}(X)\).
• The variance of a sum of independent random variables is the sum of their variances.

Under random sampling the observations \(X_1, \dots, X_n\) are independent draws from one distribution, each with variance \(\sigma^2\). So their sum has variance \(n\sigma^2\), and their mean \(\bar X\), the sum times \(1/n\), has variance \(n\sigma^2/n^2 = \sigma^2/n\); lesson 4 does this in full.

7. From the distribution to data: the sample versions

The mean \(\mu\), the variance \(\sigma^2\), the covariance and the correlation are fixed features of a distribution. From a sample of \(n\) observations, \(x_1, \dots, x_n\), we compute sample versions, which estimate them and vary from sample to sample:

In words: sample mean = sum of the observations divided by n; sample variance = sum of the squared deviations from the sample mean, divided by n - 1.

Its square root, \(s\), is the sample standard deviation. For pairs \((x_i, y_i)\), the sample covariance is the sum of the products of the deviations from \(\bar x\) and \(\bar y\), divided by \(n - 1\), and the sample correlation \(r\) divides it by the two sample standard deviations. The figures for the births, the workers and the clubs in sections 3 to 5 are of this kind.

Why divide by n - 1?

1. What we are after. \(\sigma^2\), the average squared distance of an observation from the population mean \(\mu\).
2. What we can measure. We do not know \(\mu\), so we measure distances from the sample mean \(\bar x\).
3. Why those distances run small. \(\bar x\) is the number that makes the sum of squared distances from these \(n\) observations as small as it can be. So in every sample the squared distances from \(\bar x\) add to no more than those from \(\mu\).
4. The shortfall. Under random sampling, their sum divided by \(n\) averages \((n - 1)\sigma^2/n\) across samples, not \(\sigma^2\).
5. The correction. Dividing by \(n - 1\) removes the shortfall: \(s^2\) averages \(\sigma^2\) for any \(n\) of 2 or more, so it is unbiased.

The count \(n - 1\) is called the degrees of freedom. The \(n\) deviations from \(\bar x\) add to zero, so once \(n - 1\) of them are known the last is fixed: one degree of freedom has gone into estimating the mean.

A simulation shows the shortfall. In 100,000 samples of 5 grades from a normal population with variance 100, dividing by \(n\) averaged 80.1 and dividing by \(n - 1\) averaged 100.1.

In the same simulation \(s\) itself averaged 9.41 at \(n = 5\), a little below \(\sigma = 10\), although \(s^2\) averaged 100.1; at \(n = 100\) it averaged 9.98.

The code below prints the sample versions. Sections 1 to 6 took the 1,900 matches as the whole distribution; here they are a sample from the matches that sides like these could play. They are not a random sample, and treating them as independent draws is an approximation, since each club plays 38 matches a season. Their summary numbers are these:

import delimited "premier-league-matches-2019-24.csv", clear
summarize home_goals away_goals
correlate home_goals away_goals
correlate home_goals away_goals, covariance

. summarize home_goals away_goals

    Variable |        Obs        Mean    Std. dev.       Min        Max
-------------+---------------------------------------------------------
  home_goals |      1,900    1.563158    1.344233          0          9
  away_goals |      1,900        1.31    1.238833          0          9

. correlate home_goals away_goals
(obs=1,900)

             | home_g~s away_g~s
-------------+------------------
  home_goals |   1.0000
  away_goals |  -0.1530   1.0000

. correlate home_goals away_goals, covariance
(obs=1,900)

             | home_g~s away_g~s
-------------+------------------
  home_goals |  1.80696
  away_goals | -.254713  1.53471

# home and away goals per match: mean, sd, covariance, correlation
pl <- read.csv("premier-league-matches-2019-24.csv")
goals <- pl[, c("home_goals", "away_goals")]
sapply(goals, mean)
sapply(goals, sd)
cov(goals)
cor(goals)

> # home and away goals per match: mean, sd, covariance, correlation
> pl <- read.csv("premier-league-matches-2019-24.csv")
> goals <- pl[, c("home_goals", "away_goals")]
> sapply(goals, mean)
home_goals away_goals 
  1.563158   1.310000 
> sapply(goals, sd)
home_goals away_goals 
  1.344233   1.238833 
> cov(goals)
           home_goals away_goals
home_goals   1.806962  -0.254713
away_goals  -0.254713   1.534708
> cor(goals)
           home_goals away_goals
home_goals   1.000000  -0.152955
away_goals  -0.152955   1.000000

import pandas as pd

# home and away goals per match: mean, sd, covariance, correlation
pl = pd.read_csv("premier-league-matches-2019-24.csv")
goals = pl[["home_goals", "away_goals"]]
print(goals.agg(["mean", "std"]))
print(goals.cov())
print(goals.corr())

      home_goals  away_goals
mean    1.563158    1.310000
std     1.344233    1.238833
            home_goals  away_goals
home_goals    1.806962   -0.254713
away_goals   -0.254713    1.534708
            home_goals  away_goals
home_goals    1.000000   -0.152955
away_goals   -0.152955    1.000000

The code reads premier-league-matches-2019-24.csv (1,900 matches: season, home, away, home_goals, away_goals, diff). Save it in your working folder first.

Stata, R and pandas divide by \(n - 1 = 1{,}899\), hence a covariance of -0.2547 here against -0.2546 in section 6, and a variance of home goals of 1.807 against 1.806. With 1,900 observations the divisor hardly matters; with five it does.

9. Check your understanding

Five questions, marked on the page; every option has an explanation, including the wrong ones. (Without JavaScript the explanations are shown beneath each question.)

10. Three questions about variance and correlation

11. Where next

Data and simulation. Matches: Premier League results from openfootball/england (CC0), seasons 2019/20 to 2023/24, 1,900 matches; the file the code reads is premier-league-matches-2019-24.csv. Clubs: the same source, final table of 2023/24. Wages: Wooldridge, Introductory Econometrics: A Modern Approach, data set WAGE1 (526 US workers, 1976). Births: the same book, data set BWGHT (1,388 births, 1988 National Health Interview Survey). Wooldridge results quoted, data not republished. Section 1, five samples of 10 matches: a fresh numpy default_rng(2026) and rng.choice(home_goals, size=(5, 10), replace=True), the home goals of the 1,900 matches in file order, one row per sample. Section 7, the divisor simulation: grades normal with mean 65 and standard deviation 10; for each n in 5, 25 and 100 a fresh numpy default_rng(2026) and rng.normal(65, 10, size=(100000, n)), filled row by row, one row per sample. The table of two 0/1 variables and the three-point example: illustrative. Adapted from my lecture notes for the MSc pre-sessional statistics course (ECON60901, 2015 and 2016) and for Time Series Econometrics (ECON30401) at the University of Manchester, and for Financial Econometrics (EC5609) at the University of St Andrews.