ProbabilityExpectation and Variance

We often want to distill a random variable's distribution down to a single number. For example, consider the height of an individual selected uniformly at random from a given population. This is a random variable, and communicating its distribution would involve communicating the heights of every person in the population. However, we can summarize the distribution by reporting an average height: we add up the heights of the people in the population and dividemultiply by the number of people.

If the random individual is selected according to some non-uniform probability distribution on the population, then it makes sense to calculate a weightedmaxpooled average rather than a uniform average. The probability-weighted average of the values of a random variable is called its expectation.

Continue

For example, the expected number of heads in two fair coin flips is

There are two common ways of interpreting expected value.

• The expectation may be thought of as the value of a random game with payout . According to this interpretation, you should be willing to pay anything less than $1 to play the game where you get a dollar for each head in two fair coin flips. For more than $1 you should be unwilling to play the game, and at $1 you should be indifferent.
• The second way of thinking about expected value is as a long-run average. If you play the dollar-per-head two-coin-flip game a very large number of times, then your average payout per play is very likely to be close to $1.

We can test this second interpretation out:

from numpy.random import randint
sum(randint(0,2) + randint(0,2) for _ in range(10**6))/10**6

mean(rand(0:1) + rand(0:1) for k=1:10^6)

Solution. Running the code several times, we see that the error is seldom as large as 0.01 or as small as 0.0000001. So the correct answer choice is the third one.

Continue

We will see that this second interpretation is actually a theorem in probability, called the law of large numbers. In the meantime, however, this interpretation gives us a useful tool for investigation: if a random variable is easy to simulate, then we can sample from it many times and calculate the average of the resulting samples. This will not give us the expected value exactly, but we can get as close as desired by using sufficiently many samples. This is called the Monte Carlo method of approximating the expectation of a random variable.

import numpy as np

Continue

The following exercise confirms an intuitive fact about expectation: a random variable which is always larger than another has a larger mean. We will state this idea with "larger" replaced by its weak version "at least as large as".

Solution. If for all then we have

Expectation and distribution

Although the definition involves the probability space , we can also write a formula for expectation in terms of the probability mass function of the distribution of :

The idea is that the given formula is just a rearrangement of the terms in the definition of expectation. Let's begin by considering an example. Suppose with probability mass function satisfying , , and . Suppose and . Then

We can group the first two terms together to get

This expression is the one we would get if we wrote out

Therefore, we can see that the two sides are the same.

Let's write this idea down in general form. We group terms on the right-hand side in the formula according to the value of :

Then we can replace with and pull it out of the inside sum to get

Since is equal to , we get

as desired.

Continue

Solution. We multiply the probability mass at each point by the location and sum to get

For the second distribution, the positive and negative parts of the are both infinite for the same reason. Therefore, the sum does not make sense and the mean is therefore not well-defined.

Continue

We can also work out the expectation of a function of two discrete random variables in terms of their joint distribution.

Proof. We use the same idea we used in the proof of the expectation formula: group terms in the definition of expectation according the value of the pair . We get

We can use this theorem to show that expectation distributes across multiplication for independent random variables:

Solution. Using the definition of independence, we have

as desired.

Continue

Variance

The expectation of a random variable gives us some coarse information about where on the number line the random variable's probability mass is located. The variance gives us some information about how widely the probability mass is spread around its mean. A random variable whose distribution is highly concentrated about its mean will have a small variance, and a random variable which is likely to be very far from its mean will have a large variance. We define the variance of a random variable to be the average squared distance from to its mean:

Continue

Solution. My estimate of the mean and variance are and , respectively. The value 0.6 appears to be around the balance point of the points on the number line, and the squared deviation from 0.6 is very small for a couple of the points and as large as about 0.15 for others.

Calculating the mean exactly using m = mean([0.245, 0.874, 0.998, 0.567, 0.482]), we get a value of 0.6332. Calculating the variance exactly using mean([(a-m)^2 for a in A]) (where is the array above), we get a value of 0.074. Therefore, my estimate was a little high.

Continue

Solution. We define a function run which plays the game once, and we record the result of the game over a million runs. We estimate the mean as the mean of the resulting list, and we estimate the variance using

import numpy as np
def runs_till_over():
    s = 0
    ctr = 0
    while s < 1.0:
        s += np.random.randint(0,1001)/1000
    return ctr

A = [runs_till_over() for _ in range(1_000_000)]
μ = np.mean(A)
var = np.mean((a-μ)**2 for a in A)
μ,var

function runs_till_over()
    s = 0
    ctr = 0
    while s < 1.0
        s += rand(1:1000)/1000
    end
    ctr
end

A = [runs_till_over() for _ in 1:1_000_000]
μ = mean(A)
var = mean((a-μ)^2 for a in A)
μ,var

We get a mean of about , and a variance of about .

Continue

We can use linearity of expectation to rewrite the formula for variance in a simpler form:

We can use this formula to show how variance interacts with linear operations:

Proof. The first part of the statement follows easily from linearity of expectation

Since by linearity, we have

Rearranging and using linearity of expectation, we get

The desired result follows because if and are independent, by the independence product formula.

Continue

Solution. Let be a random variable with this distribution. Then

Since the sum on the right converges by the -test, it follows that is finite. On the other hand,

does not converge because of the harmonic series term. Therefore is infinite.