Sum of Random Variables

This lecture we study the expectation of the average of $n$ i.i.d. random variables $X_{i}$ , $i = 1, \dots, n$ , i.e., the expectation of $Q_{n} = \frac{1}{n} (X_{1} + \dots + X_{n})$ . As $n \to \infty$ , we will introduce the central limit theorem and show that $Q_{n}$ converges to a normal distribution provided $Var (X_{i})$ exists.

Sums of Discrete Random Variables

Suppose $X$ and $Y$ are two independent discrete random variables with distribution functions $p_{X} (x)$ and $p_{Y} (y)$ . Let $Z = X + Y$ , we want to find the the distribution function of $Z$ .

Suppose that $X = k$ , where $k$ is some integer. Then $Z = z$ if and only if $Y = z - k$ . So the event $Z = z$ is the union of the pairwise disjoint events $(X = k) and (Y = z - k)$ where $k$ runs over the integers. Since these events are pairwise disjoint, we have $\begin{matrix} (6-1) & P (Z = z) = \sum_{k = - \infty}^{\infty} P (X = k) P (Y = z - k) \end{matrix}$ which is the distribution function of the random variable $Z$ .

Definition. Let $X$ and $Y$ be two independent integer-valued random variables, with distribution functions $p_{X} (x)$ and $p_{Y} (y)$ respectively. Then the **convolution** of $p_{X} (x)$ and $p_{Y} (y)$ is the distribution function $p_{Z} = p_{X} * p_{Y}$ given by $p_{Z} (z) = \sum_{x} p_{X} (x) p_{Y} (z - x)$ for $x \in Z$ . The function $p_{Z} (z)$ is the distribution function of the random variable $Z = X + Y$ .

Sums of Continuous Random Variables

Definition. Let $X$ and $Y$ be two continuous random variables with density functions $f_{X} (x)$ and $f_{Y} (y)$ respectively. Assume that both $f_{X} (x)$ and $f_{Y} (y)$ are defined for all real numbers. Then the **convolution** $f * g$ of $f$ and $g$ is the function given by $\begin{matrix} (6-2) & (f_{X} * f_{Y}) (z) = \int_{- \infty}^{+ \infty} f_{X} (z - y) f_{Y} (y) d y = \int_{- \infty}^{+ \infty} f_{Y} (z - x) f_{X} (x) d x . \end{matrix}$

Theorem. Let $X$ and $Y$ be two independent random variables with density functions $f_{X} (x)$ and $f_{Y} (y)$ defined for all $x$ and $y$ . Then the sum $Z = X + Y$ is a random variable with density function $f_{Z} (z)$ , where $f_{Z}$ is the convolution of $f_{X}$ and $f_{Y}$ .

Theorem. Let ${X_{i}}$ , $i = 1, \dots, n$ be a sequence of independent random variables with density functions $f_{X_{1}} (x), \dots, f_{X_{n}} (x)$ respectively, then we have $f_{X_{1} + \dots + X_{n}} (x) = f_{X_{1}} * (f_{X_{2}} * (\dots * f_{X_{n}})) (x) .$

Example. [Sum of two independent uniform random variables] Let $X$ and $Y$ be random variables describing our choices and $Z = X + Y$ their sum. Then we have $f_{X} (x) = f_{Y} (x) = {\begin{cases} 1 & if 0 \leq x \leq 1 \\ 0 & otherwise \end{cases}$ and the density function for the sum is given by $f_{Z} (z) = \int_{- \infty}^{+ \infty} f_{X} (z - y) f_{Y} (y) d y = \int_{0}^{1} f_{X} (z - y) d y .$ Now the integrand is $0$ unless $0 \leq z - y \leq 1$ and then it is $1$ . So if $0 \leq z \leq 1$ , we have $f_{Z} (z) = \int_{0}^{z} d y = z,$ while if $1 < z \leq 2$ , we have $f_{Z} (z) = \int_{z - 1}^{1} d y = 2 - z,$ and if $z < 0$ or $z > 2$ we have $f_{Z} (z) = 0$ . Hence $f_{Z} (z) = {\begin{cases} z, & if 0 \leq z \leq 1 \\ 2 - z, & if 1 < z \leq 2 \\ 0, & otherwise \end{cases}$

Example. [Sum of two independent exponential random variables] Let $X, Y$ , and $Z = X + Y$ denote the relevant random variables, and $f_{X}$ , $f_{Y}$ , and $f_{Z}$ their densities. Then $f_{X} (x) = f_{Y} (x) = {\begin{cases} λ e^{- λ x}, & if x \geq 0 \\ 0, & otherwise \end{cases}$ If $z > 0$ , $f_{Z} (z) = \int_{- \infty}^{+ \infty} f_{X} (z - y) f_{Y} (y) d y = \int_{0}^{z} λ e^{- λ (z - y)} λ e^{- λ y} d y = \int_{0}^{z} λ^{2} e^{- λ z} d y = λ^{2} z e^{- λ z},$ while if $z < 0$ , $f_{Z} (z) = 0$ . Hence $f_{Z} (z) = {\begin{cases} λ^{2} z e^{- λ z}, & if z \geq 0 \\ 0, & otherwise \end{cases}$