Cochran's theorem

In statistics, Cochran's theorem, devised by William G. Cochran,^[1] is a theorem used to justify results relating to the probability distributions of statistics that are used in the analysis of variance.^[2]

Examples

Sample mean and sample variance

If X₁, ..., X_n are independent normally distributed random variables with mean μ and standard deviation σ then

${\displaystyle U_{i}={\frac {X_{i}-\mu }{\sigma }}}$

is standard normal for each i. Note that the total Q is equal to sum of squared Us as shown here:

${\displaystyle \sum _{i}Q_{i}=\sum _{jik}U_{j}B_{jk}^{(i)}U_{k}=\sum _{jk}U_{j}U_{k}\sum _{i}B_{jk}^{(i)}=\sum _{jk}U_{j}U_{k}\delta _{jk}=\sum _{j}U_{j}^{2}}$

which stems from the original assumption that ${\displaystyle B_{1}+B_{2}\ldots =I}$. So instead we will calculate this quantity and later separate it into Q_i's. It is possible to write

${\displaystyle \sum _{i=1}^{n}U_{i}^{2}=\sum _{i=1}^{n}\left({\frac {X_{i}-{\overline {X}}}{\sigma }}\right)^{2}+n\left({\frac {{\overline {X}}-\mu }{\sigma }}\right)^{2}}$

(here ${\displaystyle {\overline {X}}}$ is the sample mean). To see this identity, multiply throughout by ${\displaystyle \sigma ^{2}}$ and note that

${\displaystyle \sum (X_{i}-\mu )^{2}=\sum (X_{i}-{\overline {X}}+{\overline {X}}-\mu )^{2}}$

and expand to give

${\displaystyle \sum (X_{i}-\mu )^{2}=\sum (X_{i}-{\overline {X}})^{2}+\sum ({\overline {X}}-\mu )^{2}+2\sum (X_{i}-{\overline {X}})({\overline {X}}-\mu ).}$

The third term is zero because it is equal to a constant times

${\displaystyle \sum ({\overline {X}}-X_{i})=0,}$

and the second term has just n identical terms added together. Thus

${\displaystyle \sum (X_{i}-\mu )^{2}=\sum (X_{i}-{\overline {X}})^{2}+n({\overline {X}}-\mu )^{2},}$

and hence

${\displaystyle \sum \left({\frac {X_{i}-\mu }{\sigma }}\right)^{2}=\sum \left({\frac {X_{i}-{\overline {X}}}{\sigma }}\right)^{2}+n\left({\frac {{\overline {X}}-\mu }{\sigma }}\right)^{2}=\overbrace {\sum _{i}\left(U_{i}-{\frac {1}{n}}\sum _{j}{U_{j}}\right)^{2}} ^{Q_{1}}+\overbrace {{\frac {1}{n}}\left(\sum _{j}{U_{j}}\right)^{2}} ^{Q_{2}}=Q_{1}+Q_{2}.}$

Now ${\displaystyle B^{(2)}={\frac {J_{n}}{n}}}$ with ${\displaystyle J_{n}}$ the matrix of ones which has rank 1. In turn ${\displaystyle B^{(1)}=I_{n}-{\frac {J_{n}}{n}}}$ given that ${\displaystyle I_{n}=B^{(1)}+B^{(2)}}$. This expression can be also obtained by expanding ${\displaystyle Q_{1}}$ in matrix notation. It can be shown that the rank of ${\displaystyle B^{(1)}}$ is ${\displaystyle n-1}$ as the addition of all its rows is equal to zero. Thus the conditions for Cochran's theorem are met.

Cochran's theorem then states that Q₁ and Q₂ are independent, with chi-squared distributions with n − 1 and 1 degree of freedom respectively. This shows that the sample mean and sample variance are independent. This can also be shown by Basu's theorem, and in fact this property characterizes the normal distribution – for no other distribution are the sample mean and sample variance independent.^[3]

Distributions

The result for the distributions is written symbolically as

${\displaystyle \sum \left(X_{i}-{\overline {X}}\right)^{2}\sim \sigma ^{2}\chi _{n-1}^{2}.}$

${\displaystyle n({\overline {X}}-\mu )^{2}\sim \sigma ^{2}\chi _{1}^{2},}$

Both these random variables are proportional to the true but unknown variance σ². Thus their ratio does not depend on σ² and, because they are statistically independent. The distribution of their ratio is given by

${\displaystyle {\frac {n\left({\overline {X}}-\mu \right)^{2}}{{\frac {1}{n-1}}\sum \left(X_{i}-{\overline {X}}\right)^{2}}}\sim {\frac {\chi _{1}^{2}}{{\frac {1}{n-1}}\chi _{n-1}^{2}}}\sim F_{1,n-1}}$

where F_1,n − 1 is the F-distribution with 1 and n − 1 degrees of freedom (see also Student's t-distribution). The final step here is effectively the definition of a random variable having the F-distribution.

Estimation of variance

To estimate the variance σ², one estimator that is sometimes used is the maximum likelihood estimator of the variance of a normal distribution

${\displaystyle {\widehat {\sigma }}^{2}={\frac {1}{n}}\sum \left(X_{i}-{\overline {X}}\right)^{2}.}$

Cochran's theorem shows that

${\displaystyle {\frac {n{\widehat {\sigma }}^{2}}{\sigma ^{2}}}\sim \chi _{n-1}^{2}}$

and the properties of the chi-squared distribution show that

${\displaystyle {\begin{aligned}E\left({\frac {n{\widehat {\sigma }}^{2}}{\sigma ^{2}}}\right)&=E\left(\chi _{n-1}^{2}\right)\\{\frac {n}{\sigma ^{2}}}E\left({\widehat {\sigma }}^{2}\right)&=(n-1)\\E\left({\widehat {\sigma }}^{2}\right)&={\frac {\sigma ^{2}(n-1)}{n}}\end{aligned}}}$

Alternative formulation

The following version is often seen when considering linear regression.^[4] Suppose that ${\displaystyle Y\sim N_{n}(0,\sigma ^{2}I_{n})}$ is a standard multivariate normal random vector (here ${\displaystyle I_{n}}$ denotes the n-by-n identity matrix), and if ${\displaystyle A_{1},\ldots ,A_{k}}$ are all n-by-n symmetric matrices with ${\displaystyle \sum _{i=1}^{k}A_{i}=I_{n}}$. Then, on defining ${\displaystyle r_{i}=\operatorname {Rank} (A_{i})}$, any one of the following conditions implies the other two:

• ${\displaystyle \sum _{i=1}^{k}r_{i}=n,}$
• ${\displaystyle Y^{T}A_{i}Y\sim \sigma ^{2}\chi _{r_{i}}^{2}}$ (thus the ${\displaystyle A_{i}}$ are positive semidefinite)
• ${\displaystyle Y^{T}A_{i}Y}$ is independent of ${\displaystyle Y^{T}A_{j}Y}$ for ${\displaystyle i\neq j.}$

Statement

Let U₁, ..., U_N be i.i.d. standard normally distributed random variables, and ${\displaystyle U=[U_{1},...,U_{N}]^{T}}$. Let ${\displaystyle B^{(1)},B^{(2)},\ldots ,B^{(k)}}$be symmetric matrices. Define r_i to be the rank of ${\displaystyle B^{(i)}}$. Define ${\displaystyle Q_{i}=U^{T}B^{(i)}U}$, so that the Q_i are quadratic forms. Further assume ${\displaystyle \sum _{i}Q_{i}=U^{T}U}$.

Cochran's theorem states that the following are equivalent:

• ${\displaystyle r_{1}+\cdots +r_{k}=N}$,
• the Q_i are independent
• each Q_i has a chi-squared distribution with r_i degrees of freedom.^[1][5]

Often it's stated as ${\displaystyle \sum _{i}A_{i}=A}$, where ${\displaystyle A}$ is idempotent, and ${\displaystyle \sum _{i}r_{i}=N}$ is replaced by ${\displaystyle \sum _{i}r_{i}=rank(A)}$. But after an orthogonal transform, ${\displaystyle A=diag(I_{M},0)}$, and so we reduce to the above theorem.

Proof

Claim: Let ${\displaystyle X}$ be a standard Gaussian in ${\displaystyle \mathbb {R} ^{n}}$, then for any symmetric matrices ${\displaystyle Q,Q'}$, if ${\displaystyle X^{T}QX}$ and ${\displaystyle X^{T}Q'X}$ have the same distribution, then ${\displaystyle Q,Q'}$ have the same eigenvalues (up to multiplicity).

Claim: ${\displaystyle I=\sum _{i}B_{i}}$.

Lemma: If ${\displaystyle \sum _{i}M_{i}=I}$, all ${\displaystyle M_{i}}$ symmetric, and have eigenvalues 0, 1, then they are simultaneously diagonalizable.

Now we prove the original theorem. We prove that the three cases are equivalent by proving that each case implies the next one in a cycle (${\displaystyle 1\to 2\to 3\to 1}$).

See also

• Cramér's theorem, on decomposing normal distribution
• Infinite divisibility (probability)

References