Skip to main content

Estimating Statistics

1. Estimator​

1.1 Notations​

  • Let n∈N∗n\in\mathbb{N}^*

  • Let D(s)\mathcal{D}(s) be some distribution depending on ss

  • Let X1,…,Xn∼D(s)X_1,\dots,X_n\sim\mathcal{D}(s) be random variables

  • Let A\mathcal{A} be a vector space of real continuous random variables. If required, A\mathcal{A} should even be an associative algebra

1.2 Definition​

An estimator of s,s, denoted s^\hat{s} (when there is no confusion) is a function An→A\mathcal{A}^n\rightarrow \mathcal{A}

Informally, this function estimates ss from the observed data.

The estimator is said to be unbiased if:

∀X1,…,Xn∈A,E[s^(X1,…,Xn)]=s\forall X_1,\dots,X_n\in\mathcal{A},\quad\mathbb{E}[\hat{s}(X_1,\dots,X_n)]=s

2. Estimating mean​

The trivial mean estimator is:

μ^(X1,…,Xn)=1n∑i=1nXi\hat{\mu}(X_1,\dots,X_n)=\frac{1}{n}\sum_{i=1}^nX_i

We have:

E[μ^(X1,…,Xn)]=E[1n∑i=1nXi]=1n∑i=1nE[Xi]=μ\mathbb{E}[\hat{\mu}(X_1,\dots,X_n)]=\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^nX_i\right]=\frac{1}{n}\sum_{i=1}^n\mathbb{E}[X_i]=\mu

So μ^\hat{\mu} is unbiased. Furthermore:

V[μ^(X1,…,Xn)]=Cov[1n∑i=1nXi,1n∑i=1nXi]=1n2∑1≤i,j≤nCov[Xi,Xj]\begin{align*} \mathbb{V}[\hat{\mu}(X_1,\dots,X_n)]&=\text{Cov}\left[\frac{1}{n}\sum_{i=1}^nX_i,\frac{1}{n}\sum_{i=1}^nX_i\right]\\ &=\frac{1}{n^2}\sum_{1\leq i,j\leq n}\text{Cov}[X_i,X_j]\\ \end{align*}

Assuming that X1,…,XnX_1,\dots,X_n are independent, we have:

V[X]=1n2∑i=1nV[Xi]=σ2n\boxed{\mathbb{V}[X]=\frac{1}{n^2}\sum_{i=1}^n\mathbb{V}[X_i]=\frac{\sigma^2}{n}}

3. Estimating variance​

Assumption: X1,…,XnX_1,\dots,X_n are independent.

3.1 Trivial estimator σ2^\hat{\sigma^2}​

σ2^(X1,…,Xn)=1n∑i=1n(Xi−μ^(X1,…,Xn))2\hat{\sigma^2}(X_1,\dots,X_n)=\frac{1}{n}\sum_{i=1}^n\left(X_i-\hat{\mu}(X_1,\dots,X_n)\right)^2

the mean of this estimator is:

E[σ2^(X1,…,Xn)]=E[1n∑i=1n(Xi−μ^(X1,…,Xn))2]=1n∑i=1nE[(Xi−μ^(X1,…,Xn))2]=1n∑i=1nE[Xi2]−2E[Xiμ^(X1,…,Xn)]+E[μ^(X1,…,Xn)2]=1n∑i=1nE[Xi2]−2E[Xiμ^(X1,…,Xn)]+E[μ^(X1,…,Xn)2]=1n∑i=1nV[Xi]+E[Xi]2−2Cov[Xi,μ^(X1,…,Xn)]−2E[Xi]E[μ^(X1,…,Xn)]+V[μ^(X1,…,Xn)]+E[μ^(X1,…,Xn)]2=1n∑i=1nσ2+μ2−2σ2n−2μ2+σ2n+μ2=1n∑i=1nn−1nσ2=n−1nσ2\begin{align*} \mathbb{E}\left[\hat{\sigma^2}(X_1,\dots,X_n)\right]&=\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n\left(X_i-\hat{\mu}(X_1,\dots,X_n)\right)^2\right]\\ &=\frac{1}{n}\sum_{i=1}^n\mathbb{E}\left[\left(X_i-\hat{\mu}(X_1,\dots,X_n)\right)^2\right]\\ &=\frac{1}{n}\sum_{i=1}^n\mathbb{E}[X_i^2]-2\mathbb{E}\left[X_i\hat{\mu}(X_1,\dots,X_n)\right]+\mathbb{E}\left[\hat{\mu}(X_1,\dots,X_n)^2\right]\\ &=\frac{1}{n}\sum_{i=1}^n\mathbb{E}[X_i^2]-2\mathbb{E}\left[X_i\hat{\mu}(X_1,\dots,X_n)\right]+\mathbb{E}\left[\hat{\mu}(X_1,\dots,X_n)^2\right]\\ &=\frac{1}{n}\sum_{i=1}^n\mathbb{V}[X_i]+\mathbb{E}[X_i]^2-2\text{Cov}\left[X_i,\hat{\mu}(X_1,\dots,X_n)\right]-2\mathbb{E}[X_i]\mathbb{E}\left[\hat{\mu} (X_1,\dots,X_n)\right]\\&+\mathbb{V}\left[\hat{\mu}(X_1,\dots,X_n)\right]+\mathbb{E}\left[\hat{\mu}(X_1,\dots,X_n)\right]^2\\ &=\frac{1}{n}\sum_{i=1}^n\sigma^2+\mu^2-2\frac{\sigma^2}{n}-2\mu^2+\frac{\sigma^2}{n}+\mu^2\\ &=\frac{1}{n}\sum_{i=1}^n\frac{n-1}{n}\sigma^2\\ &=\frac{n-1}{n}\sigma^2 \end{align*}

Thus this estimator is biased.

3.2 Bessel's Correction: σ^∗2\hat{\sigma}_*^2​

This is an unbiased estimator of the variance:

σ^∗2(X1,…,Xn)=nn−1σ^2(X1,…,Xn)=1n−1∑i=1n(Xi−μ^(X1,…,Xn))2\hat{\sigma}_*^2(X_1,\dots,X_n)=\frac{n}{n-1}\hat{\sigma}^2(X_1,\dots,X_n)=\frac{1}{n-1}\sum_{i=1}^n\left(X_i-\hat\mu(X_1,\dots,X_n)\right)^2

3.3 God's Estimator​

This in an estimator depending on the prior knowledge of the mean:

σG2^(X1,…,Xn)=1n∑i=1n(Xi−μ)2\hat{\sigma^2_G}(X_1,\dots,X_n)=\frac{1}{n}\sum_{i=1}^n(X_i-\mu)^2

The expected value of this estimator is:

E[σG2^(X1,…,Xn)]=E[1n∑i=1n(Xi−μ)2]=1n∑i=1nE[Xi2]−2E[μXi]+E[μ2]=1n∑i=1nV[Xi]+E[Xi]2−2μE[Xi]+μ2=1n∑i=1nσ2=σ2\begin{align*}\mathbb{E}\left[\hat{\sigma^2_G}(X_1,\dots,X_n)\right]&=\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n\left(X_i-\mu\right)^2\right]\\ &=\frac{1}{n}\sum_{i=1}^n\mathbb{E}[X_i^2]-2\mathbb{E}[\mu X_i]+\mathbb{E}[\mu^2]\\ &=\frac{1}{n}\sum_{i=1}^n\mathbb{V}[X_i]+\mathbb{E}[X_i]^2-2\mu\mathbb{E}[ X_i]+\mu^2\\ &=\frac{1}{n}\sum_{i=1}^n\sigma^2\\ &=\sigma^2 \end{align*}

Thus this estimator is unbiased

4. Estimating Covariance​

Let Z1,…,ZnZ_1,\dots,Z_n be nn independent and identically distributed contintuos random real vectors with mean μ\mu and covariance matrix CC

4.1 Naive Estimator​

Cov^(Z1,…,Zn)=1n∑i=1n(Zi−μ^(Z1,…,Zn))(Zi−μ^(Z1,…,Zn))T\hat{\text{Cov}}\left(Z_1,\dots,Z_n\right)=\frac{1}{n}\sum_{i=1}^n(Z_i-\hat{\mu}(Z_1,\dots,Z_n))(Z_i-\hat{\mu}(Z_1,\dots,Z_n))^T

The expected value of this estimator is:

E[Cov^(Z1,…,Zn)]=1n∑i=1nE[(Zi−μ^(Z1,…,Zn))(Zi−μ^(Z1,…,Zn))T]=1n∑i=1nE[ZiZiT]−E[Ziμ^(Z1,…,Zn)T]−E[μ^(Z1,…,Zn)ZiT]+E[μ^(Z1,…,Zn)μ^(Z1,…,Zn)T]=1n(∑i=1nE[ZiZiT])−E[μ^(Z1,…,Zn)μ^(Z1,…,Zn)T]=1n(∑i=1nCov[Zi,Zi]+E[Zi]E[Zi]T)−E[1n2∑1≤i,j≤nZiZjT]=1n(∑i=1nC+μμT)−1n2∑1≤i,j≤nE[ZiZjT]=1n(∑i=1nC+μμT)−1n2∑1≤i≤nE[ZiZiT]−1n2∑1≤i≠j≤nE[Zi]E[Zj]T=1n∑i=1nC+μμT−1n2∑1≤i≤nC+μμT−1n2∑1≤i≠j≤nμμT=C+μμT−C+μμTn−n−1nμμT=n−1nC\begin{align*} \mathbb{E}[\hat{\text{Cov}}(Z_1,\dots,Z_n)]&=\frac{1}{n}\sum_{i=1}^n\mathbb{E}\left[\left(Z_i-\hat{\mu}(Z_1,\dots,Z_n)\right)\left(Z_i-\hat{\mu}(Z_1,\dots,Z_n)\right)^T\right]\\ &=\frac{1}{n}\sum_{i=1}^n\mathbb{E}\left[Z_iZ_i^T\right]-\mathbb{E}\left[Z_i\hat{\mu}(Z_1,\dots,Z_n)^T\right]-\mathbb{E}\left[\hat{\mu}(Z_1,\dots,Z_n)Z_i^T\right]\\&+\mathbb{E}\left[\hat{\mu}(Z_1,\dots,Z_n)\hat{\mu}(Z_1,\dots,Z_n)^T\right]\\ &=\frac{1}{n}\left(\sum_{i=1}^n\mathbb{E}[Z_iZ_i^T]\right)-\mathbb{E}\left[\hat{\mu}(Z_1,\dots,Z_n)\hat{\mu}(Z_1,\dots,Z_n)^T\right]\\ &=\frac{1}{n}\left(\sum_{i=1}^n\text{Cov}[Z_i,Z_i]+\mathbb{E}[Z_i]\mathbb{E}[Z_i]^T\right)-\mathbb{E}\left[\frac{1}{n^2}\sum_{1\leq i,j\leq n} Z_iZ_j^T\right]\\ &=\frac{1}{n}\left(\sum_{i=1}^n C+\mu\mu^T\right)-\frac{1}{n^2}\sum_{1\leq i,j\leq n}\mathbb{E}\left[ Z_iZ_j^T\right]\\ &=\frac{1}{n}\left(\sum_{i=1}^n C+\mu\mu^T\right)-\frac{1}{n^2}\sum_{1\leq i\leq n}\mathbb{E}\left[ Z_iZ_i^T\right]-\frac{1}{n^2}\sum_{1\leq i\neq j\leq n}\mathbb{E}\left[ Z_i\right]\mathbb{E}\left[Z_j\right]^T\\ &=\frac{1}{n}\sum_{i=1}^n C+\mu\mu^T-\frac{1}{n^2}\sum_{1\leq i\leq n}C+\mu\mu^T-\frac{1}{n^2}\sum_{1\leq i\neq j\leq n} \mu\mu^T\\ &=C+\mu\mu^T-\frac{C+\mu\mu^T}{n}-\frac{n-1}{n}\mu\mu^T\\ &=\frac{n-1}{n}C \end{align*}

Thus this estimator is biased.

4.2 Bessel's Correction​

This is the same correction as the sample variance's correction:

Cov∗^(Z1,…,Zn)=nn−1Cov^(Z1,…,Zn)=1n−1∑i=1n(Zi−μ^(Z1,…,Zn))⋅(Zi−μ^(Z1,…,Zn))T\hat{\text{Cov}_*}(Z_1,\dots,Z_n)=\frac{n}{n-1}\hat{\text{Cov}}(Z_1,\dots,Z_n)=\frac{1}{n-1}\sum_{i=1}^n(Z_i-\hat{\mu}(Z_1,\dots,Z_n))\cdot(Z_i-\hat{\mu}(Z_1,\dots,Z_n))^T

This estimator is an unbiased estimator of the covariance matrix

4.3 God's Estimator​

This in an estimator depending on the prior knowledge of the mean:

Cov^G(Z1,…,Zn)=1n∑i=1n(Zi−μ)(Zi−μ)T\hat{\text{Cov}}_G(Z_1,\dots,Z_n)=\frac{1}{n}\sum_{i=1}^n\left(Z_i-\mu\right)\left(Z_i-\mu\right)^T

Its expected values is:

E[Cov^G(Z1,…,Zn)]=1n∑i=1nE[(Zi−μ)(Zi−μ)T]=1n∑i=1nE[ZiZiT]−E[ZiμT]−E[μZiT]+E[μμT]=1n∑i=1nC+μμT−E[Zi]μT−μE[ZiT]+μμT=1n∑i=1nC+μμT−E[Zi]μT−μE[Zi]T+μμT=1n∑i=1nC+μμT−μμT−μμT+μμT=1n∑i=1nC=C\begin{align*} \mathbb{E}\left[\hat{\text{Cov}}_G(Z_1,\dots,Z_n)\right]&=\frac{1}{n}\sum_{i=1}^n\mathbb{E}\left[\left(Z_i-\mu\right)\left(Z_i-\mu\right)^T\right]\\ &=\frac{1}{n}\sum_{i=1}^n\mathbb{E}\left[Z_iZ_i^T\right]-\mathbb{E}\left[Z_i\mu^T\right]-\mathbb{E}\left[\mu Z_i^T\right]+\mathbb{E}[\mu\mu^T]\\ &=\frac{1}{n}\sum_{i=1}^n C+\mu\mu^T-\mathbb{E}\left[Z_i\right]\mu^T-\mu\mathbb{E}\left[ Z_i^T\right]+\mu\mu^T\\ &=\frac{1}{n}\sum_{i=1}^n C+\mu\mu^T-\mathbb{E}\left[Z_i\right]\mu^T-\mu\mathbb{E}\left[ Z_i\right]^T+\mu\mu^T\\ &=\frac{1}{n}\sum_{i=1}^n C+\mu\mu^T-\mu\mu^T-\mu\mu^T+\mu\mu^T\\ &=\frac{1}{n}\sum_{i=1}^n C\\ &= C \end{align*}