Continuous Distributions 2 1. Normal Distribution 1.1 Definition A continuous random variable X X X μ \mu μ σ 2 \sigma^2 σ 2
f X ( x ) = 1 2 π e − ( x − μ ) 2 2 σ 2 f_X(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} f X ( x ) = 2 π 1 e − 2 σ 2 ( x − μ ) 2 1.2 Significance Normal distributions are important in statistics and are often used in the natural and social sciences to represent real-valued random variables whose distributions are not known.
Their importance is partly due to the central limit theorem . It states that, under some conditions, the average of many samples (observations) of a random variable with finite mean and variance is itself a random variable—whose distribution converges to a normal distribution as the number of samples increases. Therefore, physical quantities that are expected to be the sum of many independent processes, such as measurement errors , often have distributions that are nearly normal.
1.3 Standard Normal distribution It is the normal distribution with unit mean and unit variance:
S N = N ( 0 , 1 ) \mathcal{S}\mathcal{N}=\mathcal{N}(0,1) S N = N ( 0 , 1 ) 1.4.1 Opposite of standard normal random variable Let X ∼ N ( 0 , 1 ) X\sim\mathcal{N}(0,1) X ∼ N ( 0 , 1 )
∀ x ∈ R , F − X ( x ) = P ( − X < x ) = P ( X > − x ) = ∫ − x + ∞ 1 2 π e − t 2 2 dt = ∫ − ∞ x 1 2 π e − u 2 2 dt with u = − t = P ( X < x ) = F X ( x ) \begin{align*} \forall x\in\mathbb{R},F_{-X}(x)&=\mathcal{P}(-X<x)\\ &=\mathcal{P}(X>-x)\\ &=\int_{-x}^{+\infty}\frac{1}{\sqrt{2\pi}}e^{-\frac {t^2}{2}}\text{dt}\\ &=\int_{-\infty}^{x}\frac{1}{\sqrt{2\pi}}e^{-\frac{u^2}{2}}\text{dt} \text{ with }u=-t\\ &=\mathcal{P}(X<x)\\ &=F_X(x) \end{align*} ∀ x ∈ R , F − X ( x ) = P ( − X < x ) = P ( X > − x ) = ∫ − x + ∞ 2 π 1 e − 2 t 2 dt = ∫ − ∞ x 2 π 1 e − 2 u 2 dt with u = − t = P ( X < x ) = F X ( x ) As a conclusion:
X ∼ N ( 0 , 1 ) ⟹ − X ∼ N ( 0 , 1 ) X\sim\mathcal{N}(0,1)\implies -X\sim \mathcal{N}(0,1) X ∼ N ( 0 , 1 ) ⟹ − X ∼ N ( 0 , 1 ) Let a ∈ R + ∗ , b ∈ R , μ ∈ R , σ ∈ R + ∗ a\in\mathbb{R}^*_+,b\in\mathbb{R},\mu\in\mathbb{R},\sigma\in\mathbb{R}_+^* a ∈ R + ∗ , b ∈ R , μ ∈ R , σ ∈ R + ∗
Let X ∼ N ( μ , σ ) , Y = a X + b X\sim \mathcal{N}(\mu,\sigma),Y=aX+b X ∼ N ( μ , σ ) , Y = a X + b
∀ x ∈ R , F Y ( x ) = P ( Y < x ) = P ( a X < x − b ) = P ( X < x − b a ) = F X ( x − b a ) ⟹ ∀ x ∈ R , F Y ( x ) = 1 a F X ′ ( x − b a ) = 1 a f X ( x − b a ) = 1 2 π σ a e − ( x − b a − μ ) 2 2 σ 2 = 1 2 π σ a e − ( x − b − a μ ) 2 2 a 2 σ 2 ⟹ a X + b ∼ N ( a μ + b , a 2 σ 2 ) \begin{align*} \forall x\in\mathbb{R},F_Y(x)&=\mathcal{P}(Y< x)\\ &=\mathcal{P}(aX< x-b)\\ &= \mathcal{P}(X<\frac{x-b}{a})\\ &=F_X\left(\frac{x-b}{a}\right)\\ \implies \forall x\in\mathbb{R},F_Y(x)&=\frac{1}{a}F_X'\left(\frac{x-b}{a}\right)\\ &=\frac{1}{a}f_X(\frac{x-b}{a})\\ &=\frac{1}{\sqrt{2\pi}\sigma a}e^{-\frac{\left(\tfrac{x-b}{a}-\mu\right)^2}{2\sigma^2}}\\ &=\frac{1}{\sqrt{2\pi}\sigma a}e^{-\frac{\left(x-b-a\mu\right)^2}{2a^2\sigma^2}} \\ \implies &aX+b\sim\mathcal{N}(a\mu+b,a^2\sigma^2) \end{align*} ∀ x ∈ R , F Y ( x ) ⟹ ∀ x ∈ R , F Y ( x ) ⟹ = P ( Y < x ) = P ( a X < x − b ) = P ( X < a x − b ) = F X ( a x − b ) = a 1 F X ′ ( a x − b ) = a 1 f X ( a x − b ) = 2 π σa 1 e − 2 σ 2 ( a x − b − μ ) 2 = 2 π σa 1 e − 2 a 2 σ 2 ( x − b − a μ ) 2 a X + b ∼ N ( a μ + b , a 2 σ 2 ) In particular:
X ∼ N ( μ , σ 2 ) ⟺ X − μ σ = X − E [ X ] V [ X ] ∼ N ( 0 , 1 ) \boxed{X\sim\mathcal{N}(\mu,\sigma^2)\iff \frac{X-\mu}{\sigma}=\frac{X-\mathbb{E}[X]}{\sqrt{\mathbb{V}[X]}}\sim\mathcal{N}(0,1)} X ∼ N ( μ , σ 2 ) ⟺ σ X − μ = V [ X ] X − E [ X ] ∼ N ( 0 , 1 ) For a < 0 , a<0, a < 0 , X − μ σ ∼ N ( 0 , 1 ) \frac{X-\mu}{\sigma} \sim \mathcal{N}(0,1) σ X − μ ∼ N ( 0 , 1 ) − X − μ σ ∼ N ( 0 , 1 ) -\frac{X-\mu}{\sigma}\sim\mathcal{N}(0,1) − σ X − μ ∼ N ( 0 , 1 )
We have then, − X + μ ∼ N ( 0 , σ 2 ) ⟹ − X ∼ N ( − μ , σ 2 ) . -X+\mu\sim\mathcal{N}(0,\sigma^2)\implies -X\sim\mathcal{N}(-\mu,\sigma^2). − X + μ ∼ N ( 0 , σ 2 ) ⟹ − X ∼ N ( − μ , σ 2 ) .
Which implies the following:
a X + b = ( − a ) ( − X ) + b ∼ N ( ( − a ) ( − μ ) + b , ( − a ) 2 σ 2 ) = N ( a μ + b , a 2 σ 2 ) aX+b=(-a)(-X)+b\sim \mathcal{N}((-a)(-\mu)+b,(-a)^2\sigma^2)=\mathcal{N}(a\mu+b,a^2\sigma^2) a X + b = ( − a ) ( − X ) + b ∼ N (( − a ) ( − μ ) + b , ( − a ) 2 σ 2 ) = N ( a μ + b , a 2 σ 2 ) As a conclusion:
X ∼ N ( μ , σ ) ⟹ a x + b ∼ N ( a μ + b , a 2 σ 2 ) \boxed{X\sim \mathcal{N}(\mu,\sigma)\implies ax+b\sim\mathcal{N}\left(a\mu+b,a^2\sigma^2\right)} X ∼ N ( μ , σ ) ⟹ a x + b ∼ N ( a μ + b , a 2 σ 2 ) 1.5 Moments 1.5.1 Moment of a centered Normal distribution Let X ∼ U ( 0 , σ 2 ) , X\sim \mathcal{U}(0,\sigma^2), X ∼ U ( 0 , σ 2 ) ,
∀ n ∈ N ≥ 2 , E [ X n ] = ∫ R x n f X ( x ) dx = ∫ R 1 2 π σ x n e − x 2 2 σ 2 dx = ∫ R 1 2 π σ x n − 1 x e − x 2 2 σ 2 dx = [ ( ( n − 1 ) x n − 2 n 2 π σ ) × ( − σ 2 e − x 2 2 σ 2 ) ] − ∞ + ∞ − ∫ R ( ( n − 1 ) x n − 2 2 π σ ) × ( − σ 2 e − x 2 2 σ 2 ) dx = ( n − 1 ) σ 2 E [ X n − 2 ] ⟹ ∀ n ∈ N ≥ 2 , E [ X n ] = E [ X n m o d 2 ] ∏ k = 1 ⌊ n 2 ⌋ ( ( 2 k − 1 ) σ 2 ) = E [ X n m o d 2 ] σ 2 ⌊ n 2 ⌋ ∏ k = 1 ⌊ n 2 ⌋ ( 2 k − 1 ) ⟹ ∀ n ∈ N ∗ , E [ X 2 n ] = σ 2 n ∏ k = 1 n ( 2 k − 1 ) = σ 2 n ∏ k = 1 n 2 k ( 2 k − 1 ) ∏ k = 1 n 2 k = σ 2 n ⋅ ( 2 n ) ! 2 n n ! ∀ n ∈ N , E [ X 2 n + 1 ] = 0 because N ( 0 , σ 2 ) is symmetric \begin{align*} \forall n\in\mathbb{N}_{\ge 2},\quad \mathbb{E}[X^n] &=\int_{\mathbb{R}}x^{n}f_X(x)\space \text{dx}\\ &=\int_{\mathbb{R}}\frac{1}{\sqrt{2\pi}\sigma}x^{n}e^{-\frac{x^2}{2\sigma^2}}\space \text{dx}\\ &=\int_{\mathbb{R}}\frac{1}{\sqrt{2\pi}\sigma}x^{n-1}xe^{-\frac{x^2}{2\sigma^2}}\space \text{dx}\\ &=\left[\left(\frac{(n-1)x^{n-2}}{n\sqrt{2\pi}\sigma}\right)\times \left(-\sigma^2e^{-\frac{x^2}{2\sigma^2}}\right)\right]^{+\infty}_{-\infty}-\int_{\mathbb{R}}\left(\frac{(n-1)x^{n-2}}{\sqrt{2\pi}\sigma}\right)\times\left(-\sigma^2e^{-\frac{x^2}{2\sigma^2}}\right) \space \text{dx}\\ &=(n-1)\sigma^2\mathbb{E}[X^{n-2}]\\ \implies \forall n\in\mathbb{N}_{\ge 2},\quad \mathbb{E}[X^n]&=\mathbb{E}[X^{n \bmod 2}]\prod_{k=1}^{\lfloor\frac{n}{2}\rfloor}\big((2k-1)\sigma^2\big) \\ &= \mathbb{E}[X^{n \bmod 2}]\sigma^{2\lfloor\frac{n}{2}\rfloor}\prod_{k=1}^{\lfloor\frac{n}{2}\rfloor}(2k-1)\\ \implies \forall n\in\mathbb{N}^*,\quad \mathbb{E}[X^{2n}]&= \sigma^{2n}\prod_{k=1}^{n}(2k-1)\\ &=\sigma^{2n}\frac{\prod_{k=1}^{n}2k(2k-1)}{\prod_{k=1}^n2k}\\ &=\sigma^{2n}\cdot \frac{(2n)!}{2^nn!}\\ \forall n\in\mathbb{N},\mathbb{E}[X^{2n+1}]&=0 \quad \text{because} \space \mathcal{N}(0,\sigma^2) \space \text{is symmetric} \end{align*} ∀ n ∈ N ≥ 2 , E [ X n ] ⟹ ∀ n ∈ N ≥ 2 , E [ X n ] ⟹ ∀ n ∈ N ∗ , E [ X 2 n ] ∀ n ∈ N , E [ X 2 n + 1 ] = ∫ R x n f X ( x ) dx = ∫ R 2 π σ 1 x n e − 2 σ 2 x 2 dx = ∫ R 2 π σ 1 x n − 1 x e − 2 σ 2 x 2 dx = [ ( n 2 π σ ( n − 1 ) x n − 2 ) × ( − σ 2 e − 2 σ 2 x 2 ) ] − ∞ + ∞ − ∫ R ( 2 π σ ( n − 1 ) x n − 2 ) × ( − σ 2 e − 2 σ 2 x 2 ) dx = ( n − 1 ) σ 2 E [ X n − 2 ] = E [ X n mod 2 ] k = 1 ∏ ⌊ 2 n ⌋ ( ( 2 k − 1 ) σ 2 ) = E [ X n mod 2 ] σ 2 ⌊ 2 n ⌋ k = 1 ∏ ⌊ 2 n ⌋ ( 2 k − 1 ) = σ 2 n k = 1 ∏ n ( 2 k − 1 ) = σ 2 n ∏ k = 1 n 2 k ∏ k = 1 n 2 k ( 2 k − 1 ) = σ 2 n ⋅ 2 n n ! ( 2 n )! = 0 because N ( 0 , σ 2 ) is symmetric In particular, the expected value E [ X ] \mathbb{E}[X] E [ X ]
E [ X ] = 0 \boxed{\mathbb{E}[X]=0} E [ X ] = 0 Also, the variance:
V [ X ] = σ 2 \boxed{\mathbb{V}[X]=\sigma^2} V [ X ] = σ 2 1.5.2 Central Moments Let X ∼ N ( μ , σ 2 ) X\sim \mathcal{N}(\mu,\sigma^2) X ∼ N ( μ , σ 2 )
As E [ ( X − E [ X ] ) ] ∼ N ( 0 , σ 2 ) \mathbb{E}\left[\left(X-\mathbb{E}[X]\right)\right] \sim \mathcal{N}(0,\sigma^2) E [ ( X − E [ X ] ) ] ∼ N ( 0 , σ 2 )
∀ n ∈ N , { E [ ( X − E [ X ] ) 2 n ] = ( 2 n ) ! 2 n n ! σ 2 n E [ ( X − E [ X ] ) 2 n + 1 ] = 0 \forall n\in\mathbb{N}, \begin{cases} \mathbb{E}\left[\left(X-\mathbb{E}[X]\right)^{2n}\right]&= \frac{(2n)!}{2^nn!}\sigma^{2n}\\ \mathbb{E}\left[\left(X-\mathbb{E}[X]\right)^{2n+1}\right]&=0 \end{cases} ∀ n ∈ N , ⎩ ⎨ ⎧ E [ ( X − E [ X ] ) 2 n ] E [ ( X − E [ X ] ) 2 n + 1 ] = 2 n n ! ( 2 n )! σ 2 n = 0 1.5.3 Non-central moments Let X ∼ N ( μ , σ 2 ) X\sim \mathcal{N}(\mu,\sigma^2) X ∼ N ( μ , σ 2 )
∀ n ∈ N , E [ X 2 n ] = E [ ( X − E [ X ] + E [ X ] ) 2 n ] = ∑ k = 0 2 n ( 2 n k ) E [ X ] 2 n − k E [ ( X − E [ X ] ) k ] = ∑ k = 0 n ( 2 n 2 k ) E [ X ] 2 n − 2 k E [ ( X − E [ X ] ) 2 k ] = ∑ k = 0 n ( 2 n 2 k ) ( 2 k ) ! 2 k k ! μ 2 n − 2 k σ 2 k ∀ n ∈ N , E [ X 2 n + 1 ] = E [ ( X − E [ X ] + E [ X ] ) 2 n + 1 ] = ∑ k = 0 2 n + 1 ( 2 n + 1 k ) E [ X ] 2 n + 1 − k E [ ( X − E [ X ] ) k ] = ∑ k = 0 n ( 2 n + 1 2 k ) E [ X ] 2 n + 1 − 2 k E [ ( X − E [ X ] ) 2 k ] = ∑ k = 0 n ( 2 n + 1 2 k ) ( 2 k ) ! 2 k k ! μ 2 n + 1 − 2 k σ 2 k \begin{align*} \forall n\in\mathbb{N},\quad \mathbb{E}[X^{2n}]&=\mathbb{E}\left[\left(X-\mathbb{E}[X]+\mathbb{E}[X]\right)^{2n}\right] \\ &=\sum_{k=0}^{2n}{2n \choose k}\mathbb{E}[X]^{2n-k}\mathbb{E}\left[\left(X-\mathbb{E}[X]\right)^{k}\right] \\ &=\sum_{k=0}^{n}{2n \choose 2k}\mathbb{E}[X]^{2n-2k}\mathbb{E}\left[\left(X-\mathbb{E}[X]\right)^{2k}\right]\\ &=\sum_{k=0}^{n}{2n \choose 2k}\frac{(2k)!}{2^kk!}\mu^{2n-2k}\sigma^{2k} \\ \forall n\in\mathbb{N},\quad \mathbb{E}[X^{2n+1}]&=\mathbb{E}\left[\left(X-\mathbb{E}[X]+\mathbb{E}[X]\right)^{2n+1}\right] \\ &=\sum_{k=0}^{2n+1}{2n+1 \choose k}\mathbb{E}[X]^{2n+1-k}\mathbb{E}\left[\left(X-\mathbb{E}[X]\right)^{k}\right] \\ &=\sum_{k=0}^{n}{2n+1 \choose 2k}\mathbb{E}[X]^{2n+1-2k}\mathbb{E}\left[\left(X-\mathbb{E}[X]\right)^{2k}\right]\\ &=\sum_{k=0}^{n}{2n+1 \choose 2k}\frac{(2k)!}{2^kk!}\mu^{2n+1-2k}\sigma^{2k} \end{align*} ∀ n ∈ N , E [ X 2 n ] ∀ n ∈ N , E [ X 2 n + 1 ] = E [ ( X − E [ X ] + E [ X ] ) 2 n ] = k = 0 ∑ 2 n ( k 2 n ) E [ X ] 2 n − k E [ ( X − E [ X ] ) k ] = k = 0 ∑ n ( 2 k 2 n ) E [ X ] 2 n − 2 k E [ ( X − E [ X ] ) 2 k ] = k = 0 ∑ n ( 2 k 2 n ) 2 k k ! ( 2 k )! μ 2 n − 2 k σ 2 k = E [ ( X − E [ X ] + E [ X ] ) 2 n + 1 ] = k = 0 ∑ 2 n + 1 ( k 2 n + 1 ) E [ X ] 2 n + 1 − k E [ ( X − E [ X ] ) k ] = k = 0 ∑ n ( 2 k 2 n + 1 ) E [ X ] 2 n + 1 − 2 k E [ ( X − E [ X ] ) 2 k ] = k = 0 ∑ n ( 2 k 2 n + 1 ) 2 k k ! ( 2 k )! μ 2 n + 1 − 2 k σ 2 k 1.6 Sum of independent normal variables 1.6.1 Case of two centered normal variables Let X 1 ∼ N ( 0 , σ 1 2 ) , X 2 ∼ N ( 0 , σ 2 2 ) X_1\sim \mathcal{N}(0,\sigma_1^2),X_2\sim\mathcal{N}(0,\sigma_2^2) X 1 ∼ N ( 0 , σ 1 2 ) , X 2 ∼ N ( 0 , σ 2 2 ) Let Y = X 1 + X 2 Y=X_1+X_2 Y = X 1 + X 2 ∀ x ∈ R , f Y ( x ) = ∫ R f X 1 ( t ) f X 2 ( x − t ) dt = 1 2 π σ 1 σ 2 ∫ R e − ( t 2 2 σ 1 2 + ( x − t ) 2 2 σ 2 2 ) dt = 1 2 π σ 1 σ 2 ∫ R exp ( − 1 2 ( t 2 σ 1 2 + x 2 − 2 x t + t 2 σ 2 2 ) ) dt = e − x 2 2 σ 2 2 2 π σ 1 σ 2 ∫ R exp ( − 1 2 ( ( 1 σ 1 2 + 1 σ 2 2 ) t 2 − 2 x σ 2 2 t ) ) dt = e − x 2 2 σ 2 2 2 π σ 1 σ 2 ∫ R exp ( − 1 2 ( t 2 σ ∗ 2 − 2 x σ 2 2 t ) ) dtwith σ ∗ 2 = 1 1 σ 1 2 + 1 σ 2 2 = e − x 2 2 σ 2 2 2 π σ 1 σ 2 ∫ R exp ( − 1 2 σ ∗ 2 ( t 2 − 2 x σ ∗ 2 σ 2 2 t ) ) dt = e − x 2 2 σ 2 2 2 π σ 1 σ 2 ∫ R exp ( − 1 2 σ ∗ 2 ( ( t 2 − x σ ∗ 2 σ 2 2 ) 2 − x 2 σ ∗ 4 σ 2 4 ) ) dt = e − x 2 2 σ 2 2 + x 2 σ ∗ 2 σ 2 4 2 π σ 1 σ 2 ∫ R e − ( t 2 − x σ ∗ 2 σ 2 2 ) 2 2 σ ∗ 2 dt = 2 π σ ∗ e − x 2 2 σ 2 2 ( σ ∗ 2 σ 2 2 − 1 ) 2 π σ 1 σ 2 = e x 2 2 σ 2 2 ( 1 σ 2 2 ( 1 σ 1 2 + 1 σ 2 2 ) − 1 ) 2 π σ 1 σ 2 σ ∗ = e x 2 2 σ 2 2 ( 1 σ 2 2 σ 1 2 + 1 − 1 ) 2 π σ 1 2 σ 2 2 σ ∗ 2 = e − x 2 2 σ 2 2 ⋅ σ 2 2 σ 1 2 ( 1 σ 2 2 σ 1 2 + 1 ) 2 π σ 1 2 σ 2 2 ( 1 σ 1 2 + 1 σ 2 2 ) = e − x 2 2 ⋅ 1 σ 1 2 + σ 2 2 2 π ( σ 1 2 + σ 2 2 ) = e − x 2 2 ( σ 1 2 + σ 2 2 ) 2 π ⋅ σ 1 2 + σ 2 2 \begin{align*} \forall x\in\mathbb{R},f_Y(x)&=\int_{\mathbb{R}}f_{X_1}(t)f_{X_2}(x-t)\text{dt}\\ &=\frac{1}{2\pi\sigma_1\sigma_2}\int_{\mathbb{R}}e^{-\left(\frac{t^2}{2\sigma_1^2}+\frac{(x-t)^2}{2\sigma_2^2}\right)}\text{dt}\\ &=\frac{1}{2\pi\sigma_1\sigma_2}\int_{\mathbb{R}}\exp\left(-\frac{1}{2}\left(\frac{t^2}{\sigma_1^2}+\frac{x^2-2xt+t^2}{\sigma_2^2}\right)\right)\text{dt}\\ &=\frac{e^{-\frac{x^2}{2\sigma_2^2}}}{2\pi\sigma_1\sigma_2}\int_{\mathbb{R}}\exp\left(-\frac{1}{2}\left(\left(\frac{1}{\sigma^2_1}+\frac{1}{\sigma_2^2}\right)t^2-2\frac{x}{\sigma_2^2}t\right)\right)\text{dt}\\ &=\frac{e^{-\frac{x^2}{2\sigma_2^2}}}{2\pi\sigma_1\sigma_2}\int_{\mathbb{R}}\exp\left(-\frac{1}{2}\left(\frac{t^2}{\sigma_*^2}-2\frac{x}{\sigma_2^2}t\right)\right)\text{dt}\text{with }\sigma_*^2=\frac{1}{\tfrac{1}{\sigma_1^2}+\tfrac{1}{\sigma_2^2}}\\ &=\frac{e^{-\frac{x^2}{2\sigma_2^2}}}{2\pi\sigma_1\sigma_2}\int_{\mathbb{R}}\exp\left(-\frac{1}{2\sigma_*^2}\left(t^2-2\frac{x\sigma_*^2}{\sigma_2^2}t\right)\right)\text{dt}\\ &=\frac{e^{-\frac{x^2}{2\sigma_2^2}}}{2\pi\sigma_1\sigma_2}\int_{\mathbb{R}}\exp\left(-\frac{1}{2\sigma_*^2}\left(\left(t^2-\frac{x\sigma_*^2}{\sigma_2^2}\right)^2-\frac{x^2\sigma_*^4}{\sigma_2^4}\right)\right)\text{dt} \\ &=\frac{e^{-\frac{x^2}{2\sigma_2^2}+\frac{x^2\sigma_*^2}{\sigma_2^4}}}{2\pi\sigma_1\sigma_2}\int_{\mathbb{R}}e^{-\frac{\left(t^2-\frac{x\sigma_*^2}{\sigma_2^2}\right)^2}{2\sigma_*^2}}\text{dt}\\ &=\frac{\sqrt{2\pi}\sigma_*e^{-\frac{x^2}{2\sigma_2^2}\left(\frac{\sigma_*^2}{\sigma^2_2}-1\right)}}{2\pi\sigma_1\sigma_2}\\ &=\frac{e^{\frac{x^2}{2\sigma_2^2}\left(\frac{1}{\sigma^2_2(\tfrac{1}{\sigma_1^2}+\tfrac{1}{\sigma_2^2})}-1\right)}}{\sqrt{2\pi}\tfrac{\sigma_1\sigma_2}{\sigma_*}}\\ &=\frac{e^{\frac{x^2}{2\sigma_2^2}\left(\frac{1}{\tfrac{\sigma^2_2}{\sigma_1^2}+1}-1\right)}}{\sqrt{2\pi\tfrac{\sigma_1^2\sigma_2^2}{\sigma_*^2}}}\\ &=\frac{e^{-\frac{x^2}{2\sigma_2^2}\cdot\frac{\sigma^2_2}{\sigma_1^2}\left(\frac{1}{\tfrac{\sigma^2_2}{\sigma_1^2}+1}\right)}}{\sqrt{2\pi\sigma_1^2\sigma_2^2\left(\tfrac{1}{\sigma_1^2}+\tfrac{1}{\sigma_2^2}\right)}}\\ &=\frac{e^{-\frac{x^2}{2}\cdot\frac{1}{\sigma_1^2+\sigma_2^2}}}{\sqrt{2\pi\left(\sigma_1^2+\sigma_2^2\right)}}\\ &=\frac{e^{-\frac{x^2}{2(\sigma_1^2+\sigma_2^2)}}}{\sqrt{2\pi}\cdot\sqrt{\sigma_1^2+\sigma_2^2}} \end{align*} ∀ x ∈ R , f Y ( x ) = ∫ R f X 1 ( t ) f X 2 ( x − t ) dt = 2 π σ 1 σ 2 1 ∫ R e − ( 2 σ 1 2 t 2 + 2 σ 2 2 ( x − t ) 2 ) dt = 2 π σ 1 σ 2 1 ∫ R exp ( − 2 1 ( σ 1 2 t 2 + σ 2 2 x 2 − 2 x t + t 2 ) ) dt = 2 π σ 1 σ 2 e − 2 σ 2 2 x 2 ∫ R exp ( − 2 1 ( ( σ 1 2 1 + σ 2 2 1 ) t 2 − 2 σ 2 2 x t ) ) dt = 2 π σ 1 σ 2 e − 2 σ 2 2 x 2 ∫ R exp ( − 2 1 ( σ ∗ 2 t 2 − 2 σ 2 2 x t ) ) dt with σ ∗ 2 = σ 1 2 1 + σ 2 2 1 1 = 2 π σ 1 σ 2 e − 2 σ 2 2 x 2 ∫ R exp ( − 2 σ ∗ 2 1 ( t 2 − 2 σ 2 2 x σ ∗ 2 t ) ) dt = 2 π σ 1 σ 2 e − 2 σ 2 2 x 2 ∫ R exp ( − 2 σ ∗ 2 1 ( ( t 2 − σ 2 2 x σ ∗ 2 ) 2 − σ 2 4 x 2 σ ∗ 4 ) ) dt = 2 π σ 1 σ 2 e − 2 σ 2 2 x 2 + σ 2 4 x 2 σ ∗ 2 ∫ R e − 2 σ ∗ 2 ( t 2 − σ 2 2 x σ ∗ 2 ) 2 dt = 2 π σ 1 σ 2 2 π σ ∗ e − 2 σ 2 2 x 2 ( σ 2 2 σ ∗ 2 − 1 ) = 2 π σ ∗ σ 1 σ 2 e 2 σ 2 2 x 2 ( σ 2 2 ( σ 1 2 1 + σ 2 2 1 ) 1 − 1 ) = 2 π σ ∗ 2 σ 1 2 σ 2 2 e 2 σ 2 2 x 2 ( σ 1 2 σ 2 2 + 1 1 − 1 ) = 2 π σ 1 2 σ 2 2 ( σ 1 2 1 + σ 2 2 1 ) e − 2 σ 2 2 x 2 ⋅ σ 1 2 σ 2 2 ( σ 1 2 σ 2 2 + 1 1 ) = 2 π ( σ 1 2 + σ 2 2 ) e − 2 x 2 ⋅ σ 1 2 + σ 2 2 1 = 2 π ⋅ σ 1 2 + σ 2 2 e − 2 ( σ 1 2 + σ 2 2 ) x 2 Conclusion:
Y ∼ N ( 0 , σ 1 2 + σ 2 2 ) \boxed{Y\sim\mathcal{N}\left(0,\sigma_1^2+\sigma_2^2\right)} Y ∼ N ( 0 , σ 1 2 + σ 2 2 ) 1.6.2 Case of two independent normal variables Let X 1 ∼ N ( μ 1 , σ 1 2 ) , X 2 ∼ N ( μ 2 , σ 2 2 ) X_1\sim \mathcal{N}(\mu_1,\sigma_1^2),X_2\sim\mathcal{N}(\mu_2,\sigma_2^2) X 1 ∼ N ( μ 1 , σ 1 2 ) , X 2 ∼ N ( μ 2 , σ 2 2 ) Let Y = X 1 + X 2 Y=X_1+X_2 Y = X 1 + X 2 We have:
{ X 1 − μ 1 ∼ N ( 0 , σ 1 2 ) X 2 − μ 2 ∼ N ( 0 , σ 2 2 ) ⟹ ( X 1 − μ 1 ) + ( X 2 − μ 2 ) ∼ N ( 0 , σ 1 2 + σ 2 2 ) \begin{cases} X_1-\mu_1 \sim\mathcal{N}(0,\sigma_1^2)\\ X_2-\mu_2 \sim\mathcal{N}(0,\sigma_2^2) \end{cases} \implies (X_1-\mu_1)+(X_2-\mu_2)\sim\mathcal{N}\left(0,\sigma_1^2+\sigma_2^2\right) { X 1 − μ 1 ∼ N ( 0 , σ 1 2 ) X 2 − μ 2 ∼ N ( 0 , σ 2 2 ) ⟹ ( X 1 − μ 1 ) + ( X 2 − μ 2 ) ∼ N ( 0 , σ 1 2 + σ 2 2 ) So we can conclude that:
Y = X 1 + X 2 ∼ N ( μ 1 + μ 2 , σ 1 2 + σ 2 2 ) \boxed{Y=X_1+X_2\sim\mathcal{N}\left(\mu_1+\mu_2,\sigma_1^2+\sigma_2^2\right)} Y = X 1 + X 2 ∼ N ( μ 1 + μ 2 , σ 1 2 + σ 2 2 ) 1.6.3 General Case Let n ∈ N ∗ n\in\mathbb{N}^* n ∈ N ∗ Let X 1 ∼ N ( μ 1 , σ 1 2 ) , … , X n ∼ N ( μ n , σ n 2 ) X_1\sim\mathcal{N}(\mu_1,\sigma_1^2),\dots,X_n\sim\mathcal{N}(\mu_n,\sigma_n^2) X 1 ∼ N ( μ 1 , σ 1 2 ) , … , X n ∼ N ( μ n , σ n 2 ) n n n It can be trivially concluded from 1.5.2 1.5.2 1.5.2
∑ i = 1 n X i ∼ N ( ∑ i = 1 n μ i , ∑ i = 1 n σ i 2 ) \boxed{\sum_{i=1}^nX_i\sim\mathcal{N}\left(\sum_{i=1}^n\mu_i,\sum_{i=1}^n\sigma_i^2\right)} i = 1 ∑ n X i ∼ N ( i = 1 ∑ n μ i , i = 1 ∑ n σ i 2 ) 2. Γ \Gamma Γ 2.1 Definition Let α , β ∈ R + ∗ \alpha,\beta\in\mathbb{R}_+^* α , β ∈ R + ∗
Let X X X
By definition, X X X ( α , β ) (\alpha,\beta) ( α , β )
f X ( x ) = x α − 1 β α e − β x Γ ( α ) f_X(x)=\frac{x^{\alpha-1}\beta^\alpha e^{-\beta x}}{\Gamma(\alpha)} f X ( x ) = Γ ( α ) x α − 1 β α e − β x We denote it by:
X ∼ Γ ( α , β ) X\sim \Gamma(\alpha,\beta) X ∼ Γ ( α , β ) 2.2 Significance The gamma distribution has been used to model the size of insurance claims and rainfalls. This means that aggregate insurance claims and the amount of rainfall accumulated in a reservoir are modelled by a gamma process – much like the exponential distribution generates a Poisson process .
The gamma distribution is also used to model errors in multi-level Poisson regression models, because a mixture of Poisson distributions with gamma distributed rates has a known closed form distribution, called negative binomial .
In wireless communication, the gamma distribution is used to model the multi-path fading of signal power.
2.3 Exponential Distribution as a Gamma Distribution We have:
E ( λ ) = Γ ( 1 , λ ) \mathcal{E}(\lambda)=\Gamma(1,\lambda) E ( λ ) = Γ ( 1 , λ ) 2.4 Moments 2.4.1 Non-Central moments Let X ∼ Γ ( α , β ) X\sim \Gamma(\alpha,\beta) X ∼ Γ ( α , β )
∀ n ∈ N , E [ X n ] = ∫ R + x n f X ( x ) dx = ∫ R + x α + n − 1 β α e − β x Γ ( α ) dx = Γ ( α + n ) Γ ( α ) β n ∫ R + x α + n − 1 β α + n e − β x Γ ( α + n ) dx = Γ ( α + n ) Γ ( α ) β n = β − n ∏ i = 0 n − 1 α + i \begin{align*} \forall n\in\mathbb{N},\quad \mathbb{E}[X^n]&=\int_{\mathbb{R}_+}x^nf_X(x) \space \text{dx}\\ &=\int_{\mathbb{R}_+}\frac{x^{\alpha+n-1}\beta^\alpha e^{-\beta x}}{\Gamma(\alpha)} \space \text{dx}\\ &=\frac{\Gamma(\alpha+n)}{\Gamma(\alpha)\beta^n}\int_{\mathbb{R}_+}\frac{x^{\alpha+n-1}\beta^{\alpha+n} e^{-\beta x}}{\Gamma(\alpha+n)} \space \text{dx}\\ &=\frac{\Gamma(\alpha+n)}{\Gamma(\alpha)\beta^n}\\ &=\beta^{-n}\prod_{i=0}^{n-1}\alpha+i \end{align*} ∀ n ∈ N , E [ X n ] = ∫ R + x n f X ( x ) dx = ∫ R + Γ ( α ) x α + n − 1 β α e − β x dx = Γ ( α ) β n Γ ( α + n ) ∫ R + Γ ( α + n ) x α + n − 1 β α + n e − β x dx = Γ ( α ) β n Γ ( α + n ) = β − n i = 0 ∏ n − 1 α + i In particular, The expected value E [ X ] \mathbb{E}[X] E [ X ]
E [ X ] = α β \boxed{\mathbb{E}[X]=\frac{\alpha}{\beta}} E [ X ] = β α 2.4.2 Central Moments ∀ n ∈ N , E [ ( X − E [ X ] ) n ] = ∑ k = 0 n ( n k ) ( − 1 ) n − k E [ X k ] E [ X ] n − k = ∑ k = 0 n ( n k ) ( − 1 ) n − k α n − k Γ ( α + k ) β n Γ ( α ) \begin{align*} \forall n\in\mathbb{N},\quad \mathbb{E}\left[\left(X-\mathbb{E}[X]\right)^n\right]&= \sum_{k=0}^n{n \choose k}(-1)^{n-k}\mathbb{E}[X^k]\mathbb{E}[X]^{n-k}\\ &=\sum_{k=0}^n {n \choose k}(-1)^{n-k}\frac{\alpha^{n-k}\Gamma(\alpha+k)}{\beta^n\Gamma(\alpha)} \end{align*} ∀ n ∈ N , E [ ( X − E [ X ] ) n ] = k = 0 ∑ n ( k n ) ( − 1 ) n − k E [ X k ] E [ X ] n − k = k = 0 ∑ n ( k n ) ( − 1 ) n − k β n Γ ( α ) α n − k Γ ( α + k ) In particular, the variance V [ X ] \mathbb{V}[X] V [ X ]
V [ X ] = α 2 Γ ( α ) − 2 α Γ ( α + 1 ) + Γ ( α + 2 ) β 2 Γ ( α ) = α 2 − 2 α 2 + α ( α + 1 ) β 2 = α β 2 \boxed{\mathbb{V}[X]=\frac{\alpha^2\Gamma(\alpha)-2\alpha\Gamma(\alpha+1)+\Gamma(\alpha+2)}{\beta^2 \Gamma(\alpha)}=\frac{\alpha^2-2\alpha^2+\alpha(\alpha+1)}{\beta^2}=\frac{\alpha}{\beta^2}} V [ X ] = β 2 Γ ( α ) α 2 Γ ( α ) − 2 α Γ ( α + 1 ) + Γ ( α + 2 ) = β 2 α 2 − 2 α 2 + α ( α + 1 ) = β 2 α 2.5 Sum of gamma distributions 2.5.1 Two gamma distributions Let α 1 , α 2 , β ∈ R + ∗ \alpha_1,\alpha_2,\beta\in\mathbb{R}_+^* α 1 , α 2 , β ∈ R + ∗ Let X ∼ Γ ( α 1 , β ) , Y ∼ Γ ( α 2 , β ) , X\sim \Gamma(\alpha_1,\beta), Y\sim\Gamma(\alpha_2,\beta), X ∼ Γ ( α 1 , β ) , Y ∼ Γ ( α 2 , β ) , Z = X + Y Z=X+Y Z = X + Y ∀ x ∈ R + ∗ , f Z ( x ) = ∫ R f X ( t ) f Y ( x − t ) dt = ∫ 0 x f X ( t ) f Y ( x − t ) dt = ∫ 0 x t α 1 − 1 β α 1 e − β t Γ ( α 1 ) ⋅ ( x − t ) α 2 − 1 β α 2 e − β ( x − t ) Γ ( α 2 ) dt = β α 1 + α 2 e − β x Γ ( α 1 ) Γ ( α 2 ) ∫ 0 x t α 1 − 1 ( x − t ) α 2 − 1 dt = β α 1 + α 2 e − β x Γ ( α 1 ) Γ ( α 2 ) ∫ 0 1 ( x u ) α 1 − 1 ( x ( 1 − u ) ) α 2 − 1 x du with t = x u = β α 1 + α 2 x α 1 + α 2 − 1 e − β x Γ ( α 1 ) Γ ( α 2 ) ∫ 0 1 u α 1 − 1 ( 1 − u ) α 2 − 1 du = β α 1 + α 2 B ( α 1 , α 2 ) Γ ( α 1 ) Γ ( α 2 ) x α 1 + α 2 − 1 e − β x = β α 1 + α 2 x α 1 + α 2 − 1 e − β x Γ ( α 1 + α 2 ) because B ( α 1 , α 2 ) = Γ ( α 1 ) Γ ( α 2 ) Γ ( α 1 + α 2 ) ∀ x ∈ R − , f Z ( x ) = 0 \begin{align*} \forall x\in\mathbb{R}_+^*,f_Z(x)&=\int_{\mathbb{R}}f_X(t)f_Y(x-t)\text{dt}\\ &=\int_0^xf_X(t)f_Y(x-t)\text{dt}\\ &=\int_0^x\frac{t^{\alpha_1-1}\beta^{\alpha_1} e^{-\beta t}}{\Gamma(\alpha_1)}\cdot\frac{(x-t)^{\alpha_2-1}\beta^{\alpha_2} e^{-\beta (x-t)}}{\Gamma(\alpha_2)}\text{dt}\\ &=\frac{\beta^{\alpha_1+\alpha_2}e^{-\beta x}}{\Gamma(\alpha_1)\Gamma(\alpha_2)}\int_0^xt^{\alpha_1-1}(x-t)^{\alpha_2-1}\text{dt}\\ &=\frac{\beta^{\alpha_1+\alpha_2}e^{-\beta x}}{\Gamma(\alpha_1)\Gamma(\alpha_2)}\int_0^1(xu)^{\alpha_1-1}\left(x(1-u)\right)^{\alpha_2-1}x\space\text{du}\space \text{with }t=xu\\ &=\frac{\beta^{\alpha_1+\alpha_2}x^{\alpha_1+\alpha_2-1}e^{-\beta x}}{\Gamma(\alpha_1)\Gamma(\alpha_2)}\int_0^1u^{\alpha_1-1}\left(1-u\right)^{\alpha_2-1}\text{du}\\ &= \beta^{\alpha_1+\alpha_2}\frac{\Beta(\alpha_1,\alpha_2)}{\Gamma(\alpha_1)\Gamma(\alpha_2)}x^{\alpha_1+\alpha_2-1}e^{-\beta x}\\ &=\frac{\beta^{\alpha_1+\alpha_2}x^{\alpha_1+\alpha_2-1}e^{-\beta x}}{\Gamma(\alpha_1+\alpha_2)}\text{ because }\Beta(\alpha_1,\alpha_2)=\frac{\Gamma(\alpha_1)\Gamma(\alpha_2)}{\Gamma(\alpha_1+\alpha_2)}\\ \forall x\in\mathbb{R}_-,f_Z(x)&=0 \end{align*} ∀ x ∈ R + ∗ , f Z ( x ) ∀ x ∈ R − , f Z ( x ) = ∫ R f X ( t ) f Y ( x − t ) dt = ∫ 0 x f X ( t ) f Y ( x − t ) dt = ∫ 0 x Γ ( α 1 ) t α 1 − 1 β α 1 e − βt ⋅ Γ ( α 2 ) ( x − t ) α 2 − 1 β α 2 e − β ( x − t ) dt = Γ ( α 1 ) Γ ( α 2 ) β α 1 + α 2 e − β x ∫ 0 x t α 1 − 1 ( x − t ) α 2 − 1 dt = Γ ( α 1 ) Γ ( α 2 ) β α 1 + α 2 e − β x ∫ 0 1 ( xu ) α 1 − 1 ( x ( 1 − u ) ) α 2 − 1 x du with t = xu = Γ ( α 1 ) Γ ( α 2 ) β α 1 + α 2 x α 1 + α 2 − 1 e − β x ∫ 0 1 u α 1 − 1 ( 1 − u ) α 2 − 1 du = β α 1 + α 2 Γ ( α 1 ) Γ ( α 2 ) B ( α 1 , α 2 ) x α 1 + α 2 − 1 e − β x = Γ ( α 1 + α 2 ) β α 1 + α 2 x α 1 + α 2 − 1 e − β x because B ( α 1 , α 2 ) = Γ ( α 1 + α 2 ) Γ ( α 1 ) Γ ( α 2 ) = 0 So we can conclude that:
Z = X + Y ∼ Γ ( α 1 + α 2 , β ) \boxed{Z=X+Y\sim \Gamma(\alpha_1+\alpha_2,\beta)} Z = X + Y ∼ Γ ( α 1 + α 2 , β ) 2.5.2 General Case Let n ∈ N ∗ n\in\mathbb{N}^* n ∈ N ∗ Let X 1 ∼ Γ ( α 1 , β ) , … , X n ∼ Γ ( α n , β ) X_1\sim\Gamma(\alpha_1,\beta),\dots,X_n\sim\Gamma(\alpha_n,\beta) X 1 ∼ Γ ( α 1 , β ) , … , X n ∼ Γ ( α n , β ) n n n β \beta β It can be proved by induction that:
∑ i = 1 n X i ∼ Γ ( ∑ i = 1 n α i , β ) \boxed{\sum_{i=1}^nX_i\sim\Gamma\left(\sum_{i=1}^n\alpha_i,\beta\right)} i = 1 ∑ n X i ∼ Γ ( i = 1 ∑ n α i , β ) 2.6 Sum of Exponential distributions Let n ∈ N ∗ , λ ∈ R + ∗ n\in\mathbb{N}^*,\lambda\in\mathbb{R}_+^* n ∈ N ∗ , λ ∈ R + ∗ Let X 1 , … , X n ∼ E ( λ ) X_1,\dots,X_n\sim\mathcal{E}(\lambda) X 1 , … , X n ∼ E ( λ ) n n n λ \lambda λ This is a special case of 2.4 : 2.4: 2.4 :
∑ i = 1 n X i ∼ Γ ( n , λ ) \boxed{\sum_{i=1}^nX_i\sim\Gamma\left(n,\lambda\right)} i = 1 ∑ n X i ∼ Γ ( n , λ ) 2.7 Scaling of Gamma distributions Let k ∈ R + ∗ k\in\mathbb{R}_+^* k ∈ R + ∗ Let X ∼ Γ ( α , β ) X\sim \Gamma(\alpha,\beta) X ∼ Γ ( α , β ) Y = k X Y=kX Y = k X We have:
∀ x ∈ R + ∗ , f Y ( x ) = 1 k f ( x k ) = 1 k ⋅ β α ( x k ) α − 1 e − β k x Γ ( α ) = ( β k ) α x α − 1 e − β k x Γ ( α ) \begin{align*} \forall x\in\mathbb{R}_+^*, \quad f_Y(x)&=\frac{1}{k}f\left(\frac{x}{k}\right)\\ &=\frac{1}{k}\cdot \frac{\beta^\alpha\left(\frac{x}{k}\right)^{\alpha-1}e^{\frac{-\beta}{k}x}}{\Gamma(\alpha)}\\ &=\frac{\left(\frac{\beta}{k}\right)^{\alpha}x^{\alpha-1}e^{\frac{-\beta}{k}x}}{\Gamma(\alpha)} \end{align*} ∀ x ∈ R + ∗ , f Y ( x ) = k 1 f ( k x ) = k 1 ⋅ Γ ( α ) β α ( k x ) α − 1 e k − β x = Γ ( α ) ( k β ) α x α − 1 e k − β x So we have Y ∼ Γ ( α , β k ) Y\sim \Gamma(\alpha,\frac{\beta}{k}) Y ∼ Γ ( α , k β )
∀ k ∈ R + ∗ , X ∼ Γ ( α , β ) ⟺ k X ∼ Γ ( α , β k ) \boxed{\forall k\in\mathbb{R}_+^*,\quad X\sim\Gamma(\alpha,\beta)\iff kX\sim \Gamma\left(\alpha,\frac{\beta}{k}\right)} ∀ k ∈ R + ∗ , X ∼ Γ ( α , β ) ⟺ k X ∼ Γ ( α , k β )