[Statistics] Order Statistics

Order Statistics

Let $Y_1 \leq \cdots \leq Y_n$ denote the $n$ order statistics based on the i.i.d. random sample $X_1, \cdots, X_n$ from a pdf $f(x)$, cdf $F(x)$ and support $(a, b)$. That is, $Y_1, \cdots, Y_n$ represent $X_1, \cdots, X_n$ when the latter are arranged in ascending order. And, we are interesting in the PDF of them since they are involved in various statistics like the range and mid-range.

Joint PDF

\[\begin{align*} f_{Y_1, \cdots, Y_n} (y_1, \cdots, y_n) = n! \prod_{i=1}^n f(y_i) \cdot I(a < y_1 < \cdots < y_n < b) \end{align*}\]

$\mathbf{Proof.}$

The set of $(x_1, \cdots, x_n)$ can be mapped onto $(y_1, \cdots, y_n)$ where $a < y_1 < \cdots < y_n < b$ in total $n!$ ways by permutation. One of the possible set is $x_i = y_i$ for all $i$. Then, the Jacobian $J = 1$, and other set’s $J = \pm 1$ since it is just permuting rows. Thus,

\[\begin{align*} f_{Y_1, \cdots, Y_n} (y_1, \cdots, y_n) = \sum_{i=1}^{n!} | J_i | f(y_1) \cdots f(y_n)._\blacksquare \end{align*}\]

1D marginal PDF

\[\begin{align*} f_{Y_i} (y) = \frac{n!}{(i-1)! (n-i)!} F(y)^{i-1} f(y) (1 - F(y))^{n-i} \; (a < y < b) \end{align*}\]

$\mathbf{Proof.}$

Let $A \sim B(n, F(y))$ be the number of $Y_i$ such that $Y_i \leq y$.

\[\begin{align*} P(Y_k \leq y) = P( A \geq k ) = \sum_{l = k}^n \binom{n}{l} F(y)^l (1 - F(y))^{n-l} \end{align*}\]

Then,

\[\begin{align*} f_{Y_k} (y) &= \begin{align*} &\sum_{l = k}^n \binom{n}{l} l F^\prime (y) F(y)^{l-1} (1 - F(y))^{n-l} \\ &-\sum_{l = k}^n \binom{n}{l} (n-l) F^\prime (y) F(y)^{l} (1 - F(y))^{n-l - 1} \end{align*} \\ &= \begin{align*} &\sum_{l = k - 1}^{n-1} \frac{n!}{l! (n-l-1)!} F^\prime (y) F(y)^{l} (1 - F(y))^{n-l - 1} \\ &-\sum_{l = k}^{n-1} \binom{n}{l} (n-l) F^\prime (y) F(y)^{l} (1 - F(y))^{n-l - 1} \end{align*} \\ &= \frac{n!}{(k-1)! (n-k)!} F(y)^{k-1} (1 - F(y))^{n - k} f(y) \end{align*}\]

2D marginal PDF

For $i < j$,

\[\begin{align*} f_{Y_i, Y_j} (y_i, y_j) = \frac{n!}{(i-1)! (j - i - 1)! (n-j)!} F(y_i)^{i-1} (F(y_j) - F(y_i))^{j-i-1} (1 - F(y_j))^{n-j} f(y_i) f(y_j) \; (a < y_i \leq y_j < b) \end{align*}\]

$\mathbf{Proof.}$

We can derive the pdf of it like 1D marginal pdf (CDF technique), but there is a heuristic way for it: considering it as a assignment problem:

$\mathbf{Example.}$ Let $X_1, \cdots, X_n$ be i.i.d. samples of $\text{Exp}(1)$. Let $X_{(1)} < \cdots < X_{(n)}$ be order statistics. Then, show that

(1) \(\left\{\begin{array}{c}n X_{(1)} \\ (n-1)\left(X_{(2)}-X_{(1)}\right) \\ \vdots \\ (n-r+1)\left(X_{(r)}-X_{(r-1)}\right) \\ \vdots \\ \left(X_{(n)}-X_{(n-1)}\right)\end{array}\right\}\) i.i.d \(\operatorname{Exp}(1)\)
(2) \(\left(X_{(r)}\right)_{1 \leq r \leq n}=^d\left(\frac{1}{n} X_1+\cdots+\frac{1}{n-r+1} X_r\right)_{1 \leq r \leq n}\)

$\mathbf{Solution.}

Consider the transformation

\[\begin{align*} \left\{\begin{array}{c} Y_1 = n X_{(1)} \\ Y_2 (n-1)\left(X_{(2)}-X_{(1)}\right) \\ \vdots \\ Y_r (n-r+1)\left(X_{(r)}-X_{(r-1)}\right) \\ \vdots \\ Y_n = \left(X_{(n)}-X_{(n-1)}\right)\end{array}\right\} \end{align*}\]

Note that for all $1 \leq r \leq n$, the range of $Y_r$ is $(0, \infty)$. Then, each $X_{(r)}$ can be represented by $Y_r, \cdots, Y_1$:

\[\begin{align*} X_{(r)} = \frac{1}{n-r+1} Y_r + \cdots + \frac{1}{n} Y_1 \end{align*}\]

Then, the Jacobian is

\[\begin{aligned} J & =\operatorname{det}\left(\begin{array}{cccc} \frac{1}{n} & 0 & \cdots & 0 \\ \frac{1}{n} & \frac{1}{n-1} & \cdots & 0 \\ & & \ddots & \\ \frac{1}{n} & \frac{1}{n-1} & & 1 \end{array}\right) \\ & =\frac{1}{n !} \end{aligned}\]

Then, the joint PDF is $f_{Y_1, \cdots, Y_n} = e^{- \sum_{i=1}^n y_i} I_{(0, \infty)} (y_1, \cdots, y_n)$. And, the marginal PDF is clearly the PDF of $\text{Exp}(1)$, i.e., $Y_i \sim \text{Exp}(1)$.

For (2), since the PDF of $X_{(r)}$ is $f_r (x) = \frac{n!}{(r-1)! (n-r)!} e^{-x} (1 - e^{-x})^{r-1} (e^{-x})^{n - r}$, the CDF is

\[\begin{aligned} F_{X_{(r)}} (x) &= \int_{0}^x \frac{n!}{(r-1)! (n-r)!} e^{-x} (1 - e^{-x})^{r-1} (e^{-x})^{n - r} \cdot dx \\ - [\frac{n!}{(r-1)! (n-r + 1)!} (e^{-x})^{n - r} (1 - e^{-x})^{r-1}]_{0}^ \end{aligned}\]

Consider the transformation $Z_r = \frac{1}{n} X_1 + \cdots + \frac{1}{n-r+1} X_r$ for $1 \leq r \leq n$. Then, $X_r = (n - r + 1) (Z_r - Z_{r-1})$ for $1 \leq r \leq n$. Thus, the Jacobian is $n!$ clearly. The joint PDF of $Z_1, \cdots, Z_n$ is $n! e^{- \sum_{i=1}^n}$. Thus, the marginal PDF becomes

Reference

[1] Hogg, R., McKean, J. & Craig, A., Introduction to Mathematical Statistics, Pearson 2019