Sufficient Statistics

<aside> 💡 A statistic $T(X)$ is said to be a sufficient statistic for a parameter $\theta$ if the conditional distribution of the sample data $X$, given the statistic $T(X)$, does not depend on the parameter $\theta$.

</aside>

In other words, if I know the value of $T$, then knowing the actual observed data points does not provide any more information about the parameter value.

Example?

Properties

If $T(X)$ is sufficient for $\theta$, then all information about $\theta$ contained in the sample $X$ is captured by $T(X)$.
The concept of sufficiency helps reduce data complexity by summarizing information without loss of relevant details for estimation.

Example

For a random sample from a normal distribution $N(\mu, \sigma^2)$, the sample mean $\bar{X}$ is a sufficient statistic for $\mu$.

Factorization Theorem

<aside> 💡 Factorization Theorem: provides a method to determine if a statistic is sufficient. A statistic $T(X)$ is sufficient for $\theta$ if the likelihood function $L(\theta | X)$ can be factored as: $L(\theta | X) = g(T(X) | \theta) h(X)$ where:

$g(T(X) | \theta)$ depends only on the statistic $T(X)$ and the parameter $\theta$
$h(X)$ is a function that does not depend on $\theta$

</aside>

Example

For a Poisson distribution with parameter $\lambda$, the likelihood function can be expressed as:

$$ L(\lambda | X) = \frac{e^{-n\lambda} \lambda^{\sum X_i}}{\prod_{i=1}^{n} X_i!} $$

Here, $T(X) = \sum X_i$ is a sufficient statistic for $\lambda$.

Fisher Information

<aside> 💡 Fisher information $I(\theta)$ quantifies the amount of information that an observable random variable $X$ carries about the unknown parameter $\theta$. It is defined as:

$$ I(\theta) = E\left[ \left( \frac{\partial}{\partial \theta} \log L(\theta | X) \right)^2 \right] = -E\left[ \frac{\partial^2}{\partial \theta^2} \log L(\theta | X) \right] $$

</aside>