Agenda

Distribution of the Maximum Likelihood Estimator (MLE)

When sample size $n$ is large, the distribution of $\hat{\theta}$ becomes approximately normal:

$$ \hat{\theta} \sim N \left( \theta, \frac{1}{n I(\theta)} \right) $$

where $I(\theta)$ is the Fisher information. This result us to use MLEs as unbiased, efficient estimators with minimal variance in large samples.

Recall, the Fisher Information is given by,

$$ I(\theta) = -\mathbb{E} \left[ \frac{\partial^2}{\partial \theta^2} \log L(\theta; X) \right]  $$

Cramér-Rao Lower Bound (CRLB)

<aside> 💡

The Cramér-Rao Lower Bound: provides a theoretical lower limit for the variance of any unbiased estimator $\hat{\theta}$ of a parameter $\theta$. According to the CRLB, for an unbiased estimator:

</aside>

$$ \text{Var}(\hat{\theta}) \geq \frac{1}{n I(\theta)}  $$

where $n$ is the sample size and $I(\theta)$ is the Fisher information.

Implications:

<aside> 💡

Efficient estimator: an estimator that achieves the CRLB, it has the smallest possible variance among unbiased estimators.

</aside>

• The MLE is often efficient in large samples, meaning it asymptotically achieves the CRLB.

Constructing a Confidence Interval for the MLE

To construct a confidence interval for an MLE $\hat{\theta}$, we use its approximate normal distribution in large samples:

$$ \hat{\theta} \sim N \left( \theta, \frac{1}{n I(\theta)} \right) $$

Steps to Construct a $1 - \alpha$ Confidence Interval: