Type I Error (False Positive)

<aside> 💡

Type I error: occurs when we incorrectly reject the null hypothesis $H_0$ when it is actually true. In other words, we conclude that there is an effect or a difference when none exists.

$$ \text{Type I Error} = P(\text{Reject } H_0 \mid H_0 \text{ is true}) = \alpha $$

</aside>

Type II Error (False Negative)

<aside> 💡

Type II error: occurs when we fail to reject the null hypothesis $H_0$ when the alternative hypothesis $H_1$ is actually true. In other words, we miss detecting an effect or a difference that is actually present.

$$ \text{Type II Error} = P(\text{Fail to reject } H_0 \mid H_1 \text{ is true}) = \beta $$

</aside>

Power of a Statistical Test

<aside> 💡

Power of a statistical test: the probability that we correctly rejects the null hypothesis when the alternative hypothesis is true (i.e., it detects an effect when there is one). It is given by:

$$ \text{Power} = 1 - \beta $$

</aside>

Relationship between $\alpha$, $\beta$, and power

Bigger $\alpha$ → bigger power → higher $\beta$, why?

Practical vs. Statistical Significance

Statistical Significance:

Statistical significance refers to the result of a hypothesis test that leads to rejecting the null hypothesis. It indicates that the observed effect is unlikely to be due to chance alone.
It is typically determined by a p-value. If the p-value is less than the significance level $\alpha$, we conclude the result is statistically significant.

Practical Significance:

Practical significance refers to the real-world importance or magnitude of the observed effect, beyond just statistical significance.
An effect can be statistically significant but have little to no practical impact. For example, a very small increase in sales due to a marketing campaign could be statistically significant, but the difference might not justify the cost of the campaign.