Type I & Type II Errors

Type I Error ($\alpha$): This occurs when the null hypothesis ($H_0$) is actually true, but the statistical test provides enough evidence to reject it. It is often referred to as a false positive, similar to a medical test indicating a disease is present when the patient is actually healthy.

Type II Error ($\beta$): This occurs when the null hypothesis ($H_0$) is false, but the test fails to provide sufficient evidence to reject it. This is known as a false negative, comparable to a medical test failing to detect a disease that a patient actually has.

Significance Level ($\alpha$): This is the pre-determined probability of committing a Type I error. By setting $\alpha = 0.05$, a researcher accepts a $5\%$ risk of rejecting a true null hypothesis.

Power of a Test ($1 - \beta$): This represents the probability of correctly rejecting a false null hypothesis. A high power indicates a high likelihood of detecting an effect if one truly exists.

The Inverse Relationship: For a fixed sample size, decreasing the probability of a Type I error ($\alpha$) inherently increases the probability of a Type II error ($\beta$). This occurs because moving the decision threshold to be more conservative makes it harder to reject the null, even when it is false.

Probability Distributions: Errors are visualized as areas under probability density functions. $\alpha$ is the area in the tail of the $H_0$ distribution, while $\beta$ is the area of the $H_a$ distribution that falls within the non-rejection region of $H_0$.

Effect of Sample Size: Increasing the sample size ($n$) narrows the distributions (reduces standard error), which allows both $\alpha$ and $\beta$ to decrease simultaneously. This is the primary method for improving the overall accuracy of a test.

Setting the Significance Level: Researchers must choose $\alpha$ based on the consequences of a false positive. In high-stakes scenarios like criminal trials, $\alpha$ is set very low (e.g., $0.01$) to avoid convicting the innocent.

Calculating Power: To find the probability of a Type II error, one must first define a specific alternative value for the parameter. $\beta$ is then calculated as $P(\text{Fail to Reject } H_0 | H_a \text{ is true})$.

Balancing Risks: A step-by-step approach involves: 1) Identifying the costs of each error type, 2) Selecting an appropriate $\alpha$, 3) Determining the required sample size to achieve a desired power ($1-\beta$).

Contextual Identification: When asked to describe an error in context, always use the format: "The test concludes [Alternative Hypothesis] when in reality [Null Hypothesis] is true." This ensures you address both the decision and the reality.

The 'Accept' Trap: Never use the word "accept" for the null hypothesis. If you fail to reject $H_0$, it does not mean $H_0$ is true; it simply means you lack evidence to prove it false. This is a common source of Type II error confusion.

Check the Relationship: If a question asks what happens to $\beta$ when $\alpha$ is decreased, the answer is always that $\beta$ increases (assuming $n$ is constant).

Power and Beta: Remember that $\text{Power} + \beta = 1$. If you are given the power of a test, you can immediately find the probability of a Type II error.

Misinterpreting $\alpha$: A common mistake is thinking $\alpha$ is the probability that the null hypothesis is true. In reality, $\alpha$ is a conditional probability: the probability of rejecting $H_0$ given that it is true.

Sum of Errors: Students often mistakenly believe that $\alpha + \beta = 1$. This is incorrect because they are probabilities calculated under two different, mutually exclusive assumptions (one assuming $H_0$ is true, the other assuming $H_a$ is true).

Ignoring Effect Size: A Type II error is more likely when the true parameter is very close to the null value. The smaller the "effect size," the harder it is for a test to distinguish between the two distributions.