Type I and Type II Errors

Overview

In hypothesis testing, we can make two types of errors when drawing conclusions about a population based on sample data.

Error Types

Decision	$H_0$ True	$H_0$ False
Reject $H_0$	Type I Error ($\alpha$)	Correct Decision (Power)
Fail to Reject $H_0$	Correct Decision	Type II Error ($\beta$)

Type I Error ($\alpha$)

Definition

Rejecting $H_0$ when it is actually true.

Also Called

• False positive
• False alarm
• $\alpha$ error

Probability

$$P(\text{Type I Error}) = \alpha = \text{significance level}$$

Example

Concluding a drug works when it actually doesn't.

Type II Error ($\beta$)

Definition

Failing to reject $H_0$ when it is actually false.

Also Called

• False negative
• Missed detection
• $\beta$ error

Probability

$$P(\text{Type II Error}) = \beta$$

Example

Concluding a drug doesn't work when it actually does.

Power

Definition

The probability of correctly rejecting a false $H_0$.

$$\text{Power} = 1 - \beta = P(\text{Reject } H_0 \mid H_0 \text{ is false})$$

Desirable Values

• Power $\geq 0.80$ is common standard
• Higher power = better ability to detect effects

Relationships

$$\alpha \downarrow \Rightarrow \beta \uparrow \quad \text{(for fixed } n \text{)}$$ $$n \uparrow \Rightarrow \beta \downarrow \quad \text{(} \alpha \text{ constant)}$$ $$\text{Effect size} \uparrow \Rightarrow \beta \downarrow$$

Visual Representation

       H₀ true           H₁ true
       distribution      distribution
          ↓                   ↓
       ╭───╮              ╭───╮
      ╱     ╲            ╱     ╲
     ╱       ╲          ╱       ╲
────┴─────────┴────────┴─────────┴────
              │                 │
              │    Rejection    │
              │←── Region ─────→│
              Critical Value

Area under H₀ curve in rejection region = α
Area under H₁ curve NOT in rejection region = β

Tradeoff

Choice	Effect
Lower $\alpha$	Higher $\beta$ (less power)
Higher $\alpha$	Lower $\beta$ (more power)
Larger $n$	Lower $\beta$ (keeping $\alpha$ same)

Factors Affecting Power

Factor	Effect on Power
Larger sample size ($n$)	↑ Increases
Larger effect size	↑ Increases
Lower variance ($\sigma^2$)	↑ Increases
Higher $\alpha$	↑ Increases
One-tailed vs Two-tailed	One-tailed has more power

Examples

Example 1: Courtroom Analogy

$H_0$: Defendant is innocent

• Type I Error: Convicting an innocent person ($\alpha$)
• Type II Error: Acquitting a guilty person ($\beta$)

The justice system sets $\alpha$ very low ("beyond reasonable doubt").

Example 2: Medical Screening

$H_0$: Patient does not have disease

• Type I Error: False positive (unnecessary treatment)
• Type II Error: False negative (missed diagnosis)

Which error is worse depends on the disease and treatment.

Example 3: Quality Control

$H_0$: Product meets specifications

• Type I Error: Rejecting good products (waste)
• Type II Error: Accepting bad products (customer complaints)

Controlling Errors

To Reduce $\alpha$

• Lower significance level
• Tradeoff: increases $\beta$

To Reduce $\beta$ (Increase Power)

• Increase sample size
• Increase $\alpha$ (if acceptable)
• Reduce measurement error
• Focus on larger effect sizes

Power Analysis

Before conducting a study:

$$n = f(\alpha, \text{power}, \text{effect size}, \sigma)$$

Determines required sample size to detect a meaningful effect.

Practical Significance vs Statistical Significance

• Statistical significance: $p \leq \alpha$
• Practical significance: Effect is large enough to matter

A very large sample can detect statistically significant but practically unimportant effects.