When interpreting the results of a hypothesis test, there are two types of error that we can make:
- TYPE I ERROR (i.e., a FALSE POSITIVE) occurs when one rejects the null hypothesis when it is true; in other words, the Type I error is saying that a relationship exists when, in fact, it does not exist
- The probability of committing a Type I error is equal to the significance level (α); for example, if α=0.05 (95% confidence level), there is a 5% chance of rejecting the null hypothesis when it is true
- TYPE II ERROR (i.e., a FALSE NEGATIVE) occurs when one fails to reject the null hypothesis when it is false; in other words, the Type II error is saying that no relationship exists when, in fact, a relationship does exist
- The probability of committing a Type II error is associated with several factors, including sample size, relationship strength/effect size, significance level (α), variability in the data, test design, and measurement precision
Which Error is Worse: Type I, or Type II?
Let’s consider how the judicial system is structured: would we rather convict an innocent man, or let a guilty man go free?
To convict someone of a crime, the prosecution must convince the jury beyond a reasonable doubt, and the jury verdict must be unanimous. Clearly, our judicial system is structured to make it harder to convict. The judicial system would rather let a guilty man go free than convict an innocent man. In other words, the judicial system seeks to avoid a Type I error, where it asserts a relationship exists when it does not (finding an innocent person “guilty”). Instead, the judicial system would rather commit a Type II error, failing to find a relationship when it does exist (finding a guilty person “not guilty”).
The same logic underlies statistical analyses. If we commit a Type I error, we are saying a relationship exists when, in fact, it does not. In inferential statistics, we always want to err on the side of caution. Therefore, a Type II error, where we fail to identify a real relationship, is generally more acceptable.