Community medicine chapters
Hypothesis testing, p-values, confidence intervals and errors
The steps of hypothesis testing, what the null hypothesis, p-value and confidence interval really mean, the standard error of the mean, and type I (alpha) and type II (beta) errors with power.
- We test the null hypothesis (H0): that there is no difference; if the result would be very unlikely under H0 (p < 0.05), we reject it.
- The standard error of the mean (SD ÷ √n) measures how precisely a sample mean estimates the true mean; the 95% confidence interval is roughly mean ± 1.96 SE.
- Type I (alpha) error: rejecting a true H0 (false positive). Type II (beta) error: failing to reject a false H0 (false negative). Power = 1 − beta.
On this page
Overview
We rarely study a whole population. We take a sample and use inferential statistics to decide whether what we see is likely to be real or could be due to chance (sampling variation).
The main tools are hypothesis testing (p-values) and estimation (confidence intervals).
The hypotheses
- Null hypothesis (H0): there is no difference between groups, or no association between variables. Example: "Drug A lowers BP no more than placebo."
- Alternative hypothesis (H1): there is a difference or association.
We never "prove" H0. We either reject it or fail to reject it.
Steps of hypothesis testing
- State the null and alternative hypotheses.
- Choose the level of significance (α), usually 0.05.
- Choose the right statistical test for the data (see statistical tests).
- Calculate the test statistic (for example z, t or chi-square) from the data.
- Find the p-value (or compare the statistic with the critical value from tables).
- Decide: if p < α, reject H0 (the result is statistically significant); if not, fail to reject H0.
- Interpret the result in clinical terms.
The p-value
The p-value is the probability of getting a result as extreme as, or more extreme than, the one observed, if the null hypothesis were true.
- p < 0.05: such a result would occur by chance less than 5% of the time if H0 were true → statistically significant; reject H0.
- p ≥ 0.05: not significant; the result could reasonably be due to chance.
p = 0.03 means: if there were truly no difference, a result at least this extreme would be seen 3% of the time by chance.
The p-value is not the probability that H0 is true, and it says nothing about the size or clinical importance of the effect. A tiny, unimportant difference can be "significant" in a very large study.
Standard error of the mean
If we took many samples, their means would vary. The standard error of the mean (SEM) is the SD of these sample means:
SE = SD ÷ √n
- It shows how precisely the sample mean estimates the population mean.
- It falls as sample size increases: larger samples give more precise estimates.
- The distribution of sample means is approximately normal even if the data are not, when n is large (central limit theorem).
Sample of n = 36 patients; SD of systolic BP = 12 mmHg
SE = SD ÷ √n = 12 ÷ √36 = 12 ÷ 6 = 2 mmHg
If n = 144: SE = 12 ÷ √144 = 12 ÷ 12 = 1 mmHg
(4 × the sample → ½ the SE)
Quadrupling the sample size halves the standard error.
Confidence intervals
A 95% confidence interval (CI) is a range that, if the study were repeated many times, would contain the true population value 95% of the time.
For a mean: 95% CI = sample mean ± 1.96 × SE.
Sample mean = 130 mmHg, SE = 2 mmHg
95% CI = mean ± 1.96 × SE
= 130 ± (1.96 × 2) = 130 ± 3.92
= 126.08 to 133.92 mmHg
A CI shows both the size of the effect and its precision:
- Narrow CI: precise estimate (large sample).
- Wide CI: imprecise estimate (small sample or very variable data).
Using a CI to judge significance
| Measure | "No effect" value | Significant at 5% if the 95% CI… |
|---|---|---|
| Difference between means or proportions | 0 | Does not include 0 |
| Ratio (relative risk, odds ratio) | 1 | Does not include 1 |
A relative risk of 1.4 with a 95% CI of 0.8 to 2.3 is not significant, because the interval includes 1.
Errors in hypothesis testing
| H0 is actually true (no real difference) | H0 is actually false (real difference) | |
|---|---|---|
| Reject H0 | Type I error (α): false positive | Correct (power) |
| Do not reject H0 | Correct | Type II error (β): false negative |
- Type I (alpha) error: concluding there is a difference when there is not. Its probability is the significance level, usually set at 5%.
- Type II (beta) error: missing a real difference. Commonly accepted at 10–20%.
- Power = 1 − β: the probability of detecting a real difference when it exists; usually aimed at 80% or 90%.
What increases power?
- Larger sample size (the most practical way).
- Larger true effect (bigger differences are easier to detect).
- Less variability in the data.
- A higher α (but this increases type I errors).
A small study that finds "no significant difference" may have made a type II error because it lacked power.
Doing many tests on the same data increases the chance of at least one type I error (false positive). Corrections such as Bonferroni are used.
A common memory aid: a type I error is "crying wolf" (seeing an effect that is not there); a type II error is missing the wolf.
One-tailed and two-tailed tests
- Two-tailed test: looks for a difference in either direction (A better or worse than B). The usual choice.
- One-tailed test: looks in one direction only. Used only when a difference in the other direction is impossible or irrelevant; it makes significance easier to reach and is viewed with suspicion.
Statistical vs clinical significance
Statistical significance tells you a result is unlikely to be due to chance. Clinical significance asks whether the size of the effect matters to patients. Always look at the effect size and its confidence interval, not just the p-value.
Expect: the definition of the null hypothesis and p-value, the steps of hypothesis testing, the formula and meaning of the standard error, how sample size affects SE and CI width, reading a CI for significance (0 for differences, 1 for ratios), type I vs type II errors, and the meaning of power.
Quick revision
- H0: no difference. Reject it if p < 0.05.
- p-value: chance of a result this extreme if H0 were true.
- SE = SD ÷ √n; larger n, smaller SE, narrower CI.
- 95% CI = mean ± 1.96 SE.
- CI including 0 (difference) or 1 (ratio): not significant.
- Type I (α): false positive. Type II (β): false negative. Power = 1 − β.
Written by The House Job from Excel Community Medicine and Park's Textbook of Preventive and Social Medicine (named in the UHS PGET syllabus).
FAQ
What is a p-value?
The probability of obtaining a result at least as extreme as the one observed if the null hypothesis were true.
What is a type I error?
Rejecting the null hypothesis when it is actually true: a false-positive finding.
What is the power of a study?
The probability of detecting a real difference when one exists; it equals 1 minus the type II error rate.
How do you know a relative risk is significant from its confidence interval?
If the 95% confidence interval does not include 1, the result is significant at the 5% level.