Hypothesis testing
Hypothesis testing: p values, errors, statistical tests, correlation and regression
Hypothesis tests ask whether an observed difference could reasonably be due to chance. The choice of test depends on data type, number of groups, pairing and distribution, and correlation and regression describe relationships between variables. Survival analysis handles time-to-event data.
Hypothesis testing
- Null hypothesis (H₀): no difference/association. Alternative (H₁): there is one.
- p value: probability of obtaining the observed result (or more extreme) if H₀ were true. p < 0.05: reject H₀ ("statistically significant"). It is not the probability that H₀ is true, and says nothing about clinical importance.
- Statistical significance ≠ clinical significance (large studies detect trivial differences).
- One-tailed versus two-tailed tests (two-tailed is standard).
- Multiple comparisons increase type I error, so use Bonferroni correction (α/number of tests).
Errors
| H₀ true | H₀ false | |
|---|---|---|
| Reject H₀ | Type I error (α) — false positive | Correct (power, 1 − β) |
| Do not reject H₀ | Correct | Type II error (β) — false negative |
- Type I: concluding a drug works when it does not. Type II: missing a real effect (small samples).
Choosing a statistical test
| Data / comparison | Parametric (normal, continuous) | Non-parametric (skewed, ordinal, small samples) |
|---|---|---|
| Two independent groups | Unpaired (independent) t-test | Mann–Whitney U (Wilcoxon rank-sum) |
| Two paired/related measurements (before–after, matched) | Paired t-test | Wilcoxon signed-rank |
| Three or more independent groups | One-way ANOVA (F test; post hoc tests) | Kruskal–Wallis |
| Three or more related measurements | Repeated measures ANOVA | Friedman test |
| Correlation between two variables | Pearson's r | Spearman's rank (ρ), Kendall's tau |
| One sample mean versus known value | One-sample t-test (z-test if large sample/known SD) | Sign test |
| Categorical data | Test |
|---|---|
| Two or more independent proportions (contingency table) | Chi-squared (χ²) test |
| Small expected frequencies (< 5 in > 20% of cells) | Fisher's exact test |
| Paired categorical data (before–after on same subjects) | McNemar's test |
| Trend across ordered groups | Chi-squared for trend |
| Stratified 2 × 2 tables (confounding) | Mantel–Haenszel |
- Parametric tests compare means; non-parametric tests use ranks (compare medians/distributions); parametric tests are more powerful when assumptions hold.
Correlation
- Pearson's correlation coefficient (r): strength and direction of linear relationship between two continuous normally distributed variables; −1 to +1; 0 = no linear relationship.
- r = +1 perfect positive; −1 perfect negative; |r| > 0.7 strong.
- r² (coefficient of determination): proportion of variance in one variable explained by the other (r = 0.8 gives 64%).
- Correlation does not imply causation; sensitive to outliers; may miss non-linear relationships.
- Spearman's ρ: ranks; non-normal or ordinal data, monotonic relationships.
Regression
- Simple linear regression: predicts outcome (dependent, y) from one predictor (independent, x): y = a + bx (a = intercept, b = slope/regression coefficient — change in y per unit change in x).
- Multiple (multivariable) linear regression: several predictors; adjusts for confounders.
- Logistic regression: binary outcome (disease yes/no) gives adjusted odds ratios.
- Cox proportional hazards regression: time-to-event outcome gives hazard ratios.
- Poisson regression: count outcomes/rates.
Survival analysis
- Kaplan–Meier method: survival curve accounting for censored data (lost to follow-up, alive at study end); step function.
- Log-rank test: compares survival curves between groups.
- Median survival: time at which 50% have died.
- Hazard ratio from Cox regression.
Other
- Bland–Altman plot: agreement between two measurement methods (not correlation).
- Kappa: agreement for categorical ratings (see validity and reliability).
- Analysis of covariance (ANCOVA).
How FCPS asks it
Recalls test: meaning of p value; type I and II errors; matching scenario to test (paired t-test for before–after BP; unpaired t-test for two group means; ANOVA for three groups; chi-squared for proportions; Fisher's for small numbers; McNemar for paired proportions; Mann–Whitney for skewed data; Wilcoxon signed-rank for paired skewed data; Kruskal–Wallis); Pearson versus Spearman; r²; regression types by outcome; Kaplan–Meier and log-rank; censoring.
Exam trap Comparing means of two independent groups: unpaired t-test; same subjects before and after: paired t-test.
Exam trap Chi-squared compares proportions; use Fisher's exact when expected counts are small.
Exam trap Logistic regression is used for a binary outcome and gives adjusted odds ratios.
Test yourself on this topic
5 of the 27 questions in our bank on this topic. Choose an answer to see why every option is right or wrong.
A trial is designed with 80% power. What does this mean?
Answer: 80% chance to detect a true effect
Power = 1 − β, where β is the probability of a type II error (missing a real effect). Conventionally power is set at 80% (β = 20%) or 90%, with α (type I error) at 5%. Sample size depends on α, power, the smallest clinically important difference to detect, variability (SD) of the outcome, and expected dropout. Underpowered studies often produce false-negative results.
Power
- Power = 1 − β
- Usually 80% (β = 0.2), α = 0.05
- Sample size: α, power, effect size, SD, dropout
- Underpowered → type II error
Remember Power is the chance of catching a real effect.
Why the other options are wrong
- 80% chance the result is correct Not the definition.
- Significance level is 0.80 α is usually 0.05.
- 20% of patients will benefit Not related.
- The p value will be below 0.80 Not the meaning.
Researchers want to know only whether a new drug lowers cholesterol more than the standard, not whether it might be worse. Which test is this, and what is its drawback?
Answer: One-tailed; one direction only
A two-tailed test looks for a difference in either direction (new drug better or worse); the 5% significance level is split, 2.5% in each tail. A one-tailed test places all 5% in one direction, so it reaches significance more easily for an effect in the predicted direction but cannot detect an effect in the other direction, which might be harmful. Because a new treatment could be worse, two-tailed tests are the standard in clinical research; one-tailed tests are justified only when an effect in the other direction is impossible or irrelevant and must be specified in advance (e.g. some non-inferiority designs).
One- vs two-tailed tests
- Two-tailed: difference either direction (standard)
- One-tailed: one direction only; easier significance
- Drawback: misses harm in opposite direction
- Must be pre-specified
Remember Two tails watch both ways; one tail looks only where you hope.
Why the other options are wrong
- Two-tailed; smaller sample Two-tailed tests detect either direction.
- Non-parametric; ignores mean Not the question.
- Chi-square; proportions only Not the question.
- Paired; same subjects Not the question.
A 2 × 2 table has an expected cell count below 5. Which test should replace the chi-square test?
Answer: Fisher's exact test
The chi-square test compares proportions between groups but is unreliable when expected counts are small (any cell < 5 in a 2 × 2 table). Fisher's exact test calculates the exact probability. McNemar's test is for paired categorical data (before–after in the same people).
Categorical tests
- Unpaired proportions: chi-square
- Small expected counts: Fisher's exact
- Paired proportions: McNemar
Remember Small numbers? Fisher's exact.
Why the other options are wrong
- Paired t test For paired continuous data.
- McNemar's test always For paired categorical data.
- ANOVA For comparing means of 3+ groups.
- Pearson correlation For association of continuous variables.
Which statistical test is used to compare proportions in a 2 × 2 table of categorical data?
Answer: Chi-square test
The chi-square test compares observed and expected frequencies in categories, such as smokers versus non-smokers with and without cancer. With very small expected numbers (under 5), Fisher's exact test is used instead.
Which test?
- Categories (counts) → chi-square (small numbers: Fisher's)
- Two means → t-test; ≥3 means → ANOVA
- Two continuous variables → correlation / regression
Remember Chi for Counts; T for Two means.
Why the other options are wrong
- Student's t-test Compares two means of continuous data.
- Pearson correlation Measures the strength of a linear relationship between two continuous variables.
- ANOVA Compares three or more means.
- Paired t-test Compares two means in the same subjects (before and after).
A small trial reports that a herbal remedy significantly reduces nausea in pregnancy (p = 0.04). Several larger, well-conducted trials later find no effect at all. If the remedy truly has no effect, which error did the first trial make?
Answer: Type I (alpha) error
A type I error is rejecting a null hypothesis that is actually true, that is, finding a 'significant' difference when none exists. Its probability is set by the significance level (alpha), usually 5%, so about 1 in 20 tests of an ineffective treatment will appear significant by chance. A type II error is failing to detect a real effect, which is related to low power.
Type I error
- False positive result
- Rejecting a true null hypothesis
- Probability = alpha (often 0.05)
- Type II = missing a real effect
Remember Type I sees an effect that is not there.
Exam trap Lowering the significance threshold reduces type I errors but increases type II errors for a fixed sample size.
Why the other options are wrong
- Type II (beta) error Missing a real effect, the opposite situation.
- Selection bias Not indicated by the information given.
- Lack of statistical power Low power causes type II, not type I, errors.
- Confounding by indication Not indicated by the information given.
Where this sits in the CPSP syllabus
- Paper 1: Biostatistics, Hypothesis testing: p values, errors, statistical tests, correlation and regression