Validity and reliability
Validity and Reliability of Measurements: Accuracy, Precision and Kappa
Validity versus reliability, accuracy versus precision, types of validity, test-retest and inter-observer reliability, and kappa with a worked example. Also covers Bland-Altman analysis, sources of measurement error and lab quality control for FCPS Part 1.
Two separate questions about any measurement
- Validity (accuracy): does the tool measure what it claims to measure, giving the true value? It is damaged by systematic error (bias).
- Reliability (precision, reproducibility, repeatability): does it give the same result when the measurement is repeated under the same conditions? It is damaged by random error.
- A measurement can be reliable but not valid: a scale that always reads 2 kg too heavy gives consistent but wrong values. It cannot be truly valid without being reliable, because wildly scattered readings cannot reflect the true value consistently.
| Error type | Affects | Reduced by |
|---|---|---|
| Systematic (bias) | Validity, accuracy | Calibration, better design, blinding |
| Random | Reliability, precision | Repeated measurements, larger samples, standard protocols |
Remember Valid means on target; reliable means repeatable.
Exam trap Reliability is the extent to which a procedure gives the same result on repetition; it says nothing about whether the result is correct.
Accuracy and precision: the target picture
| Shots on a target | Accuracy (validity) | Precision (reliability) |
|---|---|---|
| Tightly grouped on the bullseye | High | High |
| Tightly grouped away from the bullseye | Low (biased) | High |
| Scattered evenly around the bullseye | High on average | Low |
| Scattered away from the centre | Low | Low |
Types of validity
| Type | Question |
|---|---|
| Face validity | On inspection, does it look as if it measures the concept? |
| Content validity | Does it cover all relevant aspects (judged by experts)? |
| Criterion validity | Does it agree with a gold standard? Concurrent (same time) or predictive (forecasts a later outcome) |
| Construct validity | Does it correlate with related measures (convergent) and not with unrelated ones (discriminant)? |
- For a diagnostic test, criterion validity is expressed as sensitivity and specificity against the gold standard. Specificity is the ability to correctly exclude people who do not have the disease; sensitivity is the ability to correctly identify those who do, so a negative result on a highly sensitive test rules disease out.
- Accuracy of a test overall = (true positives + true negatives) ÷ all tested: the share of results that are correct, positive and negative.
- For a study, internal validity means results are true for the participants (no bias or confounding) and external validity means they generalise to others.
Types of reliability
| Type | Meaning | Measured by |
|---|---|---|
| Test–retest (intra-observer, repeatability) | Same observer or instrument gives the same result on repeat | ICC, kappa |
| Inter-observer (inter-rater) | Different observers agree | Kappa (categories), ICC (continuous) |
| Internal consistency | Items in a questionnaire measure the same construct | Cronbach's alpha |
| Parallel forms | Two equivalent versions agree | Correlation |
Measuring agreement: kappa
- When two observers classify findings into categories (present or absent, normal or abnormal), some agreement happens by chance alone. Cohen's kappa measures agreement beyond chance.
- κ = (observed agreement − expected agreement) ÷ (1 − expected agreement).
- Worked example: two radiologists agree on 85% of chest films; chance agreement would be 50%. κ = (0.85 − 0.50) ÷ (1 − 0.50) = 0.70, good agreement.
| Kappa | Agreement |
|---|---|
| Under 0.20 | Poor |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Good (substantial) |
| Over 0.80 | Very good (almost perfect) |
- κ = 1 is perfect agreement; κ = 0 is no better than chance; negative values mean less than chance.
- Intraclass correlation coefficient (ICC) does the same job for continuous measurements.
- Bland–Altman plot: the difference between two methods is plotted against their mean, with limits of agreement (mean difference ± 1.96 SD of the differences). It is the right way to compare two measuring methods; a high correlation does not prove agreement, because two methods can rise together while one reads consistently higher.
Exam trap Kappa, not percentage agreement or correlation, measures inter-observer agreement on categorical findings, because it removes agreement expected by chance.
Variability statistics
- Standard deviation measures variability among individual values around the mean; it does not measure reliability, validity or sample size.
- Coefficient of variation (SD ÷ mean × 100%) expresses the precision of a laboratory assay: a low CV on repeated runs of one sample means good reproducibility.
Sources of measurement error
- Observer: differences between and within observers, terminal digit preference (rounding BP to 0 or 5), expectation bias.
- Instrument: poor calibration; a BP cuff that is too small overestimates BP.
- Subject: biological variation (diurnal BP, white-coat effect, posture, recent exercise).
- Reduce error with standardised protocols, training, calibration, repeated measurements, blinding of observers to the patient's group (removes observer bias) and automated devices.
Laboratory quality control
- Internal quality control: control samples of known value run with each batch and plotted on Levey–Jennings charts; Westgard rules flag random or systematic drift.
- External quality assessment (proficiency testing): an outside body sends unknown samples and compares laboratories.
- Pre-analytical errors (wrong tube, haemolysis, delay, mislabelling) are the commonest laboratory errors.
How FCPS asks about this
- Definition of reliability: the extent to which a procedure gives the same result when repeated (reproducibility).
- Statistic for agreement between two observers beyond chance on categorical findings: Cohen's kappa.
- Reliable but not valid example: a miscalibrated scale reading consistently heavy.
- What standard deviation measures: variability among individual values.
- Comparing two methods of measuring the same quantity: Bland–Altman limits of agreement, not correlation.
- Ability of a test to correctly exclude non-diseased people: specificity; sensitivity identifies the diseased.
- Internal consistency of a questionnaire: Cronbach's alpha.
Test yourself on this topic
5 of the 6 questions in our bank on this topic. Choose an answer to see why every option is right or wrong.
A weighing scale in an OPD gives readings within 0.1 kg of each other when a child is weighed three times, but every reading is 1.5 kg above the true weight. How is this scale best described?
Answer: Reliable but not valid
Reliability (precision, repeatability) is getting the same result on repeated measurement; validity (accuracy) is measuring the true value. This scale is consistent, so it is reliable, but it is systematically wrong, so it is not valid: a systematic error or bias. Recalibration would fix it. A valid measure must be reliable, but a reliable one need not be valid.
Validity vs reliability
- Reliability: same result again
- Validity: true result
- Consistent but wrong: systematic error
- Valid measures must be reliable
Remember Tight cluster off the bullseye: reliable, not valid.
Exam trap Random error lowers reliability; systematic error lowers validity.
Why the other options are wrong
- Valid but not reliable It is consistent, so it is reliable; it is not accurate.
- Both valid and reliable It is consistently wrong.
- Neither valid nor reliable It is reliable.
- Sensitive but not specific Sensitivity and specificity describe diagnostic tests.
A 50-g glucose challenge test is evaluated in an antenatal clinic. Of 120 women later confirmed by OGTT to have gestational diabetes, 102 had a positive challenge test. What is the sensitivity of the challenge test?
Answer: 85%
Sensitivity is the proportion of people with the disease who test positive: true positives divided by all who have the disease. Here it is 102/120 = 85%. The remaining 15% are false negatives. Sensitivity is calculated only from the people who have the disease, so the number of women without diabetes is not needed.
Sensitivity
- TP ÷ (TP + FN)
- Uses only the diseased
- Here 102 ÷ 120 = 85%
- High sensitivity rules out (SnOut)
Remember Sensitivity looks only at those who have the disease.
Exam trap Sensitivity and false-negative rate always add up to 100%.
Why the other options are wrong
- 15% This is the false-negative rate.
- 102% A proportion cannot exceed 100%.
- 54% Not derived from these numbers.
- 98% Not derived from these numbers.
Which statistic measures agreement between two observers beyond chance for categorical findings?
Answer: Kappa statistic
Cohen's kappa measures inter-observer agreement corrected for agreement expected by chance: 0 = chance only, 1 = perfect agreement; above 0.6 is substantial. It assesses reliability (reproducibility), not validity.
Kappa
- Inter-observer agreement beyond chance
- 0 = chance; 1 = perfect
- Measures reliability
Remember Kappa: do two doctors agree?
Why the other options are wrong
- Pearson correlation Association between continuous variables.
- Odds ratio Association in case–control studies.
- Standard error Precision of the mean.
- Relative risk Association in cohorts.
A bedside test for preterm prelabour rupture of membranes is evaluated in 500 women. It gives 90 true positives, 10 false negatives, 50 false positives and 350 true negatives. What is the overall accuracy of the test?
Answer: 88%
Accuracy is the proportion of all results that are correct: (true positives + true negatives) ÷ total = (90 + 350) ÷ 500 = 440 ÷ 500 = 88%. In this study the sensitivity is 90/100 = 90%, specificity 350/400 = 87.5%, positive predictive value 90/140 = 64% and negative predictive value 350/360 = 97%. Accuracy depends on prevalence and can hide poor performance in a small group.
Accuracy
- (TP + TN) ÷ total
- Here 440 ÷ 500 = 88%
- Depends on prevalence
- Can hide poor sensitivity
Remember Accuracy counts every correct answer, positive or negative.
Exam trap A test can have high accuracy simply because most people tested do not have the condition.
Why the other options are wrong
- 90% This is the sensitivity.
- 64% This is the positive predictive value.
- 97% This is the negative predictive value.
- 72% Not derived from these numbers.
A new blood pressure device gives readings that agree closely with intra-arterial measurements. Which property is this?
Answer: Validity (accuracy)
Validity is how well a test measures what it is meant to (closeness to the true value). Reliability is how consistently it gives the same result on repeat. A test can be reliable but not valid (consistently wrong), but a valid test must also be reliable.
Validity vs reliability
- Validity: hits the target (accurate)
- Reliability: same result each time (precise)
- Reliable but not valid = consistently wrong
Remember Valid = on target; reliable = tightly grouped.
Why the other options are wrong
- Reliability (precision) only Repeatability, not closeness to the truth.
- Sensitivity Detecting true positives.
- Specificity Detecting true negatives.
- Predictive value Probability of disease given a result.
Where this sits in the CPSP syllabus
- Paper 1: Biostatistics, Validity and reliability of measurements