Statistical Inference

CLT, Confidence Intervals, Hypothesis Testing, and Effect Sizes

Learning Objectives

  1. Explain the Central Limit Theorem and why it enables all inference
  2. Calculate and interpret confidence intervals for means and proportions
  3. Describe the logic of hypothesis testing (proof by contradiction)
  4. Interpret p-values correctly and avoid common misconceptions
  5. Distinguish Type I error (α), Type II error (β), and power (1 − β)
  6. Separate statistical significance from clinical significance
  7. Compute and interpret effect sizes (Cohen’s d)
  8. Recognize the multiple testing problem and apply corrections

Clinical Hook: When “Significant” Doesn’t Mean Important

Phase III Antihypertensive Trial (n = 10,000):

New drug reduced SBP by 2 mmHg more than standard therapy.

p = 0.03 → Headlines: “Significant BP reduction!”

. . .

But the clinical reality:

  • MCID for BP = 5–10 mmHg
  • 2 mmHg is clinically irrelevant
  • With n = 10,000, even trivial differences reach p < 0.05

This module builds inference from the ground up — and teaches you when to trust a “significant” result.

The Fundamental Problem

You measure haemoglobin in 50 patients → sample mean = 11.8 g/dL

But a different set of 50 patients → mean = 12.1 g/dL

And another 50 → mean = 11.5 g/dL

Key question: How can ONE sample tell us about the POPULATION?

Answer: The Central Limit Theorem (CLT) — the single most important result in statistics.

The Central Limit Theorem: Statement

Tip

Draw repeated random samples of size n from any population with mean μ and SD σ.

As n increases, the distribution of sample means (\(\bar{x}\)) approaches a normal distribution with:

\[\text{Mean of } \bar{x} = \mu \qquad \text{SD of } \bar{x} = \frac{\sigma}{\sqrt{n}} \text{ (Standard Error)}\]

Three key insights:

Insight Implication
Works for any population shape Even skewed data → normal sampling distribution
SE shrinks with √n More data → more precision (but diminishing returns)
“Large enough” ≈ n ≥ 30 Depends on skewness; very skewed → need n ≥ 100

CLT in Action: Skewed Data Becomes Normal

Even heavily skewed hospital-stay data → normal sampling distribution as n increases

The Standard Error: Bridge to Inference

\[SE = \frac{\sigma}{\sqrt{n}}\]

CLT Connects to Everything

The SE connects the CLT to all inference tools:

Tool Formula Pattern Uses SE?
Confidence Interval Estimate ± critical value × SE ✅
Test Statistic (Observed − Expected) / SE ✅
Sample Size (Module 6) Manipulate SE for desired precision ✅

The CLT is the engine; SE is the transmission; CIs and tests are the wheels.

From Point Estimate to Confidence Interval

Point estimate: \(\bar{x}\) = 9.2 g/dL (how precise is this?)

Confidence interval: provides a range of plausible values

\[\text{CI} = \text{Point Estimate} \pm (\text{Critical Value}) \times \text{Standard Error}\]

For a mean (σ unknown):

\[\bar{x} \pm t_{\alpha/2, \, df} \times \frac{s}{\sqrt{n}}\]

  • df = n − 1
  • For 95% CI: \(t_{0.025}\) ≈ 1.96 when n is large

For a proportion:

\[\hat{p} \pm z_{\alpha/2} \times \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\]

  • For 95% CI: \(z_{0.025}\) = 1.96
  • Check: np ≥ 5 and n(1−p) ≥ 5

Worked Example: CI for a Mean

64 newly diagnosed Type 2 DM patients — Mean FBS:

Given Value
\(\bar{x}\) 156 mg/dL
s 32 mg/dL
n 64

Step 1: SE = 32 / √64 = 32 / 8 = 4 mg/dL

Step 2: Critical value: \(t_{0.025, 63}\) ≈ 2.00

Step 3: CI = 156 ± 2.00 × 4 = 156 ± 8 = (148, 164) mg/dL

Interpretation: We are 95% confident that the true mean FBS lies between 148 and 164 mg/dL.

Worked Example: CI for a Proportion

400 adults in rural Bhopal; 72 had hypertension (BP ≥ 140/90)

Given Value
\(\hat{p}\) 72/400 = 0.18 (18%)
n 400

Step 1: SE = \(\sqrt{0.18 \times 0.82 / 400}\) = \(\sqrt{0.000369}\) = 0.0192

Step 2: CI = 0.18 ± 1.96 × 0.0192 = 0.18 ± 0.038 = (14.2%, 21.8%)

Check: np = 72 ≥ 5, n(1−p) = 328 ≥ 5 → normal approximation valid ✅

What Does “95% Confidence” Actually Mean?

CI Interpretation: Right vs Wrong

Wrong Interpretation Why It’s Wrong
❌ “95% probability the true mean is in this interval” True mean is fixed — no probability about it
❌ “95% of patients fall in this range” CI is about the mean, not individual values
❌ “95% chance a new sample mean falls here” New sample generates its own CI

✅ Correct: “If we repeated this study many times, 95% of calculated CIs would contain the true population mean.”

What Makes a CI Narrower?

The Logic of Hypothesis Testing

Mathematical analogy:

To prove √2 is irrational → assume it IS rational → derive a contradiction → reject assumption

Statistical analogy:

To test if a drug works → assume it DOESN’T (H₀) → observe data → if data are very unlikely under H₀ → reject H₀

Key: We never prove the drug works; we only ask whether “no effect” is plausible.

H₀ and H₁: The Two-Hypothesis Framework

Definition Example
H₀ (Null) No effect / no difference New drug = standard drug
H₁ (Alternative) Effect exists New drug ≠ (or >) standard drug

One-Tailed vs Two-Tailed Tests

Default: Two-tailed | Use one-tailed only if direction pre-specified and opposite is irrelevant

Test Statistics: Common Pattern

\[\text{Test Statistic} = \frac{\text{Observed} - \text{Expected under } H_0}{\text{Standard Error}}\]

Scenario Formula Distribution
One sample mean \(t = (\bar{x} - \mu_0) / (s/\sqrt{n})\) t (n−1 df)
Two independent means \(t = (\bar{x}_1 - \bar{x}_2) / SE_{diff}\) t (n₁+n₂−2 df)
Paired means \(t = \bar{x}_{diff} / (s_{diff}/\sqrt{n})\) t (n−1 df)
One proportion \(z = (\hat{p} - p_0) / \sqrt{p_0(1-p_0)/n}\) Normal (z)
Two proportions \(z = (\hat{p}_1 - \hat{p}_2) / SE_{diff}\) Normal (z)

Always: (signal − expected) / uncertainty

What Is a P-Value?

Tip

The p-value = probability of observing data as extreme or more extreme than what we got, assuming H₀ is true.

P-Value Misconceptions

❌ Misconception ✅ Reality
“p = 0.03 → 3% chance H₀ is true” P(data this extreme | H₀ true), not P(H₀ true | data)
“p = 0.03 → 97% sure result is real” Replication depends on effect size, design, not p alone
“p = 0.06 → ‘trending toward significance’” Arbitrary! Report exact p-value and effect size
“p = 0.15 → treatment has no effect” Absence of evidence ≠ evidence of absence
“p = 0.001 → very confident” Bias doesn’t care about your p-value
“Multiple p < 0.05 → strong evidence” 20 tests → expect ~1 false positive by chance

ASA (2016): Scientific conclusions require more than p < 0.05. Always report effect sizes and CIs.

The Four Outcomes of a Hypothesis Test

Reject H₀ Fail to Reject H₀
H₀ TRUE (no effect) TYPE I ERROR (α) ❌ False Positive ✅ Correct (1 − α)
H₀ FALSE (effect exists) ✅ Correct — Power (1 − β) TYPE II ERROR (β) ❌ False Negative

Convention: α = 0.05, β = 0.20, Power = 0.80

The 4:1 trade-off: False positive (adopting useless drug) is considered 4× worse than false negative (missing an effective one).

Type I Error (α): The False Positive

Definition: Concluding an effect exists when it DOESN’T

Probability: α (set by researcher, usually 0.05)

Dangerous clinical scenarios:

  • Approving an ineffective drug → patients receive it instead of proven therapy
  • Unnecessary surgery → thousands undergo biopsies for a spurious biomarker association
  • Policy changes → government bans a food additive based on a statistical fluke

What increases Type I risk? Multiple testing, p-hacking, flexible analysis plans, not pre-registering

Type II Error (β): The False Negative

Definition: Missing a real effect — declaring “no difference” when one exists

Probability: β (depends on sample size, effect size, variability)

Dangerous clinical scenarios:

  • Shelving an effective drug → immunotherapy that reduces mortality by 15% is abandoned because trial was too small (n = 80)
  • False equivalence → concluding two drugs are “the same” with n = 40 per group
  • Premature stopping → interim p = 0.08 stops a trial that would have reached p = 0.01

The #1 cause: Small sample size

Diagnostic Test Parallel

The concepts map directly to Module 5:

Hypothesis Testing Diagnostic Testing
Type I Error (α) False Positive — healthy person tests positive
Type II Error (β) False Negative — sick person tests negative
Power (1 − β) Sensitivity — correctly detecting disease
Significance level (α) 1 − Specificity

Power IS sensitivity for hypothesis tests — the ability to detect a true effect.

Power: What Determines It?

↑ Power with: ↑ sample size, ↑ effect size, ↑ α, ↓ variability

The α–β Trade-off

α–β Trade-off: Key Insight

  • Decrease α (more stringent) → β increases (miss more real effects)
  • You cannot minimize both errors simultaneously

Solution: Increase sample size — more data reduces BOTH errors

Strategy α β Power Feasibility
Liberal 0.10 0.10 90% ⚠️ More false positives
Standard 0.05 0.20 80% ✅ Most common
Stringent 0.01 0.30 70% ⚠️ Miss real effects
Standard + large n 0.05 0.10 90% ✅ Best but expensive

Statistical vs Clinical Significance: The 2×2 Framework

Real-World Examples from Indian Medicine

Study Effect n p-value MCID Verdict
Hypertension SBP ↓ 2 mmHg 10,000 0.03 5–10 mmHg Stat YES, Clinical NO ❌
Sepsis Mortality ↓ 12% 200 0.001 ≥ 5% Stat YES, Clinical YES ✅
Diabetes HbA1c ↓ 0.4% 150 0.12 0.5% Stat NO, Clinical borderline ❓
Anticoagulation Stroke ↓ 0.5% 3,000 0.001 ≥ 2% Stat YES, Clinical NO ❌

Key principle: A statistically significant result below the MCID is clinically irrelevant — no matter how small the p-value.

MCID: Minimal Clinically Important Difference

Always define the MCID before the trial:

Outcome MCID
Systolic BP reduction 5–10 mmHg
HbA1c reduction 0.5%
Pain (VAS 0–100) 10–15 points
Mortality reduction 5% absolute
Quality of life (EQ-5D) 0.05–0.08 units

Decision rule: If 95% CI lies entirely above MCID → clinically important. If CI straddles MCID → inconclusive. If CI below MCID → not important.

Effect Size: Why It Matters More Than P-Values

All three could have p < 0.05 with adequate n — but clinical relevance differs hugely!

Cohen’s d: Formula and Benchmarks

\[d = \frac{\bar{x}_1 - \bar{x}_2}{s_p} \qquad s_p = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}}\]

Cohen’s d Size Clinical Example n needed/group
0.2 Small New antacid vs old antacid ~400
0.5 Medium Lisinopril for hypertension ~64
0.8 Large Penicillin for strep throat ~25
> 1.2 Very large Insulin for new-onset diabetes ~12

Key advantage: Effect size is independent of sample size — p-value is not!

Worked Example: ACE Inhibitor Trial

ACE inhibitor vs placebo (n = 50 per group, 8 weeks):

Group SBP Reduction SD
Placebo 5 mmHg 10
ACE-I 12 mmHg 10

\[s_p = \sqrt{\frac{49 \times 100 + 49 \times 100}{98}} = 10 \qquad d = \frac{12 - 5}{10} = 0.7\]

Question Answer
Statistically significant? ✅ p < 0.001
Effect size meaningful? ✅ d = 0.7 (medium-large)
Clinically important? ✅ 7 mmHg > MCID of 5 mmHg
Should we implement? ✅ YES

The Multiple Testing Problem

Bonferroni correction: αcorrected = α / k (e.g., 10 tests → use α = 0.005)

P-Hacking: The Dark Side

What is p-hacking? Manipulating analysis to achieve p < 0.05:

  1. Test many variables → report only significant ones
  2. Try multiple statistical tests on the same data
  3. Selectively exclude outliers until p < 0.05
  4. Stop data collection when p first crosses 0.05
  5. Report only “interesting” subgroups

Solution: Pre-register analysis plans (ClinicalTrials.gov, OSF Registries)

CIs Tell You More Than P-Values

Study D: wide CI → underpowered (need more data) | Study E: narrow CI → truly no effect

CI vs P-Value: Summary

Aspect P-Value Confidence Interval
Shows effect size? ❌ No ✅ Yes
Shows precision? ❌ Implicit ✅ Yes
Avoids dichotomy? ❌ No ✅ Yes
Assess clinical importance? ❌ No ✅ Compare to MCID
Subject to p-hacking? ⚠️ Very ✅ Less so

Best practice: Report both. But if forced to choose one → choose the CI.

Using CIs for Clinical Decisions

Trial result: SBP reduced by 6 mmHg (95% CI: 3 to 9, p = 0.001)

MCID = 5 mmHg

  • Point estimate (6) > MCID ✅
  • But lower bound (3) < MCID

Interpretation: Effect likely clinically meaningful, but we cannot rule out the true effect is below the MCID. Suggestive but not definitive.

P-value alone would just say “p = 0.001, significant!” — missing this nuance entirely.

The Logical Flow of Inference

Key Takeaways

  1. CLT → sample means are normal regardless of population shape (foundation)
  2. SE = σ/√n → precision improves with √n (diminishing returns)
  3. CI = estimate ± critical value × SE → range of plausible values
  4. P-value = P(data this extreme | H₀ true) — NOT P(H₀ true)
  5. Type I = false positive (α) | Type II = false negative (β)
  6. Power = 1 − β ≥ 0.80 before starting any study
  7. Statistical significance ≠ clinical significance — always check MCID
  8. Effect size (Cohen’s d) is independent of n — report it always
  9. Multiple testing inflates false positives — correct with Bonferroni
  10. CIs > p-values — report both, but CIs tell you more

Do’s and Don’ts

✅ DO:

  • Pre-specify hypotheses
  • Report CIs AND p-values
  • Report effect sizes
  • Power analysis BEFORE study
  • Correct for multiple tests
  • Report ALL tests conducted
  • Consider clinical significance

❌ DON’T:

  • Interpret p as P(H₀ true)
  • Claim non-significant = no effect
  • Use one-tailed tests to dodge α
  • Stop when p first hits 0.05
  • Test many hypotheses uncorrected
  • Say “trending toward significance”
  • Confuse statistical and clinical significance

Formula Reference Card

Concept Formula
Standard Error (mean) \(SE = s / \sqrt{n}\)
95% CI for mean \(\bar{x} \pm t_{0.025} \times SE\)
95% CI for proportion \(\hat{p} \pm 1.96 \sqrt{\hat{p}(1-\hat{p})/n}\)
Test statistic (general) \((Observed - Expected_{H_0}) / SE\)
Cohen’s d \((\bar{x}_1 - \bar{x}_2) / s_p\)
Bonferroni \(\alpha_{adj} = \alpha / k\)
Multiple testing P \(P(\geq 1 \text{ FP}) = 1 - (1-\alpha)^k\)

Remember: The CLT makes all of this possible.

Thank You

Module 8: Statistical Inference

From CLT to CIs to Hypothesis Testing — and knowing when “significant” truly matters

Tip

Next: Module 9 — Comparing Two Groups (t-tests, Mann-Whitney, Chi-Square)