Meta-Analysis and Evidence Synthesis

Combining Evidence from Multiple Studies

Learning Objectives

  1. Distinguish systematic review from meta-analysis
  2. Read and interpret forest plots — estimates, CIs, weights, and pooled effects
  3. Choose between fixed-effect and random-effects models
  4. Calculate and interpret I² for heterogeneity
  5. Assess publication bias using funnel plots
  6. Evaluate evidence quality using GRADE
  7. Recognise when meta-analysis is inappropriate

Clinical Hook: TB in Healthcare Workers

Rajnish Joshi (now AIIMS Bhopal) led a systematic review in PLoS Medicine (2006):

“TB among Health-Care Workers in Low- and Middle-Income Countries”

Individual studies told different stories:

  • Some: LTBI prevalence 33%
  • Others: LTBI prevalence 79%
  • Risk estimates ranged from 2× to 5× the general population

. . .

The team synthesised 42 articles (51 studies):

  • Pooled LTBI prevalence: 54% (range 33–79%)
  • Revealed why estimates varied — TB burden, infection control, diagnostic method

Meta-analysis turns conflicting signals into a coherent picture.

Joshi R, Reingold AL, Menzies D, Pai M. PLoS Med 2006;3(12):e494.

Systematic Review vs Meta-Analysis

Systematic Review Meta-Analysis
What Structured method for finding, appraising, synthesising all evidence Statistical technique combining results into a pooled estimate
Nature Qualitative (can exist without statistics) Quantitative (requires numbers)
Stand-alone? Yes No — needs a systematic review
Output Evidence summary, quality assessment Pooled effect, forest plot, I²

The Hierarchy

A meta-analysis is only as good as the systematic review that feeds it. Garbage in, garbage out.

The PICO Framework

Every systematic review starts with a focused question:

Component Meaning Example (Joshi et al.)
Population Who? Healthcare workers in LMICs
Intervention/Exposure What? Working in healthcare settings
Comparator Compared to? General population
Outcome What measured? LTBI and active TB prevalence

Without a clear PICO, you don’t know what to search for, what to include, or what to pool.

The Forest Plot — Anatomy

Reading the Forest Plot

Each row = one study:

  • Square = point estimate (OR, RR, or mean difference)
  • Horizontal line = 95% confidence interval
  • Square size = study weight (larger studies → larger squares)

Bottom diamond = pooled estimate:

  • Centre = pooled effect size
  • Width = 95% CI of pooled estimate
  • If the diamond does not cross the null line (OR = 1) → statistically significant

X-axis is on a log scale for ratios — so OR = 0.5 and OR = 2.0 are equidistant from 1.0

Fixed-Effect vs Random-Effects

Fixed-Effect Random-Effects
Assumption One true effect across all studies Each study has a different true effect
Between-study variation Sampling error only Sampling error + real differences
CI width Narrower Wider (more conservative)
Weighting Proportional to study size More balanced — small studies get more weight
Use when Studies very similar; I² < 25% Studies diverse; I² ≥ 50%

Practical Rule

When in doubt, use random-effects. If heterogeneity is truly zero, both models give the same result anyway.

Fixed vs Random — Visualised

Heterogeneity — The I² Statistic

\[I^2 = \frac{Q - df}{Q} \times 100\%\]

I² value Interpretation
0–25% Low — studies agree well
25–50% Moderate — some differences
50–75% Substantial — results vary considerably
> 75% Considerable — may be too different to pool

I² tells you what percentage of variation is real (not chance).

High I² → investigate why before pooling.

Heterogeneity Example: OSA in India

Sources: Diagnostic method, population type, OSA definition, age/BMI, scoring rules

Publication Bias — The Funnel Plot

Asymmetric funnel → small studies with null results are missing → publication bias

Egger’s test provides a formal test for asymmetry (needs ≥ 10 studies)

Risk of Bias — Cochrane RoB 2

For RCTs, assess five domains:

Domain What can go wrong
Randomisation Inadequate sequence generation / allocation concealment
Deviations from intervention Unblinded participants change behaviour
Missing outcome data Differential dropout between groups
Outcome measurement Assessor knows group; subjective outcome
Selection of reported result Authors pick the 'best' outcome or time point

Traffic light: Each domain rated Low (green), Some concerns (yellow), High (red)

A study with high risk in even one critical domain can bias the whole meta-analysis.

GRADE — Rating Evidence Quality

Factor Rating
Start RCTs High (⊕⊕⊕⊕)
Observational Low (⊕⊕○○)
Downgrade Risk of bias −1 or −2
Inconsistency (I²) −1 or −2
Indirectness −1 or −2
Imprecision −1 or −2
Publication bias −1 or −2
Upgrade Large effect +1 or +2
Dose-response +1
Confounders reduce effect +1

Final ratings: High (⊕⊕⊕⊕) → Very low (⊕○○○)

A statistically significant pooled effect from very-low-quality evidence is not compelling.

PRISMA Flow Diagram

When NOT to Meta-Analyse

Warning

  1. Extreme clinical heterogeneity — apples and oranges
  2. Very high I² with no identifiable source
  3. Only 2–3 studies — narrative synthesis is better
  4. High risk of bias in most studies — precise but wrong
  5. Contradictory results with plausible mechanistic reasons

Teaching example: Pralidoxime in organophosphorus poisoning

  • Earlier observational studies suggested benefit
  • Eddleston, Joshi R et al. (PLoS Med 2009): double-blind RCT (n = 235)
  • Result: pralidoxime group had higher mortality (24.8% vs 15.8%)
  • Pooling the RCT with weaker observational studies would dilute the stronger signal

A single strong RCT can be more informative than a meta-analysis of many weak studies.

Sensitivity Analysis — Testing Robustness

Analysis What you change What it tests
Exclude high-RoB studies Drop biased studies Is the effect driven by bias?
Fixed vs random effects Switch the model Does between-study variation matter?
Leave-one-out Remove each study in turn Is one study driving the result?
Restrict to RCTs Drop observational studies Consistent in stronger designs?
Trim-and-fill Impute missing studies Publication bias impact?

If the pooled estimate changes substantially → the conclusion is not robust.

Report this transparently.

Identify the Analysis — Vignette 1

A team finds 15 RCTs comparing albendazole vs placebo for soil-transmitted helminths in Indian schoolchildren. Each trial reports cure rate. The trials use different doses (200 mg vs 400 mg) and follow-up periods (1 week vs 4 weeks). I² = 72%.

Random-effects meta-analysis with subgroup analysis — High heterogeneity (72%) warrants investigation. Subgroup by dose and follow-up duration. A fixed-effect model would be inappropriate given the substantial I².

Identify the Analysis — Vignette 2

A systematic review of Mycobacterium w (Mw) as COVID-19 adjunct therapy finds 2 small open-label trials (n = 40 and n = 55) with opposite results. Trial 1 shows benefit; Trial 2 shows no effect. Both are high risk of bias (unblinded, small).

Do NOT meta-analyse. Only 2 studies, opposite results, both high RoB. A pooled estimate would be misleading — narrative synthesis is the appropriate approach. Wait for larger, blinded RCTs before pooling.

Identify the Analysis — Vignette 3

You are reviewing a Cochrane systematic review of statins for primary prevention of cardiovascular disease. The forest plot shows 12 RCTs with a pooled RR = 0.75 (95% CI: 0.70–0.81), I² = 15%. The funnel plot is symmetric. GRADE rates the evidence as High.

This is a well-conducted meta-analysis. Low heterogeneity (I² = 15%), symmetric funnel plot (no publication bias), high GRADE quality. The pooled RR = 0.75 means a 25% reduction in cardiovascular events — a robust, clinically meaningful result.

Common Mistakes

Warning

  1. Pooling apples and oranges — combining clinically incompatible studies
  2. Ignoring I² — reporting a pooled estimate without assessing heterogeneity
  3. Choosing the model post-hoc — switching to fixed-effect because it gives a “better” p-value
  4. Equating statistical significance with clinical importance — a significant pooled OR = 1.05 may be trivial
  5. No sensitivity analysis — only reporting one set of results
  6. Funnel plot with < 10 studies — insufficient power to detect asymmetry
  7. Confusing random-effects with random sampling — the “random” refers to study effects, not patient selection
  8. Ignoring GRADE — a significant result from very-low-quality evidence is not compelling

Key Takeaways

  1. Systematic review first, meta-analysis second — never pool without structured search
  2. Forest plot = squares (studies), diamond (pooled), vertical line at null
  3. Random-effects is the safer default when heterogeneity is uncertain
  4. I² > 75% → stop, investigate, consider subgroup analysis before pooling
  5. Funnel plot asymmetry suggests publication bias (needs ≥ 10 studies)
  6. GRADE rates overall evidence quality — significant ≠ high quality
  7. Sensitivity analysis is not optional — vary assumptions, test robustness
  8. A single strong RCT (like the pralidoxime trial) can be more informative than a meta-analysis of many weak studies