Learning objectives

By the end of this session, you will be able to:

  1. Explain why a single study is not enough to answer an applied question.
  2. Distinguish a narrative review, a systematic map, a systematic review and a meta-analysis — and say what each can and cannot tell you.
  3. Name the standards that make a synthesis trustworthy (CEE, Cochrane, PRISMA 2020, ROSES).
  4. Place each step of this week in the workflow of a systematic review.

One study…

What is this?


One can. Clear, precise, detailed.

Many studies…

And now?


Many cans. Each one looks different.

…and the big picture

Georges Seurat’s A Sunday Afternoon on the Island of La Grande Jatte, rebuilt from 106,000 cans.

One study = one can.
A synthesis reveals a pattern that no single study shows — but it can also hide the details of each can.

Chris Jordan, Cans Seurat (2007), series Running the Numbers: 106,000 aluminium cans, the number used in the US every 30 seconds.

In medicine: one study, twelve children, one retraction

Year What happened
1998 Wakefield et al. (The Lancet): case series of 12 children, suggests a link between the MMR vaccine and autism. Press conference, huge media coverage.
1998–2004 MMR vaccination coverage falls in the UK; measles returns.
2004 10 co-authors retract the paper’s interpretation.
2010–11 The Lancet fully retracts the paper; Wakefield is struck off; the BMJ shows the data were fraudulent.
2014 Meta-analysis of 5 cohort (1.26 million children) and 5 case-control studies: no association between vaccination and autism.

A single, vivid study small sample, no control group, conflicts of interest — but enormous influence.

The weight of evidence all eligible studies, weighted by their precision, appraised for bias.

Source: Wakefield et al. (1998); Godlee et al. (2011); Taylor et al. (2014) (cohorts: 1,256,407 children; case-control: 9,920).

Our running example this week

The question Does diversifying farms (intercropping, agroforestry, rotations…) benefit biodiversity — without costing yield?

A published systematic map (Jones et al. 2021) (partly built from earlier meta-analyses):

  • 1,590 records screened
  • 237 studies · 4,076 comparisons
  • 48 countries · 1986–2021

We use it all week: map, effect sizes, models, bias.

Quick poll

Show of hands

Compared with a monoculture, does intercropping reduce the abundance of crop pests?

  • Yes, clearly
  • No
  • It depends

If you had read only one study…

Straub et al. 2013: intercropping increases pest abundance (+109%). Conclusion: avoid intercropping?

…or another one

Phoofolo et al. 2010: intercropping cuts pests (-86%). Conclusion: intercrop everywhere?

Primary studies disagree

Source: 28 studies, 382 comparisons from Jones et al. (2021). One point per study (random-effects summary of its comparisons), 95% CI.

A synthesis sees the pattern — and its spread

On average, intercropping reduces pests by 27%
(95% CI: 11–40% reduction).

But in a new field, the effect could be anywhere from -80% to +170%.
→ The honest answer to the poll was “it depends”.

Random-effects meta-analysis accounting for several comparisons per study: k = 382 comparisons, 28 studies. You will fit this model on Wednesday.

Two intervals, two questions

Confidence interval (CI) How precisely do we know the average effect?

-40% to -11% → shrinks as studies accumulate.

Prediction interval (PI) What effect should we expect in a new field?

-80% to +170% → does not shrink to zero with more studies: it reflects real differences between sites, crops and pests.

Heterogeneity (τ²) The spread of the true effects (between studies, and between comparisons within studies). Here the true effect differs from field to field — which is why the PI is wide.

In ecology this is the rule, not the exception (Senior et al. 2016).

Report the mean with its prediction interval. Explaining the spread (crop, pest group, region) is the job of moderators — Wednesday afternoon.

Too much to read

Publications grow exponentially — today > 3 million peer-reviewed articles per year across all sciences (Johnson et al. 2018).

IPCC assessments cite a shrinking share of the relevant literature at each cycle.

Left: Gurevitch et al. (2018). Right: Minx et al. (2017).

Syntheses boom…

Cumulative number of meta-analyses on crop diversification, by scale of the practice.

Dozens of meta-analyses on crop diversification…
Do they agree? Can we trust them?

Source: Beillouin et al. (2019) and updates.

…but their quality varies

Share of meta-analyses on crop diversification reporting each item.

Reporting standards PRISMA 2020 (Page et al. 2021)
PRISMA-EcoEvo (O’Dea et al. 2021)
ROSES (Haddaway et al. 2018)

Appraising a synthesis CEESAT (Woodcock et al. 2014)
Ten appraisal questions (Nakagawa et al. 2017)

Source: Beillouin et al. (2019) and updates.

Garbage in, garbage out: appraise before you pool

The authors of the map appraised the comparisons (replication, duration, distance between plots) — 130 of 382 could not be appraised.

Dark bar = 95% CI, light bar = 95% PI.
Restricted to valid comparisons: same mean (−27%) and a narrower prediction interval — weaker designs may add noise (or the 13 remaining studies are simply more alike). Appraisal lets you check this.

You only know this because validity was coded. Reporting standards (PRISMA, ROSES) ≠ critical appraisal of each primary study — Wednesday morning.

Your turn

30 seconds · discuss with your neighbour

Review, systematic review, systematic map, meta-analysis: what is the difference?

Describing the evidence

Method Question What makes it distinctive Output
Narrative review Broad Expert selection of studies; no explicit, reproducible method Expert overview — prone to selection bias
Scoping review Broad, exploratory Maps concepts and types of evidence; usually no appraisal Clarifies a field before a full review
Systematic map Broad: what is known? Systematic search and coding of the whole evidence base Database + map of evidence and gaps

Source: Collaboration for Environmental Evidence (2022); Grant and Booth (2009); James et al. (2016).

Answering a question

Method Question What makes it distinctive Output
Rapid review Focused, urgent Systematic review with declared shortcuts (fewer databases, one screener) Timely answer, lower certainty
Systematic review Focused (PICO/PECO) A priori protocol, comprehensive search, explicit eligibility, critical appraisal, transparent synthesis Answer to the question, with its certainty
Meta-analysis Quantitative Statistical combination of effect sizes, weighted by precision Mean effect, heterogeneity, moderators

Systematic review ≠ meta-analysis. “Systematic” describes the method; meta-analysis is a statistical tool. Gold standard: a meta-analysis within a systematic review.

Source: Collaboration for Environmental Evidence (2022); Higgins et al. (2024). Umbrella reviews apply the same logic to existing reviews.

Choosing a method

▶
▲
Aim: describe the evidence  →  quantify an effect
Rigour, transparency, effort
Narrative review
"Meta-analysis" of a convenience sample
Rapid review  ·  Scoping review
Systematic map
Systematic review (narrative or quantitative synthesis)
+ meta-analysis

A systematic map: where is the evidence?

Number of studies per practice × taxon, from Jones et al. (2021). You will build this kind of map on Tuesday afternoon.

Strengths and limits of meta-analysis

Meta-analysis can… …but it cannot
See the forest for the trees Show every tree: the average can mask contrasting effects
Quantify heterogeneity (τ², prediction interval) Make incomparable studies comparable: that is decided by the question
Estimate the average effect of a practice across the studied sites (when studies are experiments) Turn moderator comparisons (between studies) into causal evidence
Explore small-study effects and test robustness Remove bias from primary studies: garbage in, garbage out

The evidence pyramid, revisited

Experiments Observational studies Case studies Expert opinion Systematic reviews & meta-analyses

The classic pyramid puts meta-analyses on top. The revised pyramid (Murad et al. 2016):

  • Syntheses are a lens through which primary evidence is viewed — not a level above it.
  • Boundaries are wavy: a well-run observational study can beat a flawed experiment.

A synthesis is only as strong as the studies it contains and the method used to find and appraise them.

Adapted to ecology and agronomy from Murad et al. (2016).

Standards exist — we follow them this week

Guidelines CEE Guidelines and Standards, v5.1 (Collaboration for Environmental Evidence 2022) — environment
Cochrane Handbook, v6.5 (Higgins et al. 2024) — health
Campbell Collaboration — social sciences
Environment: mostly field experiments and observational studies, strong spatio-temporal variability → expect heterogeneity.

Protocol registration PROCEED (environment) · PROSPERO (health) · OSF

Reporting ROSES (Haddaway et al. 2018)
PRISMA 2020 (Page et al. 2021) and PRISMA-EcoEvo (O’Dea et al. 2021)

Reference books Koricheva et al. (2013) · Borenstein et al. (2021)
Harrer et al. (2021) (free online, R code)

Which method for which question?

A. A ministry needs, within 6 weeks, an overview of whether hedgerows benefit pollinators.

Rapid review — with its shortcuts declared.

B. Which diversification practices, taxa and regions have been studied, and where are the gaps?

Systematic map.

C. How much does intercropping change pest abundance, and why does it vary?

Systematic review with meta-analysis (and moderators).

Your week: one systematic review, step by step

Mon
Question & stakeholders
Protocol
Search · Screening
Full texts · Reporting
Tue
Data visualisation
From hypothesis to map
Codebook
R practical: evidence maps
AI tools
Wed
Effect sizes · appraisal
Quantitative extraction
Models · bias
Advanced methods
Thu
Group projects:
do it yourself
Fri
Presentations
and feedback
Joseph · Sylvie · Nicolas
Joseph · Devi · Damien
Damien · Joseph
All
All
Workflow: Question→ Protocol→ Search→ Screen→ Extract & code→ Appraise→ Synthesise→ Report

Glossary · English → French

English Français English Français
Evidence synthesis Synthèse des connaissances Effect size Taille d’effet
Systematic review Revue systématique Forest plot Graphique en forêt (forest plot)
Systematic (evidence) map Carte systématique Heterogeneity Hétérogénéité
Scoping review Revue exploratoire Confidence interval Intervalle de confiance
Rapid review Revue rapide Prediction interval Intervalle de prédiction
Meta-analysis Méta-analyse Moderator Modérateur
Critical appraisal Évaluation critique Publication bias Biais de publication
Risk of bias Risque de biais Evidence gap Lacune de connaissances

In the online notebook

Each point of this session developed further, with the same data, more pitfalls and exercises with solutions:

literaturesynthesis.github.io/notebook

Further reading

Beillouin, D., T. Ben-Ari, and D. Makowski. 2019. Evidence map of crop diversification strategies at the global scale. Environmental Research Letters 14:123001.
Borenstein, M., L. V. Hedges, J. P. T. Higgins, and H. R. Rothstein. 2021. Introduction to meta-analysis. Second edition. Wiley, Chichester.
Collaboration for Environmental Evidence. 2022. Guidelines and standards for evidence synthesis in environmental management. Version 5.1. (A. S. Pullin, G. K. Frampton, B. Livoreil, and G. Petrokofsky, Eds.). Collaboration for Environmental Evidence.
Godlee, F., J. Smith, and H. Marcovitch. 2011. Wakefield’s article linking MMR vaccine and autism was fraudulent. BMJ 342:c7452.
Grant, M. J., and A. Booth. 2009. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal 26:91–108.
Gurevitch, J., J. Koricheva, S. Nakagawa, and G. Stewart. 2018. Meta-analysis and the science of research synthesis. Nature 555:175–182.
Haddaway, N. R., B. Macura, P. Whaley, and A. S. Pullin. 2018. ROSES RepOrting standards for systematic evidence syntheses: Pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environmental Evidence 7:7.
Harrer, M., P. Cuijpers, T. A. Furukawa, and D. D. Ebert. 2021. Doing meta-analysis with R: A hands-on guide. Chapman & Hall/CRC, Boca Raton.
Higgins, J. P. T., J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, and V. A. Welch, editors. 2024. Cochrane handbook for systematic reviews of interventions. Version 6.5. Cochrane.
James, K. L., N. P. Randall, and N. R. Haddaway. 2016. A methodology for systematic mapping in environmental sciences. Environmental Evidence 5:7.
Johnson, R., A. Watkinson, and M. Mabe. 2018. The STM report: An overview of scientific and scholarly publishing. Fifth edition. International Association of Scientific, Technical; Medical Publishers, The Hague.
Jones, S. K., A. C. Sánchez, S. D. Juventia, and N. Estrada-Carmona. 2021. A global database of diversified farming effects on biodiversity and yield. Scientific Data 8:212.
Koricheva, J., J. Gurevitch, and K. Mengersen, editors. 2013. Handbook of meta-analysis in ecology and evolution. Princeton University Press, Princeton.
Minx, J. C., M. Callaghan, W. F. Lamb, J. Garard, and O. Edenhofer. 2017. Learning about climate change solutions in the IPCC and beyond. Environmental Science & Policy 77:252–259.
Murad, M. H., N. Asi, M. Alsawas, and F. Alahdab. 2016. New evidence pyramid. Evidence-Based Medicine 21:125–127.
Nakagawa, S., D. W. A. Noble, A. M. Senior, and M. Lagisz. 2017. Meta-evaluation of meta-analysis: Ten appraisal questions for biologists. BMC Biology 15:18.
O’Dea, R. E., M. Lagisz, M. D. Jennions, J. Koricheva, D. W. A. Noble, T. H. Parker, and others. 2021. Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: A PRISMA extension. Biological Reviews 96:1695–1722.
Page, M. J., J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, and others. 2021. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 372:n71.
Senior, A. M., C. E. Grueber, T. Kamiya, M. Lagisz, K. O’Dwyer, E. S. A. Santos, and S. Nakagawa. 2016. Heterogeneity in ecological and evolutionary meta-analyses: Its magnitude and implications. Ecology 97:3293–3299.
Taylor, L. E., A. L. Swerdfeger, and G. D. Eslick. 2014. Vaccines are not associated with autism: An evidence-based meta-analysis of case-control and cohort studies. Vaccine 32:3623–3629.
Wakefield, A. J., S. H. Murch, A. Anthony, J. Linnell, D. M. Casson, M. Malik, M. Berelowitz, A. P. Dhillon, M. A. Thomson, P. Harvey, A. Valentine, S. E. Davies, and J. A. Walker-Smith. 1998. RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. The Lancet 351:637–641.
Woodcock, P., A. S. Pullin, and M. J. Kaiser. 2014. Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: A methodology. Biological Conservation 176:54–62.