An introduction to meta-analyses and systematic reviews

September 2024

Damien Beillouin
CIRAD-Hortsys

Focus….

Step back

Again…

Let’s play

https://etc.ch/TWnF

At the global scale

What started the rumors?

Mis-information and meta-analyses

Misinformation:

  • Spreads via confirmation bias, and misinterpretation of studies.

  • Harms public understanding and policy decisions.

Meta-analyses and systematic reviews

  • Counters isolated, misleading findings.

  • Strengthens scientific credibility.

  • Informs public policy with solid evidence.

Let’s play (again)

State of the evidence

Contradictory evidence and meta-analyses

Contradictory evidence

  • Results from variations in study design, sample size, or methodology.

  • Leads to uncertainty in scientific conclusions and decision-making.

Meta-analyses and systematic reviews

  • based on the ‘means’ of many study -> increase statistical power

Let’s play (again?!)

Number of citations included in IPCC reports

source: Minx et al., 2017

Number of papers in ecology

source: Gurevitch et al., 2018

Number of papers and meta-analyses

Number of papers

  • over 7M papers published each year

Meta-analyses and systematic reviews

  • Meta-analyses synthesized 2 to XXXX papers!

A need for robust evidence

Some big projects/labs

Some big projects/labs

Some big projects/labs

Some big projects/labs

Results: Increase in # meta-analyses

But of variable quality

more info: shinyApp

A lot of confusion in terms and methods to synthesized data

  • Expert consultation, focus group, …
  • Rapid evidence assessment, non-systematic review
  • Systematic maps
  • meta-analyses
  • ….

A (rapid) typology of methods

Systematic (evidence) maps

  • Purpose: Provide an overview or ’landscape of knowledge” on a given issue

Systematic (evidence) maps

  • Purpose: Provide an overview or ’landscape of knowledge” on a given issue

Stengths of evidence maps

 

1. Visual representation of the knowledge accumulated/gaps

(i.e. no need to read hundreds of primary studies!)

2. Typology/categorization of the research

(often term and definition varies a lot to represent a same phenomena/practice )

3. Offer a foundation for further, more focused research synthesis

(a first step for a new meta-analyis)

4. Political/scientific agenda for future research

Weakness of evidence maps

 

1. Only descriptive - no analyses

(i.e. do not inform on the effectiveness of the interventions)

2. The results could be presented in meta-analyses

(sometimes)

Meta-analyses

  • Purpose: Provide an estimate of an effect, identify source of heterogeneity, rank the moderators of the effect

Stengths of meta-analyses

 

1. See the forest for the trees

(i.e. see effect/relationships that might not be visible in individual studies!)

2. Synthesize knowledge on controversial issues

(find average effect while trying to minimize bias)

3. Can compare apple and oranges (to some degree)

(meta-analyses are specifically design to deal with heterogeneity)

4. Examine reasons for variation

(to some degree)

Weakness of meta-analyses

 

1. See the forest for the trees

(i.e. mask specific pattern/local effect?)

2. Can not overcome bias

(fe.g. publication bias)

3. Can compare apple and oranges (to some degree)

(combine things that should not be combined)

4. Not representative

(study population =/= population of interest)

5. Not able to examine causality

What you will do this week

What you will do this week

A training based on international golden standards

An introduction to meta-analyses and systematic reviews