Learning objectives

By the end of this session, you will be able to:

  1. Explain what an effect size is and why each one comes with a sampling variance.
  2. Choose the right metric for your data: mean difference, Hedges’ g, log response ratio, log odds ratio, Fisher’s z.
  3. Compute them by hand and with metafor::escalc().
  4. Recognise the traps: small samples, near-zero means, missing SDs, unsafe conversions.

Studies do not speak the same language

Each of the 888 comparisons gives a mean in the intervention (intercropping) and a mean in the comparator (monoculture), each in the study’s own unit: per trap, per plant, per m²…

The raw differences (+115, −5.6, −0.035) cannot be compared. The ratio can: two studies halve the abundance, one barely changes it.

An effect size compares the intervention mean with the comparator mean on one common scale (direction and magnitude), and comes with its uncertainty.

Source: three comparisons from Jones et al. (2021).

Why not just use p-values?

A p-value mixes effect size and sample size: a tiny effect can be “highly significant”, a large one “not significant”.

Meta-analysis combines effect sizes and their uncertainty, not p-values (Nakagawa and Cuthill 2007).

Every study sees the truth through noise

θ̂k=θk+εkobserved=true+sampling error,Var(εk)=vk\hat\theta_k = \theta_k + \varepsilon_k \qquad \text{observed} = \text{true} + \text{sampling error}, \quad \text{Var}(\varepsilon_k) = v_k

Each effect size comes with its sampling variance vkv_k. In the meta-analysis this afternoon, each study is weighted by the inverse of its variance, wk=1/vkw_k = 1/v_k: precise studies count more. (In a random-effects model, the weight also includes the real variation between studies, τ2\tau^2: wk=1/(vk+τ2)w_k = 1/(v_k + \tau^2).)

A decision tree

Continuous outcome, two groups
means, SD, n
  • Same unit everywhere → Mean difference (MD)
  • Scores, indices, values that can be negative → standardised mean difference (Hedges' g)
  • Quantities: counts, biomass, yield → log response ratio (lnRR)
Binary outcome, two groups
counts in a 2×2 table
  • Presence/absence, survival, infestation → log odds ratio (or log risk ratio)
Relationship between two continuous variables
correlation r, n
  • e.g. yield vs landscape complexity → Fisher's z

Test: does “twice as many” make sense?
Yes (aphids, yield) → lnRR.
No → g: damage scores, soil quality indices, assays that differ between studies, values that can be negative (change in soil C stock).

Our running example → lnRR. Fix the metric in the protocol, before seeing the results.
Show of hands: which metric for your own data?

Our worked example

Aphids on fennel, intercropped with cotton vs fennel monoculture (Fernandes et al. 2013), one comparison of our dataset:

Intercropping (T) Monoculture (C)
Mean 8.37 23.98
SD 2.85 5.46
n 4 4

Convention for the whole week: T (intervention) in the numerator / first, C (comparator) second. A negative effect = fewer pests with intercropping.

The log response ratio (lnRR)

lnRR=ln(X‾TX‾C)\ln RR = \ln\left(\frac{\bar X_T}{\bar X_C}\right)

vlnRR=SDT2nTX‾T2+SDC2nCX‾C2v_{\ln RR} = \frac{SD_T^2}{n_T \bar X_T^2} + \frac{SD_C^2}{n_C \bar X_C^2}

(Hedges et al. 1999)

Back to percent for reporting:

change=100×(elnRR−1)%\text{change} = 100 \times \left(e^{\ln RR} - 1\right)\%

Why the log? The ratio is asymmetric (halving = 0.5, doubling = 2); its log is symmetric (−0.69, +0.69) and closer to normal.

Why ecologists like it Unit-free and proportional: “intercropping reduces aphids by 65%” speaks to agronomists — and its value does not depend on the SD (only its variance does).

Exercise 1 · By hand (pairs, 8 min)

Calculator ready

With the aphid example (T: 8.37 ± 2.85, n = 4; C: 23.98 ± 5.46, n = 4), compute lnRR, its variance, and the % change.

lnRR = ln(8.37 / 23.98) = -1.053
v = 2.85² / (4 × 8.37²) + 5.46² / (4 × 23.98²) = 0.0419 (SE = 0.205)
Change = 100 × (e-1.053 − 1) = -65%

Now let R do it (live coding, 5 min)

Predict first What value of yi should R return for our aphid example?

library(metafor)
# ic <- read.csv("intercropping_biodiversity.csv")      # running example

dat <- escalc(measure = "ROM",                           # lnRR ("ratio of means")
              m1i = b_mean_t, sd1i = b_sd_t, n1i = b_n_t,  # T = intercropping
              m2i = b_mean_c, sd2i = b_sd_c, n2i = b_n_c,  # C = monoculture
              data = ic)
# other metrics: measure = "SMD" (Hedges' g), "OR", "ZCOR"

dat[dat$es_id == 2011, c("es_id", "yi", "vi")]          # our worked example

    es_id      yi     vi 
137  2011 -1.0526 0.0419 
c(NA_yi = sum(is.na(dat$yi)), vi_zero = sum(dat$vi == 0, na.rm = TRUE))  # what to deal with
  NA_yi vi_zero 
    156      46
  • yi = effect size, vi = sampling variance: the input of every meta-analysis (rma(), this afternoon).
  • Zero mean → NA; zero SD → vi = 0 (“infinite” precision!): count both before rma().

Hedges’ g, step by step

Idea: express the difference in standard-deviation units.

d=X‾T−X‾CSDpooledd = \frac{\bar X_T - \bar X_C}{SD_{pooled}}

Useful when studies use different scales (scores, indices), where “twice as many” means nothing.

Small samples inflate d → multiply by a correction J < 1: that is Hedges’ g (Hedges 1981).

With our aphids:

1. Difference: 8.37 − 23.98 = -15.61 aphids

2. Pooled SD (equal n: average the variances): √((2.85² + 5.46²) / 2) = 4.36

3. d = -15.61 / 4.36 = -3.58 standard deviations

4. With 4 + 4 plants, J = 0.87 → g = -3.12

escalc(measure = "SMD") does all of this — and the variance. You never compute g by hand in practice.

Trap 1 · Means close to zero

Your call

156 comparisons in our data have a zero mean. What do you do with them for a lnRR?

Exact zeros: no lnRR (and most also have SD = 0, so no g). Report them; test the conclusion with and without.

Near zero: lnRR is biased. Geary’s rule: each mean ≥ 3 SE from zero → 28% of our usable comparisons fail it.

What to do

  • Near zero or small n: bias-corrected lnRR (Lajeunesse 2015) — formula and R code on the handout.
  • Never add an arbitrary constant (+1) without testing its influence.

Trap 2 · Missing or disguised SDs

The common case: a standard error SD = SE × √n
553 of our 888 comparisons were reported with a SE.

Also possible: from a 95% CI, or from a median and IQR (Wan et al. 2014) — see the formula sheet.

215 comparisons (24%) have no usable SD.
Never set it to 0 or drop silently. Robust fix for lnRR: compute the sampling variance from the typical coefficient of variation of the other studies (CV = SD/mean, averaged with weights n, then squared) (Nakagawa et al. 2023), after screening implausible CVs (Wednesday 16:15). Not possible for zero means.

Traps 1 and 2 together: 224 of 888 comparisons (25%) are unusable as such for lnRR. This afternoon we set them aside — and say so.

Trap 3 · lnRR and g: same direction, different magnitudes

The 664 usable comparisons, computed both ways:

  • Same direction always (both follow X‾T−X‾C\bar X_T - \bar X_C)
  • Similar ranking (Spearman ρ = 0.84)
  • But very different magnitudes: g explodes when the SD is tiny

g depends on the SD (trap size, plot size, subsamples); lnRR does not. Never mix both in one analysis, and do not convert lnRR ↔︎ g.

Other kinds of data, in one slide

Yes/no outcomes Plants infested or not, seedlings alive or dead → a 2×2 table of counts → log odds ratio (or log risk ratio, the log of the ratio of proportions).
escalc("OR", ai=, bi=, ci=, di=)

Correlations e.g. habitat cover vs pest control → Fisher’s z.
Pool z, not r, then back-transform.
escalc("ZCOR", ri=, ni=)

Conversions between metrics Possible (log OR → d, r → d), but only for a minority of studies: flag them and run a sensitivity analysis without them. Never lnRR ↔︎ g. Formulas in the formula sheet (Borenstein et al. 2021 chap. 7).

Quick quiz

1. Earthworm biomass (g/m²) under conservation vs conventional tillage, studies in many different units. Metric?

lnRR — positive, proportional, unit-free.

2. lnRR = −0.36. Change in %?

100 × (e−0.36 − 1) = −30%.

3. Proportion of fields where a weed is present, with vs without cover crop. Metric?

log odds ratio (or log risk ratio).

4. Why use g rather than d?

d overestimates |effect| in small samples; g corrects it.

Going further

All formulas (with the exact J, bias-corrected lnRR and its variance, conversions and their variances): see the A4 formula sheet handout.

  • Variability as an outcome: lnCVR, the log ratio of coefficients of variation (Nakagawa et al. 2015).
  • Paired / before–after designs (SMCC, ROMC in escalc) need the correlation between measurements.
  • Several treatments vs one control create dependent effect sizes → multilevel models, this afternoon.

Can we just average the percentages?

Same 440 pest comparisons, two “averages” (unweighted):

  • mean of the % changes: +33%
  • back-transformed mean lnRR: -25%

Why so different? A % change is asymmetric: halving gives −50%, doubling gives +100%. A few large increases pull the mean of percentages up. Average on the log scale, back-transform at the end.

Which effect size?

Pairs, 3 minutes

For each case, which effect size would you use?

  1. Presence or absence of a pest in 20 intercropped and 20 monoculture fields.
  2. Wheat yield in t/ha, same unit in every study.
  3. Species richness, with several monocultures at zero.
  4. Correlation between landscape complexity and pest abundance across farms.

1 · log odds ratio ("OR") · 2 · mean difference ("MD") or lnRR if you want a relative effect · 3 · SMD ("SMD"), lnRR is undefined · 4 · Fisher’s z ("ZCOR")

In the online notebook

Each point of this session developed further, with the same data, more pitfalls and exercises with solutions:

literaturesynthesis.github.io/notebook

References

Borenstein, M., L. V. Hedges, J. P. T. Higgins, and H. R. Rothstein. 2021. Introduction to meta-analysis. Second edition. Wiley, Chichester.
Fernandes, F. S., F. S. Ramalho, W. A. C. Godoy, J. K. S. Pachu, R. B. Nascimento, J. B. Malaquias, and J. C. Zanuncio. 2013. Within plant distribution and dynamics of hyadaphis foeniculi (Hemiptera: Aphididae) in field fennel intercropped with naturally colored cotton. Florida Entomologist 96.
Hedges, L. V. 1981. Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics 6:107–128.
Hedges, L. V., J. Gurevitch, and P. S. Curtis. 1999. The meta-analysis of response ratios in experimental ecology. Ecology 80:1150–1156.
Jones, S. K., A. C. Sánchez, S. D. Juventia, and N. Estrada-Carmona. 2021. A global database of diversified farming effects on biodiversity and yield. Scientific Data 8:212.
Lajeunesse, M. J. 2015. Bias and correction for the log response ratio in ecological meta-analysis. Ecology 96:2056–2063.
Nakagawa, S., and I. C. Cuthill. 2007. Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Reviews 82:591–605.
Nakagawa, S., D. W. A. Noble, M. Lagisz, R. Spake, W. Viechtbauer, and A. M. Senior. 2023. A robust and readily implementable method for the meta-analysis of response ratios with and without missing standard deviations. Ecology Letters 26:232–244.
Nakagawa, S., R. Poulin, K. Mengersen, K. Reinhold, L. Engqvist, M. Lagisz, and A. M. Senior. 2015. Meta-analysis of variation: Ecological and evolutionary applications and beyond. Methods in Ecology and Evolution 6:143–152.
Wan, X., W. Wang, J. Liu, and T. Tong. 2014. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Medical Research Methodology 14:135.