Mon, Aug 15, 2022: 2:00 PM-2:15 PM
520E
Background/Question/MethodsModern meta-analyses have three basic goals: 1) quantify the average effect of a factor; 2) assess if this effect varies among studies; and 3) explore what other factors (e.g., traits of the organism, the environment or the experimental protocol) potentially explain the observed heterogeneity in effects. Central to achieving these goals then, is the meaning of “an effect”. This apparently simple problem is deceptively complex, and this complexity forms the core of this presentation. We start by reflecting on the 30-year history of effect sizes in ecology, by comparing effect size metrics used in the first decade of meta-analysis’ use in ecology with the effect size metrics used in the most recent decade. We then provide concrete examples of problems that underly the use of several common effect size metrics: Hedges’ d, the correlation coefficient, and log(response ratio), and illustrate how more explicit connections between ecological processes, response variables, and effect size metrics can facilitate (vs. obscure) progress in our discipline.
Results/ConclusionsHedges’ d (a measure of the difference between two groups expressed in standard deviation units defined by among-replicate variation), the correlation coefficient, and the log(response ratio) are the most common metrics of effect used in ecology today; indeed, the relative frequency of using these three metrics in ecological meta-analyses has changed very little over the past 30 years, despite research demonstrating their limitations. Further, although the interpretation of meta-analytic results depends strongly on the choice of an effect size metric, fewer than 20% of all meta-analyses provide conceptual justification for their chosen metric. We discuss the use of absolute vs. relative differences and standardized (by variance) vs. unstandardized effect size metrics, and put forth arguments for the conditions under which each might be best used (vs. yield misleading results). We illustrate ways effect size metrics can be more explicitly linked to a conceptual model of an ecological process to reduce the effects of extraneous variation and facilitate interpretation, with the hope that future evaluations will more often avoid some of the pitfalls highlighted in this talk.
Results/ConclusionsHedges’ d (a measure of the difference between two groups expressed in standard deviation units defined by among-replicate variation), the correlation coefficient, and the log(response ratio) are the most common metrics of effect used in ecology today; indeed, the relative frequency of using these three metrics in ecological meta-analyses has changed very little over the past 30 years, despite research demonstrating their limitations. Further, although the interpretation of meta-analytic results depends strongly on the choice of an effect size metric, fewer than 20% of all meta-analyses provide conceptual justification for their chosen metric. We discuss the use of absolute vs. relative differences and standardized (by variance) vs. unstandardized effect size metrics, and put forth arguments for the conditions under which each might be best used (vs. yield misleading results). We illustrate ways effect size metrics can be more explicitly linked to a conceptual model of an ecological process to reduce the effects of extraneous variation and facilitate interpretation, with the hope that future evaluations will more often avoid some of the pitfalls highlighted in this talk.