Meta-analysis –a quantitative way of summarizing information from different studies– is one of the cornerstones of science. In ecology, the use of meta-analysis has been growing exponentially since the 1990s, but the quality of ecological meta-analyses has lagged behind other disciplines, such as medicine. The methodological literature is inaccessible for many ecologists, and software packages often hide important aspects of the analysis that can affect the results, particularly given unique characteristics of ecological meta-analyses (e.g., low replication, high among study heterogeneity, and low number of studies). Our goals are to: 1) evaluate characteristics of published data commonly used in ecological meta-analysis, 2) compare traditional and Bayesian meta-analytic methods for synthesizing such data, and 3) evaluate which method is superior, given characteristics typical of ecological data. We used simulation experiments to compare the performance of both methods applied to a random effects meta-analysis involving different number of replicates, within versus among study variation, and number of studies. To evaluate performance of each method, we assessed uncertainty of the estimated overall effect size, coverage (whether or not the 95% CI contained the “truth”), and bias (difference between the estimate and the truth) relative to the true values used to simulate the data.
Results/Conclusions
In our review of published ecological meta-analyses, the number of within study replicates varies greatly, but the mode and median of this distribution are very low (usually <10). Similarly, the average number of studies used to estimate an average effect size is low, but highly variable (usually <25). Our simulations showed that traditional and Bayesian meta-analytic methods yield overall effect sizes with high uncertainty when the number of studies synthesized is approximately <30, which is more pronounced when the number of within study replicates is also low (e.g., <5). In most cases, low coverage produced by traditional meta-analyses is easily corrected by application of the Knapp-Hartung-Sidik-Jonkman correction for confidence intervals. In general, the performance of both methods was similar for the same combinations of factors. Bayesian methods, however, are more flexible by allowing for different model formulations, reflecting different assumptions about how the data arise, but they also require greater involvement of the user and are more computationally intensive. Bayesian methods also appear sensitive to the choice of the prior for the among-study variance when the number of studies is low. These results highlight the need to understand the choices inherent in using meta-analytic software in an ecological context.