Hill numbers have become standard for measuring diversity in ecology. Depending on the value of a control parameter, Hill numbers reflect species richness, Shannon’s entropy, or Simpson’s diversity. Here, we discuss an alternative interpretation of Hill numbers as means of species rarity (the reciprocal of relative abundance) in a community. In this interpretation, the control parameter moves the index from arithmetic to geometric to harmonic means. Thus, Hill numbers differ in the scale on which species rarity is considered, but always weights species the same way (by their abundance) regardless of the scale being used. This interpretation is distinct from the common idea that the Hill parameter controls how rare vs. common species are “weighted.” The mean rarity perspective suggests a novel approach to estimating true diversity from a sample, using the fact that the sample mean is an unbiased estimate of the population mean. Although how to estimate a species’ rarity from sample data remains unsolved, we test the promise of this idea using what we call God’s estimator: All-knowing, God can plug in the true population parameters for each species. Nevertheless, maybe for fun, God chooses to compute the sample mean, weighting species’ true rarities by their sampled frequency.
Results/Conclusions
We compared Hill diversity estimates using simulated communities with ~10^5 – 10^7 individuals, 30-150 species, a variety of species-abundance distributions, and samples of 100-300 individuals. We compared God’s estimator, which uses species frequencies in the sample of 100-300 along with their true population rarities, to the naïve estimator, which estimates rarities directly from species frequencies in the small sample, and also to published non-parametric Hill number estimators. God’s estimator was both less biased and more accurate for all simulated communities, sample sizes, and values of the Hill control parameter. God’s estimator always had lower root-mean-squared error (RMSE), and in the case of species richness with highly diverse communities, the RMSE for God’s estimator remained low (within a few species) while the RMSE for the naïve and non-parametric estimators were huge (dozens of species in our simulations). This is expected: God’s estimator uses information not available to other methods (nor to ecologists in reality). However, these results suggest that improved heuristic estimates of species rarities might enable enable better diversity estimates, and pave a route to more principled confidence intervals for Hill numbers.