Friday, August 7, 2009: 9:00 AM
Grand Pavillion III, Hyatt
Background/Question/Methods Although many techniques exist to model species potential geographic distributions using occurrence records and environmental variables in a GIS context, species with few known occurrence records largely have evaded rigorous evaluation. Recently, a delete-1 jackknife procedure was proposed by other authors to assess significance (Does the model predict test localities better than random?) for models of such species. We extend this approach to consideration of two measures of performance (omission rate and Area Under the Curve of the Receiver Operating Characteristic plot, AUC/ROC), considering both the average and variance of each (correcting for non-independence of iterations). We conduct our tests for maximum entropy (Maxent) models of the spiny pocket mice Heteromys australis and H. teleus in Ecuador and southwestern Colombia. Models vary according to the constraints (“feature classes”) used in the model and the regularization value (level of protection against overfitting) employed. Hence, we run models with varying combinations of feature class (hinge, linear, linear/quadratic, and hinge/linear/quadratic) and regularization multiplier (0.5–2.0). We then use the measures of performance to determine whether “tuning” models for these species (determining the best combination of feature class and regularization multiplier) leads to higher performance than the default settings.
Results/Conclusions Performance varied greatly among combinations of feature class and regularization multiplier, with the best non-default settings generally superior to default ones. For H. teleus, omission rate was lowest (best) for hinge features and high regularization. The highest (best) AUC values for that species corresponded to linear features with low and intermediate regularization; and to hinge features and high regularization. Similarly, for H. australis, the lowest omission rate corresponded to hinge features and high regularization, and hinge features consistently produced the highest AUC. Hence, for both species, the optimal overall performance (based on both low omission rate and high AUC) was for hinge features and high regularization (superior to the default settings of linear features and intermediate regularization). These optimal (tuned) settings also showed low-to-intermediate variance in each measure of performance. In addition, subjective expert opinion judged maps of the species’ predicted potential distributions obtained with optimal settings to be superior to predictions made using default settings. In summary, at least for species with few occurrence records, researchers may be able to obtain better predictions by tuning rather than using default settings. Many aspects of tuning experiments can be automated (run by code), greatly facilitating their use.
Results/Conclusions Performance varied greatly among combinations of feature class and regularization multiplier, with the best non-default settings generally superior to default ones. For H. teleus, omission rate was lowest (best) for hinge features and high regularization. The highest (best) AUC values for that species corresponded to linear features with low and intermediate regularization; and to hinge features and high regularization. Similarly, for H. australis, the lowest omission rate corresponded to hinge features and high regularization, and hinge features consistently produced the highest AUC. Hence, for both species, the optimal overall performance (based on both low omission rate and high AUC) was for hinge features and high regularization (superior to the default settings of linear features and intermediate regularization). These optimal (tuned) settings also showed low-to-intermediate variance in each measure of performance. In addition, subjective expert opinion judged maps of the species’ predicted potential distributions obtained with optimal settings to be superior to predictions made using default settings. In summary, at least for species with few occurrence records, researchers may be able to obtain better predictions by tuning rather than using default settings. Many aspects of tuning experiments can be automated (run by code), greatly facilitating their use.