A number of recent papers have highlighted the idea of using iterative forecasting in a hypothesis testing framework as a way of accelerating learning. The proposed cycle involves using models (which embody our current hypotheses) to make out-of-sample predictions into the future, collecting data, validating results, and then assimilating this new data into models to update parameters and/or state variables. It is argued that this cycle closely mirrors the cyclic nature of the scientific method, places particular emphasis on having specific, refutable predictions, and that the validation against yet-to-be-collected data acts as preregistration and guards against overfitting. But how do you do this in practice, and how do tasks like model selection change in an iterative setting? We explore these issues using both a simulated data experiment (true model and parameters known) and a reanalysis of a vegetation phenology forecast. Specifically, model selection was conducted in these case studies both using an iterative approach and using standard post-hoc model fitting, model selection, and cross-validation. For both we used a Bayesian state-space framework, with the simulation exploring a series of dynamic linear models of increasing complexity and the phenology modeling using nonlinear logistic models with different meteorological covariates.
Results/Conclusions
For choosing among competing models, we employ an out-of-sample predictive loss metric, which aims to balance validation error and total predictive uncertainty (parameter, initial condition, driver, and process errors). In an iterative context, all predictive uncertainties are initially at their maximum (prior) values, and thus the simplest model (random walk) initially dominates. After 10-20 time steps, parameter and process error estimates begin to converge for simple models and we are able to begin selecting among alternative model structures and covariates. Model selection takes longer for the phenological models, where the key information comes from once-a-year-transitions, and thus choosing between hypotheses requires multiple years of data regardless of measurement frequency. In addition to simple iterative model-selection, we also assessed the ability of iterative validation metrics to help diagnose potential covariates and models structural errors. Finally, posthoc data analysis consistently selected for more complex model structures, and had a higher rate of overfitting than the iterative approach. Similarly, we found that multi-model iterative approaches need to retain alternative models beyond the point when a candidate model is ‘significant’ to prevent the premature ‘chance’ convergence on incorrect models.