Early warning signals and the prosecutor’s fallacy

So continuing on from the last blog, might our warning systems on tipping points have been hindered?

Early warning systems have been proposed to forecast the possibility of a critical transition, such as the eutrophication of a lake, the collapse of a coral reef or the end of a glacial period. Because such transitions often unfold on temporal and spatial scales that can be difficult to approach by experimental manipulations, research has often relied on historical observations as a source of natural experiments.

Here, Boettiger and Hastings, examine a critical difference between selecting systems for study based on the fact that we have observed a critical transition and those systems for which we wish to forecast the approach of a transition. This difference arises by conditionally selecting systems known to experience a transition of some sort and failing to account for the bias this introduces - a statistical error known as the prosecutor’s fallacy. The term is however most often associated with prosecuting lawyers arguing for the guilt of a defendant in a criminal trial.

By analyzing simulated systems that have experienced transitions purely by chance, they reveal an elevated rate of false-positives in common warning signal statistics.

The attempts to detect early warning signs for critical transitions are based on the concept of deteriorating environment as embodied in a changing parameter (Scheffer et. al., 2009), which is a different kind of transition than the alternative of stochastic system (i.e. non- deterministic, so probabilities are used to work out potential outcomes) in an environment that is otherwise constant and exhibiting no directional change. When trying to use historical data to understand critical transitions, we often do not know which category, changing environment or simply chance, an observed large change falls into, which leads to uncertainty.

Boettiger and Hastings have shown here that systems that undergo rare sudden transitions owing to chance look statistically different from their counterparts that do not, even though they are driven by the same stochastic process (non-deterministic).

In particular, such conditionally selected examples are more likely to show signs associated with an early warning of an approaching tipping point, such as increasing variance or increasing autocorrelation, as measured by Kendall’s  (used to measure the association between two measured quantities).

This increases the risk of false positives - cases in which a warning signal being tested appears to have successfully detected an underlying change in the system leading to a tipping point, when in fact the example comes instead from a stable system with no underlying change in parameters.

It does seem tempting to argue that this bias towards positive detection in historical examples is not problematic each of these systems did indeed collapse; so the increased probability of exhibiting warning signals could be taken as successful detection. Unfortunately this isn’t the case. At the moment the forecast is made, these systems are not likely to transition, because they experience a strong pull towards the original stable state. As the system gets farther from its stable point, it is more likely to draw a random step that returns it towards the stable point. However of course there is also the chance that it will continue away from its original stable point, thus any systems that do cross a tipping point would do so rather quickly.

The authors do also go on to demonstrate a model-based approach that is less subject to this bias than those more commonly used in summary statistics as well as highlight the fact that experimental studies with replicates avoid this pitfall entirely – largely through running many models and improving knowledge of the system to remove bias.  However I think that’s enough for now and this new method is still to be fully evaluated and/or used by the scientific community.

Read the full paper at:  http://rspb.royalsocietypublishing.org/content/279/1748/4734.full.pdf+html

Reference:

Boettiger, C. and Hastings, A. (2012) Proc. R. Soc. B  vol. 279, no. 1748. 4734–4739.

Scheffer, M. et al. 2009 Early-warning signals for critical transitions. Nature 461, 53–59. (doi:10.1038/nature08227)