Early warning signals and the prosecutor’s fallacy

So continuing on from the last blog, might our warning systems on tipping points have been hindered?

Early warning systems have been proposed to forecast the possibility of a critical transition, such as the eutrophication of a lake, the collapse of a coral reef or the end of a glacial period. Because such transitions often unfold on temporal and spatial scales that can be difficult to approach by experimental manipulations, research has often relied on historical observations as a source of natural experiments.

Here, Boettiger and Hastings, examine a critical difference between selecting systems for study based on the fact that we have observed a critical transition and those systems for which we wish to forecast the approach of a transition. This difference arises by conditionally selecting systems known to experience a transition of some sort and failing to account for the bias this introduces - a statistical error known as the prosecutor’s fallacy. The term is however most often associated with prosecuting lawyers arguing for the guilt of a defendant in a criminal trial.

By analyzing simulated systems that have experienced transitions purely by chance, they reveal an elevated rate of false-positives in common warning signal statistics.

The attempts to detect early warning signs for critical transitions are based on the concept of deteriorating environment as embodied in a changing parameter (Scheffer et. al., 2009), which is a different kind of transition than the alternative of stochastic system (i.e. non- deterministic, so probabilities are used to work out potential outcomes) in an environment that is otherwise constant and exhibiting no directional change. When trying to use historical data to understand critical transitions, we often do not know which category, changing environment or simply chance, an observed large change falls into, which leads to uncertainty.

Boettiger and Hastings have shown here that systems that undergo rare sudden transitions owing to chance look statistically different from their counterparts that do not, even though they are driven by the same stochastic process (non-deterministic).

In particular, such conditionally selected examples are more likely to show signs associated with an early warning of an approaching tipping point, such as increasing variance or increasing autocorrelation, as measured by Kendall’s  (used to measure the association between two measured quantities).

This increases the risk of false positives - cases in which a warning signal being tested appears to have successfully detected an underlying change in the system leading to a tipping point, when in fact the example comes instead from a stable system with no underlying change in parameters.

It does seem tempting to argue that this bias towards positive detection in historical examples is not problematic each of these systems did indeed collapse; so the increased probability of exhibiting warning signals could be taken as successful detection. Unfortunately this isn’t the case. At the moment the forecast is made, these systems are not likely to transition, because they experience a strong pull towards the original stable state. As the system gets farther from its stable point, it is more likely to draw a random step that returns it towards the stable point. However of course there is also the chance that it will continue away from its original stable point, thus any systems that do cross a tipping point would do so rather quickly.

The authors do also go on to demonstrate a model-based approach that is less subject to this bias than those more commonly used in summary statistics as well as highlight the fact that experimental studies with replicates avoid this pitfall entirely – largely through running many models and improving knowledge of the system to remove bias.  However I think that’s enough for now and this new method is still to be fully evaluated and/or used by the scientific community.

Read the full paper at:  http://rspb.royalsocietypublishing.org/content/279/1748/4734.full.pdf+html

Reference:

Boettiger, C. and Hastings, A. (2012) Proc. R. Soc. B  vol. 279, no. 1748. 4734–4739.

Scheffer, M. et al. 2009 Early-warning signals for critical transitions. Nature 461, 53–59. (doi:10.1038/nature08227)

Global climate change: Did we pass a tipping point?

I’ve been hearing cries desperate for some serious stats, so here you are, we can look to the numbers for help….

Systems that have an internal feedback mechanism can often principally behave in a non-linear fashion. Given the complex nature of climate on earth, one can come to expect such actions.

Abrupt shifts on climate conditions were first discovered at high latitudes in the Northern Hemisphere when investigating Greenland ice cores. On long timescales of roughly 100,000 years the seemingly fast changing glacial-interglacial cycles, and on the relatively shorter time scale of approximately 2,000 years the occurrence of Dansgard-Oeschger events (rapid warming followed by gradual cooling) are striking evidence for such non-linear behaviour.

Nevertheless it is common practice to publish linear trends from many global data sets without considering their relevance. The classical example is the often cited (IPCC, 2007) global warming trend of 0.05°C/decade for air temperature during the last 150 years.

Using a structural test based on the F statistics (not important for us but it’s a test of data with an F distribution under a null hypothesis) Stips and Lilover were able to test for ‘break points’.  They show that most analysed global climate time series contain statistical significant changes (breaks in the mean or in the slope of linear regression). Thus revealing the existence of break points for most investigated parameters of relatively recent climate science. Breakpoints which are detected at a comparable time in many different regional and global climate variables are a strong indication for the existence of a regime shift in the state of the climate, and so could be very telling.

Fig. 1 Time series of global annual air temperature anomalies. The straight bold black line is the trend from the linear regression, which does not provide a satisfactory fit to the data. The test for breakpoints in the trend finds a significant breakpoint at 1976. The very different slope of the regression lines (approximate factor ¹⁰ different) for the data before 1976 and after 1976 (green lines) is clearly visible.

Looking above the time series of global land air temperatures plotted in figure 1 together with the calculated linear trend of -0.05k/decade (black solid straight line) which is statistically significant. The inadequacy of the above linear trend model over such a long period is obvious when looking at the difference of the observed 2009 mean temperature to that estimated from the calculated linear trend. The trend calculation gives an underestimation of about 48% (~0.36°C to low, compared to a total change of 0.75°C).  The most significant breakpoint occurred in 1976, possibly signaling a significant shift on our climate given its magnitude. A keynote of this is that the trends prior to and post 1976 were found to vary by a factor of 10 (see the green lines). The trend is 0.024k/decade before 1976 and 0.218k/decade thereafter, which is actually about 4 times the mentioned overall IPCC trend! Thus there is a high degree of uncertainty if one were to extend this time series onto the future.

A similar inspection of several other global atmospheric and oceanographic data time series provides reasonable doubt concerning the validity of the linear model for century long time series. It should be clear that when applying any statistical method to real world data the underlying process dynamics have to be considered, in order to avoid misinterpretation of the statistical results.

In summary the statistical tests reveal a significant linear trend and significant break points in the mean as well as the trend of temperature data. Therefore based only on the statistical results we cannot decide if the analysed data represent a linear or non-linear system process. Any projection of future global air temperature from past records will depend strongly on our decision of the underlying process model, as displayed by the enormous difference between trends calculated with and without consideration of breakpoints.  Reconstructed prehistoric climate indices provide clear evidence for the non-linear behaviour of the climate system and for the existence of abrupt changes (tipping points). Therefore calculation of linear trends can only be justified for sufficiently short intervals. The final conformation for the real process dynamics has to come from an accurate and complete geophysical description of the climate system, a difficult and complex task still to be accomplished.

To answer the question posed for the paper, and from the above, the existence of the major late seventies breakpoint could be considered as a sign for a state change in global air temperature pointing to a change in the climate system – it’s quite likely the earth did pass a tipping point.

This paper appears in: Baltic International Symposium (BALTIC), 2012 IEEE/OES, Issue

Date :8-10 May 2012, Written by: Stips, A.K.; Lilover, M.

Reference List

Bai, J. and Perron, P. (1994). “Least Squares Estimation of a shift in Linear Processes”, Journal of Time Series Analysis, 15, 453-472.

Bai, J. and Perron, P. (1998). :Estimating and Testing Linear Models with Multiple Structural Changes”, Econometrica, 66, 47-78.

IPCC, “Summary for `Policymakers. In: Climate Change 2007: The Physical Science Basis. Contribution of Working Group 1 to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change”, [Solomon, S., D. Qin, M. Manning, Z. Chen, M. Marquis, K.B. Averyt, M. Tignor and H.L. Miller (eds.)]. Cambridge University Press, Cambridge, United Kingdom and New Yprk, NY, USA, 2007.