Ignorance is bliss (sometimes)

Ecologists (and lots of other people) often say that the world, or some feature of it, is 'random' or 'stochastic'. But what exactly does that mean?

One view is that randomness is real; some features of the world are inherently probabilistic. Quantum mechanics is the paradigmatic example here, but that doesn't mean there aren't others. An alternative view is that calling something 'random' is shorthand for our ignorance. If we knew enough about the precise shape of a coin, the force with which it was flipped, the movement of the surrounding air, etc., we could accurately predict the outcome of any particular coin flip, which is deterministic. But we don't have that information, so we pretend that coin flipping is a random process and make probabilistic statements about the expected aggregate outcome of many flips.

Does the distinction between these two views matter for ecologists? It's tempting to say no. In practice there's no possibility we'll ever have enough information to predict the roll of a die, so we lose nothing by treating it as random. No less an authority than Sewell Wright was of this view. But I'm going to suggest that's incorrect; I think ecologists do need to decide whether they think randomness is real, or merely determinism disguised by our ignorance. And I'll further suggest that the appropriate choice can vary from case to case and is only sometimes dictated by empirical facts.

If apparent randomness is just ignorance of relevant information, then when we learn new information the apparent randomness of events should decline. This happens whenever you add an additional predictor variable to a statistical model, increasing explained (deterministic) variation and reducing unexplained (random) variation. A personal favorite example of mine is recent work on the 'decision' by a bacteriophage as to whether to lyse its bacterial host. This decision had been regarded as a paradigmatic example of a probabilistic biological process. But it turns out that the decision is actually quite (although perhaps not entirely) deterministic, and depends on the size of the host cell. Cell size varies, and phage decision making looks random if you don't account for that variation. This is a specific example of a general principle: if some process (like the lysis decision) is not random with respect to some property or outcome of interest (like cell size), then it's simply false to treat that process as random.

But is it always a good idea to try to minimize apparent randomness by incorporating all relevant information? In ecology, Jim Clark has argued as much (if I understand him correctly). But I'm not so sure I agree. If calling something random is merely to statistically summarize the net effects of various unknown deterministic processes, well, summaries are really useful. Think for instance of genetic drift, and its ecological equivalent, demographic stochasticity. Genetic drift and demographic stochasticity arise from random variation in the birth and death rates of individuals that is independent of their phenotypes and other properties, and so would occur even if all individuals were otherwise identical. I'm happy to stipulate that, if we knew enough, much or even all of this apparent randomness could be explained away. But why would we want to explain it away? What would we gain? I'd argue that we'd actually lose a lot. We'd be replacing the generally-applicable concepts of genetic drift and demographic stochasticity (and the associated well-developed, highly elegant, and well-tested mathematical theory) with a stamp collection of inherently case-specific, and hugely complex, deterministic causal stories. The complex deterministic causal factors generating apparently-random variation in the birth and death rates of, say, different E. coli genotypes in a laboratory culture have nothing to do with the complex deterministic causal factors generating apparently-random variation in the birth and death rates of, say, introduced rats on a marine island. The important thing is that deterministic causal factors in both cases have apparently-stochastic consequences described by models of genetic drift and demographic stochasticity. Laplace's demon, which has perfect information about the position and movement of every particle of matter in a deterministic universe, would see no randomness—thereby making it completely ignorant about one of the most important and best-confirmed concepts in all of ecology and evolution (see here for more on this).

And while Laplace's demon is a mere philosopher's dream, even trying to emulate it has its pitfalls. In Do lemmings commit suicide? Beautiful hypotheses and ugly facts, population ecologist Dennis Chitty describes his career-long unsuccessful struggle to identify the causes of population cycles in small mammals. His lack of success is almost certainly attributable, at least in part, to his search for a deterministic sequence of causal events that always drives population cycles. Observations that a particular causal factor was apparently weak or absent in some cases (or even absent at one particular time for one particular population of one particular species) repeatedly caused him to modify or abandon his causal hypotheses. In contrast, modern stochastic dynamics has been quite useful for inferring the causes of population fluctuations (e.g., Henson et al. 2002 Oikos). See here for further discussion of the pitfalls of insisting on an overly-detailed 'low-level' description of one's study system.

Bottom line: if randomness is ignorance, sometimes ignorance is bliss.

p.s. The distinction between real and merely apparent randomness crops up outside of science too, for instance in professional sports. Traditionally, events in sports—such as who wins and who loses—often are explained by appeal to specific details associated (or putatively associated) with the event. Perhaps the winning team exhibited a stronger 'will to win', or is 'on a hot streak right now', while the losers were 'tired' and 'wilted under pressure'. For many traditionalists, much of the appeal of sports is in these explanatory stories. But such claims invariably are post hoc and so impossible to test—had the outcome been different, we'd have told a different story to explain it. Recently, statistically-minded observers (especially of baseball) have begun insisting that many events in sports really are random, or at least are best thought of as random because we are ignorant of their causes (although we may think we're not). As another example, religious beliefs sometimes have been interpreted as a way for believers to see deterministic causality, order, and a purposeful plan in a universe that would otherwise appear random, uncontrollable, and purposeless.

UPDATE: xkcd hits the nail on the head, as usual.