The unnerving nature of collider bias

By Brian Clegg

Not for the first time, I was inspired by listening to the excellent The Studies Show podcast. In their 26 November edition, Tom Chivers and Stuart Ritchie introduced collider bias, which is a horrible statistical anomaly that can make it seem that a study shows something entirely different from reality. What’s particularly worrying is that, as the podcast demonstrates, many scientists aren’t aware of this potential issue.

I had assumed collider bias would be something to do with the kind of huge statistical analysis necessary to interpret what’s going on in a piece of equipment like the Large Hadron Collider at CERN – but in reality the ‘collision’ in question is simply down to the way a pair of arrows point to the same location on a kind of flow diagram. What this statistical anomaly can produce, though, is the kind of result we love to hear (and scientists love to find) – results that make you go ‘Huh? That’s surprising.’

Examples given included that of those hospitalised with COVID-19, smokers were more likely to survive than non-smokers; amongst cardiovascular disease sufferers, obese patients live longer; success at basketball is not linked to a player’s height; and PhD candidates who have high scores in the tests often used to decide if someone should start on a PhD are not more likely to succeed than those who score badly.

To understand this, Tom and Stuart ask us to imagine a study of Hollywood stars. They suggest you get to be a Hollywood star because you are either beautiful or a talented actor (or both). Assuming more actors major on one attribute than both, then, for the population sample that is ‘Hollywood stars’, you will find that beauty is negatively correlated with acting talent. Actors with talent, it would seem, tend not to be beautiful, and vice versa. This would be statistically true, but there is no causal link. The real danger is then to apply the same reasoning to the population at large and think that to be a great actor, a person should be ugly. But it’s an artefact of the way Hollywood stars are chosen, not a true causal relationship.

In the surprising examples mentioned above, where this has occurred in real studies (often resulting in convoluted arguments as to why, say, being obese gives better survival from cardiovascular disease), it’s because in each case we are looking at a sub-population – for example professional basketball players or PhD candidates, not considering people at large. So, for example, successful PhD candidates tend to be either highly intelligent or very hard workers (or both). But by only looking at successful PhD candidates, those two groups will dominate, where looking at the population at large, highly intelligent (and hence high scoring) people will be more likely to gain a PhD.

In their podcast (to be honest, one of their more meandering episodes, as this is a really difficult effect to describe), Tom and Stuart point out that this is relatively easy to spot when the result is so counter-intuitive, a reasonable flag to check if there is something wrong with the analysis. But the error can be missed if it’s less stand-out. 

Arguably a starting point should be that if you are studying a group that isn’t typical of the population at large, then you need to be aware of this danger. This should be of particular interest, for example, to psychologists, who often do studies using university students as participants, because they are cheap and readily available. Unfortunately, though, they may well beg a collider bias population just waiting to happen.

These articles will always be free – but if you’d like to support my online work, consider buying a virtual coffee:

See all Brian's online articles or subscribe to a weekly email free here

Now Appearing is the blog of science writer Brian Clegg (www.brianclegg.net), author of Inflight Science, Before the Big Bang and The God Effect.