Readings: Significance, Kale, bankruptcy, and streaming media


[I]t will become clear that there is only one principle that can be defended under all circumstances and in all stages of human development. It is the principle: anything goes.
― Paul Feyerabend

Anything goes … means … everything stays.
― Alan Chalmers

It’s the best possible time to be alive, when almost everything you thought you knew is wrong.
― Tom Stoppard, Arcadia

The philosophy of science is a strange place in academia. To the extent that it is philosophy it isn’t science, and to the extent that it is science, it isn’t philosophy. Perhaps the only thing that both philosophers and scientists can agree on is that they can’t stand philosophers of science, and they wish they didn’t get so much goddamn attention for something that doesn’t actually work. We just don’t have a comprehensive theory for how science works; we just know that it works better than the alternatives, kinda, most of the time. 

I was reminded of it again this week in reading the noisy discussion about a recent Nature commentary on the merits of retiring the 0.05 significance hurdle in publishing research findings.  There are compelling arguments to be made about getting rid of an arbitrary hurdle, like 0.05, especially one so easily gamed in large data sets, especially of the sort of that pharmaceutical companies marshall to get expensive drugs with minimal effect past the U.S. Food & Drug Administration. Hey, the tests’ results were significant, c’mon. 

This, of course, is not unique to medical statistics. It is a restatement, of sorts, of Goodharts’ Law, that, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”. Once we have a measure nailed, in other words, and it is used to judge and control processes, especially a human-centric one, it ceases being useful. This is the dilemma we face in medical statistics, where, to a first approximation, we have little idea whether products whose effects are deemed statistically significant actually have significant effects. As Goodhart would say, the measure of significance has collapsed. 

But with what do we replace it? SImply cautioning people against blind fealty to 0.05 significance isn’t enough, as decades of academic research have shown, where faculty slavishly still report the measure, despite much private eye-rolling, knowing full well that without it they won’t get published. And having no hurdle comes with its own problems, as Chalmers’ rejoinder to Paul Feyerabend highlights. While anything may “go” in science — if said research works — if it truly goes then we have no basis for deciding what matters and what works — other than what works, of course, which may turn out to be all we really had in the first place.

Are we genuinely prepared for a 0.05-less world of research? Or do we prefer to continue with statistical thumbsucking and gameable measures? My bet is that very little changes. 





From the German youth version of The Voice (who knew such things existed?), two sisters perform Radiohead’s Creep, to remarkable effect. It is yowling, dissonant, and dangerous, like something that should be played over the end credits in Fight Club II as the world tumbles into itself.