Everyone’s misunderstanding the joke here. It’s not about “food scare stories” per se, it’s about a particular statistical fallacy that is pervasive whenever science intersects the desire for funding: inferring from the fact that a statistical test rejects the null hypothesis with a high p value is strong evidence of the null hypothesis being false.
In the medical context, as in the cartoon, the effect typically arises where researchers are presented with a huge amount of data with thousands of features, and are testing for an enormous number of possible conditions: some feature will be claimed to affect the condition tested for at a high p-value, even though the set of all features and conditions tested for a priori is not reported.
In the legal context, this typically arises in, say, discrimination cases, where hundreds of possible statistical tests can each be performed on thousands or millions of populations: each district (for school or voting cases); each branch or set of branches (for commercial ones). When some number of these tests, on some cross-section of the population, finds that the result rejects the null hypothesis (of non-discrimination), that test alone can be reported.
It arises basically whenever people have an incentive (conscious or otherwise) to justify transfer of assets through some statistical methodology: with enough tests and enough data, some null hypotheses will always be rejected.
In theory, the set of all possible tests used are determined before the data is analyzed (ideally, even collected); and all test results are reported. But in practice that doesn’t happen when a lot of money is at stake.
A good introduction to this modality of argument might be “The Cult of Statistical Significance” (forget the authors).
Anyway, xkcd wasn’t targeting medical tests per se, as this kind of argument crops up constantly.
Also, KDP, that line about “lies…statistics” is ridiculously cliched, and nobody’s lying here: people are making improper inferences from statistics that they should know they don’t understand. Anyway, I get sick of hearing that – statistical analysis is one of the great inventions of the modern mind, if used disinterestedly and rigorously.
I stand corrected about the statistics portion of my remark. I agree that statistical analysis must be used disinterestedly and rigorously – it just seems that no one will use them in an unbiased manner.
Randall Munroe is my hero. His disclaimer is both amusing and accurate. In many of his strips, despite the stick figures, the text is really not suitable for children or adults without an understanding of math, physics, statistics, engineering and computing. Legal jargon may be nowhere near as confusing to the casual reader as the xkcd strip.
xkcd is sui generis; I don’t know that any other comic strip has so aptly captured the spirit of a particular subculture, with such wryness and understatement.
8 Comments
It looks like the state of scientific research is sliding into lies, damned lies and statistics.
Research, no. Reporting, yes.
Everyone’s misunderstanding the joke here. It’s not about “food scare stories” per se, it’s about a particular statistical fallacy that is pervasive whenever science intersects the desire for funding: inferring from the fact that a statistical test rejects the null hypothesis with a high p value is strong evidence of the null hypothesis being false.
In the medical context, as in the cartoon, the effect typically arises where researchers are presented with a huge amount of data with thousands of features, and are testing for an enormous number of possible conditions: some feature will be claimed to affect the condition tested for at a high p-value, even though the set of all features and conditions tested for a priori is not reported.
In the legal context, this typically arises in, say, discrimination cases, where hundreds of possible statistical tests can each be performed on thousands or millions of populations: each district (for school or voting cases); each branch or set of branches (for commercial ones). When some number of these tests, on some cross-section of the population, finds that the result rejects the null hypothesis (of non-discrimination), that test alone can be reported.
It arises basically whenever people have an incentive (conscious or otherwise) to justify transfer of assets through some statistical methodology: with enough tests and enough data, some null hypotheses will always be rejected.
In theory, the set of all possible tests used are determined before the data is analyzed (ideally, even collected); and all test results are reported. But in practice that doesn’t happen when a lot of money is at stake.
A good introduction to this modality of argument might be “The Cult of Statistical Significance” (forget the authors).
Anyway, xkcd wasn’t targeting medical tests per se, as this kind of argument crops up constantly.
Also, KDP, that line about “lies…statistics” is ridiculously cliched, and nobody’s lying here: people are making improper inferences from statistics that they should know they don’t understand. Anyway, I get sick of hearing that – statistical analysis is one of the great inventions of the modern mind, if used disinterestedly and rigorously.
I stand corrected about the statistics portion of my remark. I agree that statistical analysis must be used disinterestedly and rigorously – it just seems that no one will use them in an unbiased manner.
Asdfasdf – one of the best explanations I’ve seen of the whats and whys of our statistical analysis failures today. Great job!
Odds are that nobody will change, though.
Randall Munroe is my hero. His disclaimer is both amusing and accurate. In many of his strips, despite the stick figures, the text is really not suitable for children or adults without an understanding of math, physics, statistics, engineering and computing. Legal jargon may be nowhere near as confusing to the casual reader as the xkcd strip.
xkcd is sui generis; I don’t know that any other comic strip has so aptly captured the spirit of a particular subculture, with such wryness and understatement.