Assume you have learned that the winning numbers in the state lottery are 1, 2, 3, 4, 5, and 6. Would you suspect that the drawing is faulty? You very well may, but why? After all, the probability of this sequence is no smaller than the probability of 9, 15, 21, 40, 54 and 11 (the winning numbers in California’s draw #723 from May 25). In fact, all sequences are equally unlikely, but still every time one of these unlikely sequences manages to be drawn. Now assume that in addition the winning ticket is held by a barber. Would this raise any suspicion of foul play? Ridiculous, if barbers win do they not deserve their earnings? But if the winner has long been the barber and friend of the person overseeing the lottery, this could be grounds for suspicion. Is this fair? Should it be legal grounds to sustain a fraud indictment? And if once is not convincing enough, what if it happens 20 times? How could we convincingly argue that this event (which is not less likely as the event that some other sequence of 20 people win), suggests that the drawings were not truly random? Isn’t the problem with randomness that you can never be sure?
Rigged lotteries and other related scenarios are the topic of a thought provoking paper: Impugning Randomness, Convincingly by Yuri Gurevich and Grant Olney Passmore. Let us return to the lottery where the winning numbers are 1, 2, 3, 4, 5 and 6. How can we argue that a fixed sequence of numbers is not “random enough”? Some readers may guess at this point that a possible approach is to invoke Kolmogorov Complexity, which is indeed what the aforementioned paper does. Intuitively, the sequence 1, 2, 3, 4, 5 and 6 has a short description (and thus a small Kolmogorov Complexity). Since only relatively few sequences can have a short description, the probability that any short-description sequence is drawn is very small and thus it is strong evidence that the drawing was not uniform. Why aren’t we done? The main remaining challenge if we take this route is that the description length of a string depends on the description language (alternatively, on the universal Turing machine with respect to which we measure the Kolmogorov Complexity). This challenge is the main focus of the paper, and is the reason we cannot view the problem as solved.
When reading this interesting paper I was thrown back to a memory from my high school years. A long time ago, my high school grade was subjected to a lecture by a missionary Rabbi. He talked about Bible Codes: these are unexpected patterns that were found in the Bible. For example, it is claimed that by taking every 50th letter of the Book of Genesis starting with the first law, the Hebrew word “torah” (Bible) is spelled out. The lecturer, attempting to be subtle, claimed that he does not submit these codes as a proof for the existence of god, but scientists have determined that they prove that the author of the Bible had IQ 5000 (or some other meaningless number). I stood up and argued against (passionately but unfortunately not very eloquently nor too effectively). What I intuitively understood was that there are so many possible surprising sequences, that the existence of some could be a matter of luck. I was guessing that such sequences exist in any large enough text, of either heavenly or earthly origin.
Bible codes became the center of additional controversy when Witztum, Rips and Rosenberg (WRR) described in a paper the results of two experiments. Similar Bible codes matched the names of famous Rabbis that existed long after the Bible was written, and the appearances of these names was argued to be statistically significant (that is, unlikely to be explained by sheer luck). As someone who always viewed Bible codes (when taken too seriously) as the realm of missionaries and quacks, I was dismayed to see it gain credence from a mathematician of such high caliber. Fortunately, the opponents of WRR had a much more able voice than in my high-school story. I will not get too deeply into the details of this debate, but will just mention very informally that the opponents argued that the data selection was biased (specifically, the choices of the exact spelling of the Rabbi’s names among the possible spellings) and that even assuming the phenomenon WRR tried to demonstrate, the results are simply too good to be true. In particular, the confidence in the two experiments was too close. I will demonstrate this argument with a story due to Ehud Friedgut which appeared here that ties back quite nicely with the start of this post.
A man claims to be able to hit a globe hanging 200 meters away with a bow and arrow while blindfolded. An experiment is set; he shoots two arrows (a few minutes apart) and then sends his son to fetch the globe (which is too far from other observers’ sight). The son returns with a globe and two arrows stuck into it as close as physically possible. This level of accuracy makes it even harder to believe the integrity of the experiment but we cannot yet prove our suspicion. But now assume that we learn that while the arrows were shot, the globe was rapidly spinning around its axis (without the father and son’s knowledge). This means that, regardless of the father’s archery skills, the longitude of the two arrows should be distributed uniformly. Therefore, while it is still possible that the two arrows will end up adjacent, it will happen with extremely low probability and we can therefore view their position as a proof of probabilistic nature that the experiment was rigged.
There is much more to discuss on what can be learned about a distribution from one or a few samples and about the integrity of scientific exploration. I hope to revisit these topics in the future.