Monday 23 July 2012

Resolving the impossible lottery

In a previous post, I have posed a question which seems to admit of two contradictory explanations. Go read that post before this one or this post won't make a whole lot of sense!

The Correct Answer

My answer is that you should never believe you have won the lottery. If the population is arbitrarily large, then the probability that you have been chosen to win the lottery is arbitrarily small. It can never be the most reasonable explanation.

If the probability that you are hallucinating is non-zero, as it presumably is, then even this is a more likely explanation than that you have actually won. Other possibilites include that you have somehow misunderstood the nature of the lottery or the population of your nation, such that the probability of winning is much higher than you had believed. You might even be living in The Truman Show, and the experimenters have decided to let you win the lottery to see how you react.

It doesn't matter how crazy the proposed explanation is. As long as your odds of winning legitimately are arbitrarily low, then any crazy explanation with a non-zero probability is more likely than that you have actually won and have a correct understanding of how the lottery works.

But somebody has to win, right?

Yes, somebody has to win. And before the fact, that person's chance of winning is also effectively zero. So how come it makes sense to believe that that person has won, but not to believe that you have won yourself?

I think the problem can be phrased in terms of coincidence. The probability of an unspecified random stranger winning the lottery is high - in fact it's certain. If, however, the person who wins is specified or is otherwise a person who is significant to you, then that is a coincidence, and the probability of that coincidence happening is zero, more or less.

And here is where the logic of my post on coincidences comes into play. We should not be surprised to find that coincidences happen in the population at large, in fact we should expect them. However, we should not expect to experience any truly remarkable (i.e. one in a billion) coincidences ourselves, simply because most people do not. Strangers winning the lottery is analogous to amazing coincidences happening in the general population. You winning the lottery is an amazing coincidence happening to you.

Coincidences and significance

Coincidences are all about what we find significant. If a married couple both happen to share the same birthday, we find that to be a coincidental because we observe a significant and surprising pattern. This is of no importance to the universe, and yet we find it interesting.

Now this example may not be sufficiently improbable to cause us to believe there must be some non-chance explanation, but it wouldn't be unreasonable to wonder whether there might be. We might wonder, for example, if the couple met each other on their birthday as they each held celebrations in the same pub.

In fact, I suspect that of the couples who share a birthday, a significant fraction of them (perhaps not a majority) found each other for reasons relating to their birthdays.

And it is so for many coincidences. Whenever events conspire in an improbably significant way, we have reason to guess that there may be some non-chance explanation. The more improbable the coincidence, the stronger should be our suspicion that the coincidence is not the product of chance alone.

We have evolved to be pattern-seeking individuals for a reason. It is genuinely useful to be able to spot these patterns and attribute reasons to them rather than simply regarding coincidental occurrences as the products of chance. This can be overdeveloped, leading to pareidolia, paranoia and superstition, but it can also help to detect genuine phenomena.


It may not matter to the universe that we observe a coincidence that only we deem to be significant, but if we follow up on our hunches that truly improbable coincidences are unlikely to be the products of chance, then we will more often than not be correct.

The many correct paranoid winners

Perhaps the key to resolving the paradox of this puzzle is to realise that of those people who believe they have won the lottery on any given day, the overwhelming majority of them are wrong. Only one person can win every day, but in a population that is arbitrarily large, there will be a huge number of people who mistakenly believe they have won for one crazy reason or another.

Some will be delusional. Others will be the beneficiaries of particularly benevolent conspiracy theories. Others will have been deceived by freakish coincidences involving data corruption or bank errors - coincidences inconceivably improbable and yet more probable than actually winning legitimately.

Of those "winners" who follow my advice and distrust their apparent lottery win, the overwhelming majority of them will be correct.

As weird as it seems, it really is more rational to refuse to believe you have won the lottery, no matter what evidence is presented to you, if the odds of winning the lottery are arbitrarily low.

14 comments:

  1. I think this argument is fallacious in that it mistakes the "absolute" probability for the "relative" probability; what is relevant in this case is not the probability dealing with the overall likelihood of winning the lottery--the absolute probability--but rather the probability of being falsely informed of winning such a lottery, and awarded handsomely for it, over actually winning the lottery--the relative probability.

    So the maths concerning the probability of winning the lottery, Im claiming, is irrelevant to the issue. What concerns us is simply the probability consisting of the number of individuals who are awarded money under the false claim that they won said lottery, divided by the total number of people awarded money on account of winning said lottery. And my guess is that this will result in a very small fraction (i.e. that most the people who are awarded on account of winning the lottery have actually won the lottery; nearly no one is given a huge sum of money, told it is due to winning the lottery, yet have not actually won--this just isn't a good scam).

    Thus, I think the correct answer is that given that one has been informed of having won the lottery and received a healthy payment as a reward, then one very probably has in fact won the lottery, despite the fact that this is practically impossible.

    (Again, to be clear, the only numbers that are relevant to the question are those of the people who are erroneously awarded on account of winning the lottery divided by the total number of people who are awarded on account of winning the lottery--where the total is derived from adding both the true and false winners together; this will give us the desired probability, which I believe would be very, very low.)

    ReplyDelete
  2. I think there's a quick answer to your point.

    ' the only numbers that are relevant to the question are those of the people who are erroneously awarded on account of winning the lottery divided by the total number of people who are awarded on account of winning the lottery
    '

    Exactly.

    Now, what number of people would erroneously believe they have won the lottery on any given day? Some staggeringly small fraction of the population, I'm sure. Let's say 0.000000000000000000000000000000000000001%.

    Now what's 0.000000000000000000000000000000000000001% of an infinite population? An infinite number of people.

    Conversely, how many people have actually won the lottery? One.

    Therefore, (effectively) 100% of the people who believe they have won the lottery are mistaken.

    See where I'm coming from?

    The point of the thought experiment is to illustrate how unreliable our intuitions are when dealing with the infinite. The problem is that it's damned hard to convince people their intuitions are wrong in this case.

    Perhaps this thought experiment is just a little bit too unintuitive to be useful!

    ReplyDelete
  3. Hmm, I realise that I never said "infinite" in the original post, I said "arbitrarily large".

    The two are effectively the same though - the argument holds if you make the population big enough.

    For the numbers I used in my last comment, you'd need to have a population larger than 10 to the power of 41 for the argument to hold, which is still far smaller than a Googolplex, which was mentioned on the first post.

    ReplyDelete
  4. Disagreeable,

    If the population is infinite, the problem is not well defined. There is no way to set up a fair lottery, where every person out of that infinite pool has the same initial chance. How do you draw a random number from a prior range that is unlimited?

    If your population is p, your surprise if you win the lottery, expressed between 0 (zero surprise) and 1 (maximum surprise), should be 1 - 1/p. No surprise here...

    ReplyDelete
    Replies
    1. Hi energie_sombre,

      The population is not infinite. It's arbitrarily large. My point is that as the population *tends* to infinity, your surprise should tend to 1, and when the population is large enough, you should never believe that you have won the lottery. Some other explanation, no matter how improbable, must be true.

      This aims to show the fallacious nature of the intuition that as long as somebody has to win, it could be you.

      Delete
    2. In my understanding, if the population is finite, the problem is well defined. If it is infinite, it is not.

      I don't know what the difference between "arbitrarily large" and infinite is. Your surprise goes arbitrarily close to one, therefore it is one. (If it were not, but a number say q<1, you could find a number halfway between q and 1, so the surprise would not be arbitrarily close to 1.)

      Delete
    3. The difference between arbitrarily large and infinite is how the notion of limits in calculus works. You get undefined behaviour if you work with literal infinity, but you get a concrete answer if you work with limits as certain variables tend to infinity.

      In practical terms for this problem, "arbitrarily large" means that no matter how unlikely you are to be mistaken about having won the lottery, there is some size of population such that it is even more unlikely that you have actually won the lottery.

      In other words, if you don't buy my argument for a population of 100 trillion individuals, I can just reiterate my argument for a population of 100 quintillion individuals. I can repeat this process ad nauseum until you are forced to accept the argument.

      Delete
    4. If the population tends to infinity, there is no limit, you get infinity and not a concrete answer, and the prior probability is not defined.

      Delete
    5. No, there is a limit, and the limit is a probability of zero.

      It seems that you're not familiar with the concept of limits?

      Let's take a trivial example. Suppose we want to calculate the value of 1/(2-x) as x tends to 2. You can't just substitute 2 for the value of x, because you get a division by zero which is undefined.

      But if you substitute in 1/(2-1), then 1/(2-1.5), then 1/(2-1.75) etc, you will see a trend. It's possible to prove that the expression as a whole approaches positive infinity as x approaches 2 (starting from x < 2). Starting from x > 2, the expression approaches negative infinity.

      You can see a graph of what's going on here:

      http://www.wolframalpha.com/input/?i=1%2F%282-x%29

      Similarly, you can show that the value of the expression tends to 0 as x approaches infinity.

      In the case of the improbable lottery, we're trying to calculate a probability, 1/x. Now, one divided by infinity is undefined, but 1/1, 1/2, 1/3, ... 1/1000 shows a clear trend of tending to zero. As x becomes arbitrarily large, so does the probability approach zero. You can see what's going on here:

      http://www.wolframalpha.com/input/?i=1%2Fx

      So the point I'm making is more or less true if we assume a population of a billion billion people. It becomes truer as you increase the size of the population, so it's even more so for a trillion trillion trillion and far more so again for a Googolplex.

      Delete
    6. You should have a quick look at this if you still don't understand what I'm talking about:

      http://en.wikipedia.org/wiki/Limit_(mathematics)

      Delete
    7. I can assure you, I am quite familiar with limits. Maybe I didn't express myself clearly.

      I get your case where we want to calculate the probability 1/x. Now for every finite value x (population size), the probability 1/x is larger than zero. So a lottery win might be extremely improbable, but such a case is not interesting for any x, even if x = Googolplex.

      The interesting case IMHO is when you let x actually go to infinity, a process that is well defined mathematically, even though this very limit, lim(x for x->infinity), does not exist (it's not an integer, but it "is infinite"). You can also say that for any integer y, x > y.

      In that case, the probability lim(1/x for x->infinity) is exactly zero. The apparent paradox comes from the fact that one person still wins the lottery. I think this has to be resolved using measure theory, where this event of winning a lottery in an infinite population has measure zero. But I still see a problem in defining the prior probability.

      Delete
    8. I apologise if I've been patronising. It's hard to know who you're dealing with when you're talking to an anonymous person on the internet, but in the context of your other comments and your appreciation for quantum physics I should have known better than to suppose you didn't understand limits.

      >So a lottery win might be extremely improbable, but such a case is not interesting for any x, even if x = Googolplex.<

      This is all that is required for the point I'm making in this post. It may not be interesting to you, but many people seem to find it impossible to accept the reality that you should be suspicious of a win in these circumstances to the point of disbelief.

      What you say about the interesting case of x->infinity may well be correct but it falls outside the scope of what I was trying to express in this post, and also likely outside of my expertise.

      Delete
    9. Well, I enjoyed reading your blog post and the following discussions! It's always nice to think about these counter-intuitive logical and mathematical puzzles.

      Delete
    10. Me too, energie_sombre. Thanks for commenting!

      Delete