Black Swans, Big Data, Our Intuition, and the "Birthday Paradox"

Jan 18, 2016

SPOILER ALERT: This post is based on a fascinating old riddle. If you want to play along, cover up all but the first paragraph below and ponder it.

At a dinner party last night, one of the guests posed a question: Imagine a roomful of people chosen at random. How many people need to be in the room for there to be at least a 50% probability that at least two of the people have the same birthday? (For you smarties in the audience, that’s birthday, NOT birthdate, and the room contains no twins, triplets, etc.)

The answers from the guests varied widely, from as few as 51 people in the room, to as many as 183 (i.e., one-half of the 366 different possible birthdays). In fact, the answer is an amazingly low 23 people! If the room has 50 people, there is a 97.0% probability that at least two people share the same birthday; with 70 people, the probability soars to 99.9%, and to 99.99997% with 100 people (i.e., only one chance in 3.3 million that each of the 100 people has a different birthday).

For a complete yet readable discussion of this problem, including a straightforward discussion of the solution and a nicely done graph, see the Wikipedia article. Or try it out yourself empirically, with coworkers or family & friends.

All this is interesting, but why does this matter, and what can we financial analysts learn from this problem? Well, for starters:

Black swan events do happen, more often than we expect. Much has been written about the increasing occurrence of events thought to be incredibly improbable. In an increasingly complex world, we can expect more black swans simply because there are more people, more possible interactions, and more other factors at play. Maybe that “black swan” is just a "shared birthday."
Big Data doesn’t always help. The more information we are able to collect and sift through, the more amazing, improbable relationships we’ll find. However, as Nassim Nicholas Taleb (author of The Black Swan and Fooled by Randomness) has observed, what we’ve discovered may just be randomness, and not an important causal relationship.
Intuition can be misleading. When it comes to probabilities, uncertainty, and risk, our instincts don’t always lead us to the right answer. Sometimes we need to hear from a clear-thinking, unsentimental third party.

In other words, sometimes those black swans aren’t nearly as black as we think. And the real art of financial modeling is not in enabling your audience to understand what is most likely to happen, but what the range of possible outcomes is. Do your models achieve that?

“Painting with Numbers” is my effort to get people to focus on making numbers understandable. I welcome your feedback and your favorite examples. Follow me on twitter at @RandallBolten.