From yesterday's WSJ:
Shattering the Bell Curve
The power law rules.
BY DAVID A. SHAYWITZ
Tuesday, April 24, 2007 12:01 a.m. EDT
Life isn't fair. Many of the most coveted spoils--wealth, fame, links on the Web--are concentrated among the few. If such a distribution doesn't sound like the familiar bell-shaped curve, you're right.
Along the hilly slopes of the bell curve, most values--the data points that track whatever is being measured--are clustered around the middle. The average value is also the most common value. The points along the far extremes of the curve contribute very little statistically. If 100 random people gather in a room and the world's tallest man walks in, the average height doesn't change much. But if Bill Gates walks in, the average net worth rises dramatically. Height follows the bell curve in its distribution. Wealth does not: It follows an asymmetric, L-shaped pattern known as a "power law," where most values are below average and a few far above. In the realm of the power law, rare and extreme events dominate the action.
For Nassim Taleb, irrepressible quant-jock and the author of "Fooled by Randomness" (2001), the contrast between the two distributions is not an amusing statistical exercise but something more profound: It highlights the fundamental difference between life as we imagine it and life as it really is. In "The Black Swan"--a kind of cri de coeur--Mr. Taleb struggles to free us from our misguided allegiance to the bell-curve mindset and awaken us to the dominance of the power law.
The attractiveness of the bell curve resides in its democratic distribution and its mathematical accessibility. Collect enough data and the pattern reveals itself, allowing both robust predictions of future data points (such as the height of the next five people to enter the room) and accurate estimations of the size and frequency of extreme values (anticipating the occasional giant or dwarf.
The power-law distribution, by contrast, would seem to have little to recommend it. Not only does it disproportionately reward the few, but it also turns out to be notoriously difficult to derive with precision. The most important events may occur so rarely that existing data points can never truly assure us that the future won't look very different from the present. We can be fairly certain that we will never meet anyone 14-feet tall, but it is entirely possible that, over time, we will hear of a man twice as rich as Bill Gates or witness a market crash twice as devastating as that of October 1987.
The problem, insists Mr. Taleb, is that most of the time we are in the land of the power law and don't know it. Our strategies for managing risk, for instance--including Modern Portfolio Theory and the Black-Scholes formula for pricing options--are likely to fail at the worst possible time, Mr. Taleb argues, because they are generally (and mistakenly) based on bell-curve assumptions. He gleefully cites the example of Long Term Capital Management (LTCM), an early hedge fund that blew up after its Nobel laureate founders "allowed themselves to take a monstrous amount of risk" because "their models ruled out the possibility of large deviations."
Mr. Taleb is fascinated by the rare but pivotal events that characterize life in the power-law world. He calls them Black Swans, after the philosopher Karl Popper's observation that only a single black swan is required to falsify the theory that "all swans are white" even when there are thousands of white swans in evidence. Provocatively, Mr. Taleb defines Black Swans as events (such as the rise of the Internet or the fall of LTCM) that are not only rare and consequential but also predictable only in retrospect. We never see them coming, but we have no trouble concocting post hoc explanations for why they should have been obvious. Surely, Mr. Taleb taunts, we won't get fooled again. But of course we will.
Writing in a style that owes as much to Stephen Colbert as it does to Michel de Montaigne, Mr. Taleb divides the world into those who "get it" and everyone else, a world partitioned into heroes (Popper, Hayek, Yogi Berra), those on notice (Harold Bloom, necktie wearers, personal-finance advisers) and entities that are dead to him (the bell curve, newspapers, the Nobel Prize in Economics).
A humanist at heart, Mr. Taleb ponders not only the effect of Black Swans but also the reason we have so much trouble acknowledging their existence. And this is where he hits his stride. We eagerly romp with him through the follies of confirmation bias (our tendency to reaffirm our beliefs rather than contradict them), narrative fallacy (our weakness for compelling stories), silent evidence (our failure to account for what we don't see), ludic fallacy (our willingness to oversimplify and take games or models too seriously), and epistemic arrogance (our habit of overestimating our knowledge and underestimating our ignorance).
For anyone who has been compelled to give a long-term vision or read a marketing forecast for the next decade, Mr. Taleb's chapter excoriating "The Scandal of Prediction" will ring painfully true. "What is surprising is not the magnitude of our forecast errors," observes Mr. Taleb, "but our absence of awareness of it." We tend to fail--miserably--at predicting the future, but such failure is little noted nor long remembered. It seems to be of remarkably little professional consequence.
I suspect that part of the explanation for this inconsistency may be found in a study of stock analysts that Mr. Taleb cites. Their predictions, while badly inaccurate, were not random but rather highly correlated with each other. The lesson, evidently, is that it's better to be wrong than alone.
If we accept Mr. Taleb's premise about power-law ascendancy, we are left with a troubling question: How do you function in a world where accurate prediction is rarely possible, where history isn't a reliable guide to the future and where the most important events cannot be anticipated?
Mr. Taleb presents a range of answers--be prepared for various outcomes, he says, and don't rush for buses--but it's clear that he remains slightly vexed by the world he describes so vividly. Then again, beatific serenity may not be the goal here. As Mr. Taleb warns, certitude is likely to be found only in a fool's (bell-curve) paradise, where we choose the comfort of the "precisely wrong" over the challenge of the "broadly correct." Beneath Mr. Taleb's blustery rhetoric lives a surprisingly humble soul who has chosen to follow a demanding and somewhat lonely path.
I wonder how many of us will have the courage to join him. Very few, I predict--unless, of course, something unexpected happens.
Dr. Shaywitz is a physician-scientist in New Jersey. You can buy "The Black Swan" from the OpinionJournal bookstore.
