Xml Long entries
Xml All entries
Xml All entries + links

Hosted by New World Order Norway Nwo

Lessons from the anti-library

No book better summarizes what I've been reading and thinking about in the last couple of years than Fooled by Randomness by Nassim Nicholas Taleb, which I reviewed in 2005. Fooled by Randomness was a work of practical philosophy, using examples from the financial world to attack the layers of illusory knowledge that we build on top of random events around us.

The financial world is a good place to find such layers, with its simplifications and retroactive explanations. Lucky traders become "geniuses", unexpected up- and downturns are found to be "inevitable", and the bell curve is assumed to be a fundamental property of reality. But this process of make-belief is not restricted to finance, it is universal. We do it to our lives, and we do it to the world we live in.

In The Black Swan - The Impact of the Highly Improbable, Taleb continues this theme of uncertainty and the illusion of expert knowledge. It is a better, deeper book, and less likely than Fooled by Randomness to be mistaken for financial advice. It is a book about skepticism, wild randomness and the power of stories. It's about how not to be a sucker to the unexpected and the unknown.

Taleb introduces the Hayekian and almost taoistic metaphor of the anti-library: A library of the books you haven't read, of the things you don't know. A massive collection of unknowledge, the anti-library contains all the books that may still change your life. Most of your favourite books are there, hidden and forgotten. It has facts you need to know, authors you'd worship. Only people who read very little can think the anti-library doesn't matter. To book-lovers, and especially generalists like me, who jump all over the place without focus, the stacks of the anti-library loom higher and darker for every book we read. Every book that changes me reminds me of the ones that still might.

The Black Swan is not about unread books as such, but about all the important, life-changing, world-changing information that we don't have. Unknowledge is an unsettling concept. Like the Death of Discworld, our eyes glaze over it and hurriedly find something more comforting to focus on. Taleb asks us to look at it, and acknowledge that it's there, a massive source of wild randomness that will eat your predictions for breakfast, and change your world again, and again, and again. As it has done so often in the past.

The title comes from a classic problem of induction, (it took only the unexpected observation of one black swan to ooverturn centuries of observations of white ones), but I like Taleb's turkey metaphor better: Imagine that you're a turkey. You've eaten well and lived in safety every day of your life. Everything in your experience tells you that tomorrow will be no different. Then Thanksgiving arrives - a Black Swan event.

A Black Swan is an unexpected event that has a massive impact and seems perfectly explainable in retrospect. It is a September 11, a stock market crash, a World War 1. Think of it as a story-breaker. At any given time we have in our heads a story of the world we live in, a story that may be bleak or rosy, but either way makes sense. All of history could be told through the stories that competed for people's attention at any given time. Medieval theology, enlightenment rationalism, marxist history. All of them stories, compelling stories that for a time, to some people, seemed to describe reality.

Then a Black Swan comes along - a new idea, a new technology, a revolution, a war, a terrorist attack - and exposes the stories for the fabrications they are. It's like when you look up at the night sky and glimpse the true, insane scale of the universe, but only for a minute, before it returns to being a roof with pretty lights again. After a Black Swan the ground seems fluid, the eternal malleable, but it doesn't last. Soon the illusion returns in the form another story: A revision of the old, or an entirely new story. The new story incorporates the Black Swan, makes it obvious, natural, inevitable. The world is again safe and rational - until the next Black Swan arrives.

We've recently lived through an event like this, a burst of insane reality followed by a return to fake normalcy. As I've described elsewhere, I was caught up in the rationalization process that followed. Taleb writes, and I agree, that we took the wrong lessons from the September 11 terrorist attacks. We learned how to avoid another attack just like it. We learned to fear the specific causes of that attack. What we should have learned was to trust our stories less. We learned that 20th century history culminated in Osama bin Laden. What we should have learned was that history doesn't march, it jumps.

Good historians know that, but the bad ones tell better stories, and their good, false stories make up the scenery the news media stages its productions against. In the hands of journalists, pundits and media experts, history becomes a mythology - a collection of moral lessons for our time. In reality, history is wildly random. While it happens it is shocking, insane and painful. The stories come later, to heal the trauma, and make-believe a sane world.

Taleb calls the world of predictable stories Mediocristan, a place ruled by mild forms of randomness such as the bell curve. Black Swans are creatures of Extremistan, a world ruled by the unlikely and the extreme. Taleb distinguishes between the popular concept of randomness, associated with bell curves, games of chance, and quantum uncertainty, and the relevant, wild kind of randomness, which comes out of anything that is unknown.

A bell curve or a game of chance is predictable because it obeys the law of large numbers. As you get more samples, the results converge. Even quantum randomness cancels out on the large scale - individually, fundamental particles are unpredictable. By the billions, they're pretty dependable.

Wild randomness does not obey the law of large numbers. It does not converge. A thousand days of not being eaten tells the turkey nothing about what to expect on Thanksgiving. Wild randomness has nothing to do with the much-abused uncertainty principle. Things don't have to be unknowable in theory to be unknown in actuality. As complexity theory shows, even a deterministic world would be unpredictable.

The Unknown is the source of randomness we should be concerned about. It's the Unknown that will change our lives, not the safe probabilites of a die throw. Taleb calls our use of game-based randomness to model real life the ludic fallacy. We mistake the world for a game of chance, when it is much more random than that.

Not all Black Swans are negative, and not all are sudden. From one perspective, all modern history is one huge, slowly unfolding Black Swan. This is reflected in the ever-changing stories we live in. Our stories used to be more stable, your children saw the same world you did. Now the stories are as shortlived as a summer blockbuster. And still we believe in them, in the next as sincerely as in the last.

Taleb admits that the Black Swan too is a story. But it's a better story. We can only understand the world through stories, there's no way to change that. What we need, (and again, Pratchett readers will see an analogy), are good stories to fight the bad ones. Stories that prepare us for the unpredictable, encourage us to doubt what we know. As Taleb writes, you can't avoid crossing the street, but you can try not to do it blindfolded. You can try not to be a sucker.

In Fooled by Randomness, Taleb pointed to Stoicism as a way to deal with the whims of fortune. The Stoics believed that you should invest your happiness in things that were in your power: The choices you make, your actions, in short, your integrity. Unhappiness comes from clinging to things that are not in your power: Fame, money, success and good health. External things come and go, only your choices are truly your own.

In The Black Swan, Taleb draws on another overlooked philosophical tradition, the Skepticism (or Pyrrhonism) of Sextus Empiricus, continued in modern times by Michel de Montaigne and David Hume, and reflected in the ideas of Popper and Hayek. The Skeptic tries to keep the mind in a state of suspended judgment, neither agreeing nor disagreeing, staying open to contrary facts and viewpoints. A Skeptic believes in reality, but not in human reasoning. A Skeptic laughs at certainty, and feels most comfortable as a Devil's Advocate.

Unfortunately, like the Stoic sage, the true Skeptic is an unattainable ideal. So we must do our best with what we have. Taleb's advise is to pick your battles. Skepticism is hard work, so reserve it for the beliefs and choices that really matter. Be practical. Let the theoreticians fret about the unattainability of perfect knowledge - in the real world we have to make decisions. Models based on mild randomness underestimate rare events, learn to exploit that. You can't estimate the probability of an unexpected disaster, but you may be able to limit its effects. You can't predict success, but you can expose yourself to lucky breaks.

Merely by thinking about the improbable, the unknown and the uncertain, you have an advantage over those who don't. Comforting simplifications make you over-confident. Uncertainty makes you curious and cautious. It's not much, but it might be all we have.


Comments and trackbacks

  1. Gunnar, MD, 2007-05-17
    Philosophical musings which do not accord with reality. Reality exists. Reason is the best way to attain knowledge, since reality exists, and logic is merely an affirmation that A=A, ie reality exists. Reality is not "random". It it is filled with chaotic systems, and many that defy prediction. Human society is somewhat chaotic, but hardly unpredictable. A human being has the benefit of reason, while a turkey does not.
  2. Bjørn Stærk, 2007-05-17
    "Reality is not "random". It it is filled with chaotic systems, and many that defy prediction."

    Which is the sense randomness is used in here.

  3. Gunnar, MD, 2007-05-17
    I just realized why many people, including yourself, think that no one really knows anything, that reality is unknowable. It's because people are not THINKING!

    Reality appears random, because so many people have not acquired any critical thinking skills.  Our schools are producing shallow followers, non critical thinking, anti-conceptual regurgitators. So, they take their cue from the MSM. Since I have been paying attention (~1973), I have seen the MSM construct hundreds of scenarios that while they are hot, seem like they must be the truth. But like all "manias", once something contradictory comes along, or once the action moment they were trying to influence passes, they disappear, and no one thinks about them again.

    People subconsiously pick up on the fact that the MSM is often wrong, and think, "gosh, people don't know much". Sep 11th was not a surprise random event for a lot of people. Only for people who buried their heads in the sand, believing everything the MSM puts out.

    I recommend these resources.
  4. Mark, Bellevue, 2007-06-03
    To my mind, Popper and Hayek - not skeptics, btw - developed the right way of viewing all knowledge as useful, practical, elegant in its never-to-be-realized potential of being a "true" description of objective reality; always contingent. They had a great respect for many of the methodologies that gave raise to sustained or survived knowledge yet understood that a beautiful, elegant (newtonian) description of reality was just one einstein away from being supplanted. Popper's theories of evolutionary epistemology, as you have pointed out, call for everything to be scrutinized and criticizable.

    Hayek was probably the first to fully illuminate the importance of the "anti-library". There are actually two "anti-libraries", aren't there? There's the collectively obtained and utilized corpus of knowledge - internalized genotypic knowledge and externalized phenotypic & collective social knowledge. Mises talks about complex society as a result of human action and not of human design. The there's the huge anti-library of the unknown. The set of knowledge that is, to date, removed from human (or animal) understanding. Random events coming from this domain, frankly, scare me. These are the black swans of some nearby brown dwarf perturbing the oort cloud...

    Stuart Kauffman's "At Home in the Universe" paints a very disturbing picture that its just a matter of time for one of these types of "black swans" to come along and take out a species. Sobering, too, is the thought that the overwhelming number of species that have ever existed are presently extinct.

    Bill Joy, Vernor Vinge, have very eloquently described the dangers of black swans arising from human knowledge.

  5. joshua, 2007-06-12
    so maybe we should stop being so smart and thinking so much. And just live
  6. herbie, NY NY, 2007-06-12
    "reality is unknowable. It's because people are not THINKING!" Hmm, that must come as a real shock to people who try and understand quantum mechanics:
  7. Gunnar, MD, 2007-06-13
    >> Hmm, that must come as a real shock to people who try and understand quantum mechanics

    The issue with Quantum mechanics refers to the fact that some minute aspects of reality are unobservable. Since reason starts with observation, some obscure part of reality is unknowable.

    But someone who claims that reality is unknowable is claiming that ALL of reality is unknowable. That's quite a bit different. This incredible logic error by Herbie in NY is yet another example of a lack of critical thinking skills. This applies to myself, as well as most other people.
  8. herbie, NY NY, 2007-06-13
    "But someone who claims that reality is unknowable is claiming that ALL of reality is unknowable". Well I would surmise that to make that statement you would have to know some part of reality. Assume a statement on a page that says the statement on the reverse side is true. When turned over it says statement on the other side is false.
  9. sunnyside, 2007-06-20
    any one knows what may happen to him/her in 10 years? look back at your life and see if what you have become is what you have predicted?
  10. Tor, Bergen, 2007-07-10

    I certainly found 'Black Swan' to be an interesting read. In a sense, the central ideas of the book are completely "obvious". Yet Taleb makes it very clear how selectively the mind applies these ideas, and how awful humans really are at dealing with black swans in a rational manner. The best part of this book is when flaws in my own thinking are pointed out, which happens with alarming regularity. I also enjoyed the subjective approach taken by the author.

    At the same time I cannot help but think Taleb's taken a bit too much "tran". His vicious attacks on the Gaussian obscures the central issue, namely the *fallacious use of it*, in particular that it is horribly ill-suited for black swan habitat. This does not imply that the Gaussian distribution itself is a problem, as it is quite a potent tool whenever we find ourselves looking at mediocristan.

    (One might argue that mediocristan is boring. Well, sure. I think statistics is utterly boring myself. That does not make the Gaussian useless or evil - one just has to be aware of its basic limitations. My own statistics education is in fact quite contrary to what Taleb writes in the book - in my recollection, focus was mainly on the *multitude* of different tools and distributions available, the relations between them, and when they apply.)

    For much the same reasons, I found Taleb's fawning over Dr. Mandelbrot to be somewhat beside the point. There are a great number of statistical distributions and tools to model various forms of power-law behaviour very nicely, and as I see it the sole emphasis on fractals (and chaos theory), both here and elsewhere, consists largely of hype. The issue at hand, again, is that analysis will bear little relation to the real world unless you actually use the right (mathematical) tool for the job.

    The other central lesson of the book is that a model is just that: a model. Life is complicated. Be critical of assumptions, particularly if they have large consequences. It may seem completely obvious. But Taleb makes it painfully apparent to the reader how easy it is to misstep, and how often we do.

Write a comment

Comments have been disabled.