From long tail to variegated tails

For close to 200 years, social scientists lived under the tyranny of normal distribution, which focused their attention on most probable outcomes and on the averages, means and medians. We now need to be careful not to commit the same error with the long-tail distribution. Book on this subject, The Long Tail, became a bestseller and a required reading for management consultants and investment analysts. Its author, Chris Anderson, has been promoting the long-tail persistently and intelligently, using all traditional and new media channels at his disposal as editor of Wired. We are firm believers in the pertinence of long-tail. There is however a risk of overselling and overstating the importance of long-tail in the economy and society.

Under a long-term distribution, probability of various events is widely dispersed rather than concentrated: low frequency events represent a larger share of the total than under normal distribution. Thus, in order to understand the future and earn money in e-commerce, one no longer needs to focus exclusively on most probable events or on biggest selling items. The main message of long tail is the increasing variety. This message also applies to statistical distributions itself. Normal distribution will remain dominant, as long as random events dominate all aspects of economic and social interactions. Furthermore, there is more than one family of long-tail distributions. Those discussed by Anderson follow the power law, according to which relative frequency of events is inversely proportional to their size. One example of power law is the frequency of earthquakes: bigger earthquakes happen less frequently than the big ones. The relationship between the two is regular and scale independent. The only parameter that changes is speed at which the frequency decreases. Anderson argues that as the costs of search on the Internet are drastically cut, frequency decreases more slowly. However, while this may be true for purchase of books and records, it is not necessarily true for the ranking of online bookselling and music shops (or search engines for that matter). Such ranking usually follows Zipf or Pareto power law, which reflects oligopolistic concentration, where the biggest firm is twice as big as the second largest, which three times bigger than the third largest, etc. Those laws are also called 20/80, as 20% of firms generate 80% of revenues.

An important characteristic of Anderson distribution is that there is no significant variation in the value of events. If this assumption is relaxed and large variation is allowed, a new type of distribution appears - the so called Gauchy distribution. Under Gauchy, events at the end of tail can take extreme values. Thus Gauchy allows the modelling of low-frequency, high impact events. Airline accidents provide a good example here. These are considerably, very considerably, less frequent than car-driving accidents (official US statistics indicate about one accident for 600 000 flying hours.) Yet, should the accident happen, the chances of survival are much much lower for airline accident than for a car crash. The implication of this fact is that air safety cannot be approached using normal distribution and looking at most probable events (which is no accident). The notion of average has no operational meaning. Airline operators need to focus on extreme cases and plan their safety measures for extremely low probabilities.

Cauchy distribution is significant in financial markets. Fro instance, Cauchy distribution arises when one seeks to compare listed UK and German software companies. If one ignores value, UK software sector appears better performing than the German one: more listed companies in a greater variety of sectors and application. But one cannot ignore the value: the leading German company is SAP and its market cap is close to 50 billion euros, ten times larger than the best-capitalised UK company (Sage plc), and over thirty times larger than the second biggest German company (Software AG).

Thus, while long-tail distribution directs our attention to weak signals, Cauchy distribution demonstrates the non-linearity of low probability events. Both of them (and a normal distribution as well) are necessary to understand the interactions of chance and purposefulness in our complex world.