Does Human Knowledge Double Every 5 Years?

Authorgeddon and Book Publishing

Well, I guess we need to hope that the Singularity happens before 2052. That's the year when Authorgeddon is predicted to happen, according to Wikipedia. In case you're wondering, Authorgeddon is the moment in time when the number of books published in the U.S. is expected to surpass the number of people in the U.S. who are willing to admit having read a book in the previous year.

"A study by print on demand company Lulu.com pinpoints 2052 as the year when Authorgeddon will arrive. The study arose from statistics published by R.R. Bowker, the company that controls ISBN distribution in the U.S., which reported that the number of books published in America in 2004 hit a record 195,000 -- a 14% increase over the previous high of nearly 175,000 recorded the year earlier. Bowker reports that the average annual rise in published titles over the last three years has been 14.6%."

That 14% increase looks pretty interesting--if it held up it would mean that the number of books published in the U.S. would double every 5 years. So, how does that stack up to real publishing statistics?

From Bookwire (owned by RR Bowker):

1993: 104,124

1995:  113,589

1999:  119,357

2000:  122,108

2001:  141,703

2002:  147,120

2003:  171,061

2004:  190,078

Sadly, in 2007 the number of books published in the U.S. plummeted to 172,000 new titles (I'll bet it's that damn Internet thingie). The overall CAGR is 3.65%, which would double every 19.33 years. But even looking at statistics to 2004 only (2005 might have been a blip, after all) it gives us CAGR of 5.14%. So even by cherry-picking the time frame to include the rosiest picture of growth, we get about 1/3rd of the growth we need. The concept of human knowledge doubling every five years did not originate with US book publishing statistics, I'm afraid.

The Other Half Of This Experiment

Of course, the other half of this experiment is to see if I can do something relevant and important with free tools available to all. If I can double human knowledge regarding the doubling of human knowledge, all for free and for fun, I think that might be important in and of itself.

The Outcome Fallacy and the Limits of Desktop Research

Throughout this blog, I look for positive outcomes and improved results coming from increased human knowledge regarding big problems. I rather blithely say 'if we have doubled our knowledge regarding cancer, why aren't survival rates doubling?", and things like that.

But there is no reason to assume that this should be the case, and I know that. It could be that for a problem as difficult as cancer, a doubling of human knowledge will only produce a 10% improvement in survival rates. I don't know. I don't think anybody else does either. (For God's sake, if you do know, tell me.)

Intractable problems may not have a solution at all. Or we may need to multiply our knowledge 100 times before we can solve them.

But looking for a dramatic improvement in a problematic situation seems legitimate--and we have already found one specific instance where human knowledge has dramatically improved the outcomes of an intractable problem--battlefield survival rates.

So, fair warning. I'll continue to use language loosely and describe human knowledge and outcomes with only the metrics that Google proffers. On the other hand, if this blog ever starts gettng traffic, you will find me flexible about ways of improving both my language and my metrics.

It Should Be Lumpy, Right?

We wouldn't expect a uniform doubling of human knowledge within a 5-year timeframe, right? Even if this is a law as inexorable as Moore's (and that doesn't seem as inexorable as it once was, or am I reading the Dell ads wrong?), one would reasonably expect that human knowledge would advance a little in some areas and a lot in others. You with me?

Given what the world's attention has been focused on since 2002, here are some of the areas where I think human knowledge should have increased fastest:

• Epidemiology, thanks to SARS and Avian flu
• Biofuels
• Vaccines
• Proteome
• Oil extraction from rock and shale
• Mars
• Genetically Modified Organisms
• Fishery management
• Global warming
• Solar activity
• The Long Tail
• Stem cell cultivation
• Infectious diseases

I will treat this as a proxy list of subject areas to explore. I'm going to be digging around to see if we've seen dramatic advances in any or all of these. I'll probably dedicate a few posts to each of them. Got any suggestions?

Hopeful Signs From Personal Experience

I have learned a lot in the past 5 years. I'm not sure if it's doubled my personal knowledge, and I'm not sure that even if that has taken place, it proves that human knowledge has doubled...

I've read about 125 books in the past 5 years, and some of them have been important. Jared Diamond's Guns, Germs and Steel (click here if you're shopping from the UK) may have been the most important of them.

This is partly because of the theses the author explores, well enough detailed elsewhere that I don't feel the need to regurgitate it here. But it is primarily because at the end of the book Diamond redefines what a historian needs to look at to write good history. And this is evidence of a doubling of knowledge.

My 30 seconds or so at university was spent studying anthropology, and the best part of anthropology was the freedom of the subject matter. Just about everything touched on anthropology (or is that vice-versa?), from biology to zoology. Advances in any one of the sciences extended the frontiers of anthropology. More recently, that seems to have been true of economics as well. But when Diamond told historians that good history required the same wide-ranging view of geology, geography, biology et al, he doubled the field--and maybe that doubled the knowledge of history in one blow. How liberating. Maybe those folks will start having more fun now.