Does Human Knowledge Double Every 5 Years?

Human Knowledge and Mars

NASA and the Smithsonian Institute maintain a database called the Astrophysics Data System. Harvard University is kind enough to host it here. Available to you and I is an abstracting service which will return brief descriptions of articles found by keyword searches. You see where I'm going here?

For a description of coverage and limitations of the database, click here. Then, let's have some fun.

A search for the word Mars in titles and abstracts performed on April 21 2007 returned 20,978 results. Let's pretend for the sake of argument that this represents all human knowledge ever gained about Mars. (This may be as close to the truth as we will ever get on this weblog when searching for information about fields of study--but it is obviously not true, so take it with a grain of salt.)

What is the CAGR (Compound Annual Growth Rate) in publications? Well, if we were really strict, we would measure from the date of the first document, which is... 1644. That would give us a CAGR of 2.32% and we would all get very depressed. But a more realistic starting point would be 1830, when regular publications started. This would remove only 10 documents from our total, and would yield a CAGR of 5.78%. That's much better, but far short of doubling every five years. (At that rate, human knowledge about Mars would double every 12.34 years.) Nonetheless, half of the documents in the database have been published since 1993. (Hey--that doubled in 14 years... is the rate of doubling slowing down?)

Just the metadata I got from the site is fascinating, and I'll be referring to it later, looking for short term bursts and lags (and wondering if Mars' distance from the Earth is one of the causes, for example). But here's some interesting information.

The most recent, published in April 2007, is titled "Subsurface Radar Sounding of the South Polar Layered Deposits of Mars and written by Plaut, Jeffrey J.; Picardi, Giovanni; Safaeinili, Ali; Ivanov, Anton B.; Milkovich, Sarah M.; Cicchetti, Andrea; Kofman, Wlodek; Mouginot, Jérémie; Farrell, William M.; Phillips, Roger J.; and 14 coauthors

Document #3001 is titled "Review of the trajectory and atmospheric structure reconstruction for Mars Pathfinder", written by Withers, Paul; Towner, Martin; Hathi, Brijen; Zarnecki, John

Document #6001 is titled "History and perspectives of scientific ballooning," by Frank Lefevre. Hmm. Well, document #6002 is "Stability and evolution of the climate system of Mars," written in 2001 by Stability and evolution of the climate system of Mars

Document #9001 is titled "Mobilization of dust on the Mars surface by the impact of small cosmic bodies," written in 1997 by Rybakov, V. A.; Nemtchinov, I. V.; Shuvalov, V. V.; Artemiev, V. I.; Medveduk, S. A.

Document #12001 is titled "Dry Carbonate Formation on Mars: A Plausible Sink for an Early Dense CO2 Atmosphere?", written in 1990 by Stephens, S. K.; Stevenson, D. J.

Document #15001 is titled "Stratigraphic relationships within Martian polar CAP deposits" written in 1982 by Howard, A. D.; Cutts, J. A.; Blasius, K. R.

Document #18001 is titled "The major Martian dust storms of 1971 and 1973" written in 1974 by Martin, L. J.

The very first document, written in 1644 by Grandami, Jacques, is titled "Nova demonstratio immobilitatis terrae petita EX virtvte magnetica."

If that's a little dense for you, the second, written in 1672 by Flamstead, John, is "An Extract of Mr. Flamstead's Letter Written to the Publisher from Derby Novemb. 16. 1672. Concerning the Appulses, by Him Calculated for A. 1673. of the Moon, and the Other Planets to Fixed Stars; Together with an Observation of the Planet Mars, Made by the Same"

Authorgeddon and Book Publishing

Well, I guess we need to hope that the Singularity happens before 2052. That's the year when Authorgeddon is predicted to happen, according to Wikipedia. In case you're wondering, Authorgeddon is the moment in time when the number of books published in the U.S. is expected to surpass the number of people in the U.S. who are willing to admit having read a book in the previous year.

"A study by print on demand company Lulu.com pinpoints 2052 as the year when Authorgeddon will arrive. The study arose from statistics published by R.R. Bowker, the company that controls ISBN distribution in the U.S., which reported that the number of books published in America in 2004 hit a record 195,000 -- a 14% increase over the previous high of nearly 175,000 recorded the year earlier. Bowker reports that the average annual rise in published titles over the last three years has been 14.6%."

That 14% increase looks pretty interesting--if it held up it would mean that the number of books published in the U.S. would double every 5 years. So, how does that stack up to real publishing statistics?

From Bookwire (owned by RR Bowker):

1993: 104,124

1995:  113,589

1999:  119,357

2000:  122,108

2001:  141,703

2002:  147,120

2003:  171,061

2004:  190,078

Sadly, in 2007 the number of books published in the U.S. plummeted to 172,000 new titles (I'll bet it's that damn Internet thingie). The overall CAGR is 3.65%, which would double every 19.33 years. But even looking at statistics to 2004 only (2005 might have been a blip, after all) it gives us CAGR of 5.14%. So even by cherry-picking the time frame to include the rosiest picture of growth, we get about 1/3rd of the growth we need. The concept of human knowledge doubling every five years did not originate with US book publishing statistics, I'm afraid.

Happy Coincidence

A chart I put up on the weblog cited Derwent's World Patents Index as a source. It is found at Thomson.com, and it has information regarding 31.66 million patent documents. (Okay, let's future proof this--in 2012, will it have information on 63 million patent documents?)

They charge to get in and look around, so I'm hovering on the periphery, seeing if I can find more meta-data about the database. For example, an article titled 'Open access to the medical literature: How much content is available in published journals?' by Marie McVeigh and James Pringle, published in 2005, indicates that about 20% of medical literature published in 174 journals since 1992 is 'open access', that is, freely available online. They estimate that a bit less than 60% of medical literature in that time period can be accessed electronically. We also find that Thomson's ISI Web of Knowledge has 20 million users from 81 countries accessing a database of 22,000 journals, 23 million patents and much more. All good supporting data for my second hypotheses, about the increasing availability of human knowledge. But I would love to see the growth rate for new data coming into that database, and I can't find it.

However, I did find this--an article called 'Eastward ho! --The geographic drift of global R&D' written in March 2007 by Bob Sternbridge. It touches on a number of topics I've written about below, and has some interesting numbers:

• The number of computing patents filed in China has risen from 2,000 to 20,000 in five years! Clear doubling in the right time frame. (Pity the increase was not replicated in Japan or Korea, which were also included in the study)
• The number of communications patents filed in China has also risen in the same time frame, from 8,000 to 30,000--again a doubling! Sadly, again not replicated in Japan or Korea.
• More importantly, the total number of Chinese patent applications is growing by 25% per annum. But the article then says that Japan and US patent filings have been flat or declining in recent years. And a lot of the patent filings are by multinational corporations
• R&D spending in China has risen from 89.6 billion yuan in 2000 to 245 billion yuan between 2000 and 2005
• The number of researchers in China grew 77% over the ten years from 1995 to 2004

I think it will take me a couple of hours to digest this and incorporate it into previous posts. More later.

An Anecdote Regarding Hypothesis 2

The second hypothesis I am investigating with this weblog is that the availability of existing human knowledge is expanding exponentially. If true, I think it would reduce the burden of expectations on those who think new human knowledge is doubling every five years, if that turns out not to be the case. If a brilliant Chinese programmer can see the appropriate article online today, where he couldn't in the past, it might guide his efforts more effectively--pretty common-sensical, I think.

So, this article from the Dalhousie News is interesting. In it, we see that the Nova Scotian university now has access to 15,000 academic journals--up from 4,000 five years ago. Essentially, these common-sense Canadians formed a buyer's club and got good deals on academic journals online. Good for them.

I'm starting to like Hypothesis Number Two. Every time I search for evidence regarding the increased availability of human knowledge, I feel like I'm swimming with the river, easily finding data that seems reliable and pertinent. Every time I look for evidence about the primary hypothesis (that actual human knowledge is doubling every five years), I really feel like I'm swimming against the current.

Whew--We Find That Something Has Doubled

It would have been truly strange not to find some area where there is evidence of a doubling of activity. I actually think we'll find more as we go on. But the past two weeks have been somewhat discouraging, so I'm happy to report that we see clear evidence of a doubling in the growth rate of patent filings in China over the past 5 years, and a doubling of the total number of academic patent filings worldwide between 1995 and 2000.

An article titled 'Rapid Patenting Growth By Academic Institutes In The People's Republic Of China' by Simon Pratt, published in June 2005 shows a doubling of the growth rate (not the actual numbers) of patent filings by academic institutions in China, and a doubling of academic patent filings worldwide by universities between 1995 and 2000 (see chart below). Sadly, that growth rate has not been sustained since 2000.

It is not unalloyed good news--worldwide, academic patent filings are a response to governmental decisions to allow universities to retain, license and exploit intellectual property, so there may just be substitution here--the same patent might previously have been filed by an individual at the academic institution or someone who had access to the reseach. But I'm so happy to find something relevant that has doubled in five years that this seems like a minor quibble.

Here is the chart showing the number of academic patent filings worldwide since 1980. As with all the charts on this weblog, clicking on the chart will bring it up in a readable size.

More on Human Knowledge and Global Warming

In an article titled 'General Circulation Models of the Atmosphere,' Spencer Weart tell us that John von Neumann began advocating using computers to study the climate back in 1946, shortly after his computer ENIAC became operational. Getting grants from the Navy, the Air Force and the Weather Bureau, the study of climate began.

So 1946 was the starting point. For the next 25 years, the American Meteoroligical Society's  entire spectrum of journals (including the Journal of Applied Meteorology, the Bulletin of the American Meteorological Society, Journal of Atmospheric Sciences, Journal of Physical Oceanography, Monthly Weather Review and more...) published a total of 111 papers containing the phrase "climate models" somewhere in the text or title. The first one was titled 'The Unusual Weather and Circulation of the 1948-1949 Winter,' by William H. Klein. It was published in April 1949, 58 years ago. Two articles were published containing that phrase in 1950.

Let's pretend that each paper published with the phrase "climate models" is one unit of advancing human knowledge. If human knowledge doubles every five years, how many papers should be published in 2005? We know that the percentage growth should be 14.87%, and we know we should expect to see 2,048 articles in 2004. So... drum roll please... it's 881. Less than half of what we would expect. Lest you think it was just an off year, in 2005 it was 899 and in 2006 it actually dropped to 894.

I can think of a dozen possible explanations--a presidential administration unfriendly to the concept, waiting on the IPCC to publish findings in 2007... but the fact is, if we have doubled our knowledge of climate models in the past 5 years, we're not writing about it in America.

Is the rest of the world making up for America's shortfall? Well, not if you're depending on the International Journal of Meteorology, which returns 8 results for a search on climate models over a 25-year span. (But I think it's a search issue.) The International Journal for Climatology returns 37 results. I'm sure I'm missing important sources, but none are jumping up and biting me. The Royal Meteorological Society has 36 results. The UK government is considerably more supportive of research into global warming than the current American administration. I really thought there would be more.

Human Knowledge and Global Warming

A Google search for "improvements in climate models" returns 2.64 million results. Of those, Google Scholar claims 138,000 articles about the search term.

Of the first 20 results in Google Scholar, only one was written after the year 2000. Published in the Journal of Climate, it describes The Community Climate System Model, Version 2.  However, searching with the same term on the Journal of Climate's search engine returns 3,348 results.  Sorting in reverse chronological order, the first 10 were published this month, April 2007. In fact, I counted 279 entries published in the past year. Here are the annual results for fifth years historically:

2006: 273

2001: 203

1996: 162

1991: 54

1986: 49

1981: 24

1976: 14

1971: 3

1966: 0

Global warming is a popular field for researchers and government funding is there. Advances in computer hardware and software means that improvements in global climate models are possible, as well as desirable.

The fact that the last time entries doubled over a five-year span was between 1991 and 1996 is a bit disturbing. A new field of science with optimum research conditions that cannot sustain doubling of cutting edge knowledge after a mere 25 years cannot bode well for those who depend on doubling every five years.

Maybe the papers are four times as good...

A Marker is Put Down

Derek J. de Solla Price, writing in 1965 in Science, gives us a benchmark. In a paper entitled Networks of Scientific Papers, (PDF) exploring the links between citations and references, de Solla Price writes:

"We can calculate roughly that, since the body of world literature has been growing exponentially for a few centuries and probably will continue at its present rate of growth of about 7 percent per annum, there will be 7 new papers each year for every 100 previously published papers in a given field." A footnote refers to his 1963 book, Little Science, Big Science, published by Columbia University Press.

So, we return to the Rule of 72 website to find that a 7% growth rate would result in the doubling of scientific publications every 10.3 years.

Not good news, but that was in 1965. Is there more recent information?

Yes, and its worse news. The National Science Foundation published a chart showing growth in scientific and technical papers for selected countries. It's here--click on the image to see a readable version.

For the OECD, it shows very slow growth (1.5%) between 1987-1994, and no growth (0.1%) from 1995-1999.

This is really bad news. The higher growth rate doubles in 48 years, the lower growth rate in 720 years. I hope we've been making up for lost time since 1999...

One indication why this may have happened is reduced government funding for research and development. The graph above comes from a 2002 report called Science and Engineering Indicators. It writes, "The decline in the share of government funds for R&D is a key trend common to all major industrial nations and many other OECD countries. In the mid-1980s, these nations derived an average of 45 percent of their R&D funds from government sources; by 1998, this figure had fallen to less than one-third. The relative retrenchment reflects the broad growth of industrial R&D, reductions in defense R&D in some key nations, and broader economic and spending constraints on governments. As a consequence, government funding for industrial R&D performance also fell, averaging 23 percent in 1983 but only 10 percent in 1998 for OECD as a whole."

An Almost Eureka Moment

Bless you Tatyana Dorokhova, and bless the Astronomical Observatory of Odessa National University!

In your Abstract for your article, Exponential Growth of Scientific Information and Orientation in World Wide Web (published in the July 2-5, 2002 issue of Library and Information Services in Astronomy IV), you write,

"A swift growth in the number and length of publications in the  narrow, but advanced field of astronomy is shown with certain research examples."

However, although your title refers to exponential growth, your numbers do not. You quote H. Abt, an editor of the Astrophysical Journal, as writing "The average number of papers increased exponentially by nine percent before mid 1970's and by four percent per year thereafter."

Sadly, this look at postwar astronomy seems to show an explosion of knowledge after the war, aided by improvements in astronomical tools (radio telescopes, etc.), and then a return to a very respectable growth rate that would double human knowledge in this sector in 18 years. (Lest you think I actually did the math, I refer you to the Rule of 72 website, which has a doubling calculator.) Even the 9% growth rate before the mid-1970's would yield a doubling only once every 8 years.

Once again, we are frustrated in our attempts to show doubling of human knowledge every five years.

Are there more people generating knowledge?

One proxy for measuring the creation of knowledge could be the number of people engaged in that activity. Not all knowledge is scientific (is it?), but governments tend to keep an eye on how many people are either practicing science or studying it. In the U.S. and the U.K., people are worried that the numbers are decreasing. Does the rest of the world make up for the shortcomings?

Well, someone has done some work on this: Via Google Answers we see... (and boy, is Google Answers great)

• The number of scientists in the United States doubled very quickly in the past. "the number of scientists increased from 150,000 to 2,685,000 between 1950 and 2001" but growth from 2001 projected to 2010 is much more modest, with only medical scientists having doubled in the previous decade and projected to grow at the substantial rate of... 26.5% in this decade we're in.
• According to the OECD Observer (January 2004), the number of researchers in the OECD rose 42%  from 1990-2000, which won't get the doubling job done. The article talked briefly about foreign researchers filling the gap...

So how about those Chinese scientists? Those Indian software geniuses?

• An article in Intellectual Property Watch says the number of patents filed in India has increased 50% in the past 5 years, without offering numbers. That's a big increase, but only half of what would be needed to double human knowledge, if we used patent filings as a proxy for new human knowledge (and found that percentage increases were similar worldwide). And if we are counting on India and China to fill a gap caused by U.S. and U.K. reductions in research and researchers, this would not support that thesis.
• In fact, patent applications to the World International Patent Office last year reached a record 145,300 in 2006, according to this article in Inc.com. But the article notes that this increase was only 6.4% year-on-year, and the United States accounted for one third, followed by Japan and Germany. This shows no evidence of 5-year doubling (you need an annual growth rate of 14.78% to double in five years), and no evidence that Asia will replace more traditional centers of knowledge production.