Does Human Knowledge Double Every 5 Years?

Oncology

Entrez is the integrated, text-based search and retrieval system used at NCBI for the major databases, including PubMed, Nucleotide and Protein Sequences, Protein Structures, Complete Genomes, Taxonomy, and others. It is a service of the National Library of Medicine and the National Institutes of Health. However, it is included within Scirus, so the results will be taken from Scirus.

Searching with the term 'oncology' returns 148,053 results. 57,936 were published between 2002 and 2006 (inclusive). That's only 39% of the total, so it doesn't look like human knowledge of cancer has doubled in the past five years on that basis. In the preceding five years (1997-2001), only 38,153 results were published. That means that the most recent five year period did not even double the preceding five year period.

The CAGR for academic publications for oncology is 9.23% (since 1920), which would double every 7.85 years. Patent growth since the first filing in 1973 is 28.62%, which would double every 2.75 years. Using the same base date of 1920 would yield a CAGR of 10.02%, doubling every 7.26 years. I don't know which date distorts the picture less.

And the fact is, we don't know how many times we will have to double our knowledge of cancer before we will be able to make a big difference in the lives of those who have it. I have to say, I think I'd rather see the growth rate we note in nanotechnology here--and yes, I think I am saying I would make the trade off. I'm not advocating central planning of scientific research--I'm just wondering if research incentives are correctly allocated.

Now champions of nanotechnology wll say that nanotechnology in general has many direct applications to medicine in general, and further that some nanotechnology research is specifically concerned with oncology. And it's true. My reservation comes from my suspicion (for which I have zero evidence) that this is engineering led (hey--come here and see what my new toy can do!) rather than medicine-led (can you build something that will do x, y, and z in the following sequence and make sure it does not do a, b or c?) If my suspicion is correct, this wil prolong the search for solutions rather than shorten it.

Is there a truce in the war on cancer? Best put out that cigarette. Because we see that, although oncology did see a doubling of knowledge every five years up to 1991, that hyper-growth tailed off afterwards. Note that hypergrowth started before Nixon's war on cancer. Note that growth since hypergrowth stopped has been perfectly respectable--14,000 papers published in 2006 (more in 2006 than in the entire decade of the 80's), and 4,000 patents filed. And note especially that growth in patents has continued to double every five years. Make of it what you will.

The figures are below. Click to see the numbers.

So Where Did This Crazy Idea Come From?

We saw below that President Clinton said that human knowledge doubled every five years in at least two different speeches. We saw below that a respected German professor in Artificial Intelligence said that the number of scientific publications doubled every five years. But neither of them cited a source. I doubt if President Clinton did the research, and I can't find anything published by the German professor that deals with this. So where did it come from?

On several searches I've seen reference to UNESCO as a source of this information. And indeed, at the 11th General Conference titled Universities--Gateway to the Future, in 2000 Mahdi Elmandjra said, "Richard Knight estimates that the total knowledge of mankind doubles every seven years and adds,"90% of the present knowledge of mankind has been produced in the course of the last thirty years. But if we define knowledge as the capacity to survive on earth in a sustainable manner then 90% of human knowledge has been lost during the last 30 years."

But Richard Knight (referenced in other papers of the conference) is a professor of architecture at the University of Genoa in Italy. I cannot find anything published by him--and the quote says seven years, not five.

Where does this idea come from?

Relevant Research from 1988

It would have been very strange if I were the only one asking this question and doing some research. I'm glad to find some prior work on this, not least because it means I'm not the only one who cares.

A report on the International Conference on Fifth Generation Computer Systems in 1988, written by Jakob Nielson, refers to research by Jorg Siekmann of the University of Kaiserslauten in the then West Germany.

Nielson writes, "Siekmann reported that the number of scientific journals doubles every 15 years, the number of books in university libraries doubles every 10 years, and the number of scientific publications doubles every 5 years. The knowledge explosion has already reached a level where in chemistry it is often cheaper to conduct a possible duplication of an experiment than to search the literature to find whether the result of a previous experiment can be used." That last bit is scary--I wonder if it's still true. Let's see if we can find more about Siekmann and his research.

Google likes him--a little too much. There are 108,000 search returns on his name.

The University has some material about him: Jörg H. Siekmann is the head of several research groups working in artificial intelligence at the "Universität des Saarlandes" and at the German Research Centre for Artificial Intelligence (DFKI) in Saarbrücken. He is a professor in the department of computer science and a director at the DFKI. The five research groups are:

Automated Reasoning for Mathematics: OMEGA 
Competence Center e-Learning
Formal Methods: VSE
Multi-Agent Systems
The IT-Security Evaluation Centre

Currently he is the chairman of the collaborative research centre "Resource-adaptive cognitive Processes" (SFB 378) funded by the German Science Foundation (DFG). He is chairman of the network of excellence on computational logic (CoLogNet), and vice president of the International Federation on Computational Logic (IFCoLog) . His main research interests are Artificial Intelligence, Automated Reasoning, Multiagent Systems and e-Learning in Mathematics.

But I can't find any more information about his statement at the conference. However, I did stumble across something quite interesting that might lead to the origin of the statement  Human Knowledge Doubles Every Five Years. See next post (above).

How Important Is All This?

Now that I've been doing this for a month, the question of how important this all is is becoming more relevant. When you're in the middle of the hard slog of gathering data, it's normal to get frustrated. It's also normal to get a little grandiose in your thinking about the worth of what you're doing, partly to motivate yourself to keep going.

So how important is all this? If I come up with the finding that human knowledge doubles every ten years instead of every five, who will care? What will it change?

I mentioned a few things in an earlier post that I am starting to think are actually relevant questions for society. If we are convinced that human knowledge doubles every five years, then we don't need to quit smoking, as cancer will be cured before it strikes us. Similarly, we don't need to worry about pollution, because we'll sort it out before the effects can truly harm society. You can adopt this lazy attitude about any of the problems our world confronts.

But it would be a mortal pity if human knowledge doubles every six or ten or twelve years and you die of cancer the year before the cure is discovered instead of the year after.

The idea that human knowledge doubles every five years is widespread. A Google search using the term human knowledge doubles every five years returns 1,680,000 results (obviously only a fraction are relevant).

If that information is wrong, and individuals are making important decisions and organisations are misallocating resources in this mistaken belief, then yes, this research is important and worth doing. Gee, I feel reassured.

One Month Progress Report

Well, this weblog has now been up for 30 days. I had actually intended to do all the basic reseach in this time frame, but life has somehow interfered. A major reorganisation at work has slowed me down.

Nonetheless, we've managed to explore the growth in human knowledge in a number of areas, so let's recapitulate:

1. We do find a doubling of the evidence of human knowledge (journal publications and patents) in some sectors, such as nanotechnology and global warming, new fields that double from a low base. I'm tempted to label this the Manhattan Project effect, but I'm not going to do that until I study that sector since 1945.
2. We do not see a doubling in areas with a longer time series, such as astrophysics or the study of Mars, although there are periods of impressive growth in each.
3. In some areas, we see different trends for patents as opposed to journal publications. For example, in epidemiology we see an 8.85% CAGR in journal publications in the recent past, while patents have grown by 22.76% annually since 1987.
4. We do see evidence that human knowledge is available to twice as many people, and that has grown at a rate that more than doubles every five years. (Just looking at the growth of Wikipedia and the numbers of people connected to the Internet makes that fairly obvious).
5. We do not see evidence that the number of scientists and researchers doubles every five years, and it is not at all obvious that emerging country growth is making up for a slowdown or actual decline in OECD numbers.

But there's a lot more work to do, and any of those five points could change. I'm having fun doing this, so we'll just carry on.

Epidemiology

Well, Scirus, the database of Scientific knowledge I've been using frequently on this weblog, has a lot of material relating to epidemiology. I feel a bit vulnerable relying on this source so heavily--if anybody has suggestions for freely available information, I will happily (well, grumblingly and slowly and swearing occasionally) replicate these searches on it.

Searching for epidemiology on Scirus returns 1,500,983 results. 315,678 of these are journal articles. 8,530 are patent results and the other million plus are 'other' web results. So, focusing on journal articles and patent results since 1920, let's see if the numbers double every 5 years.

Human knowledge (as expressed by journal articles published) about epidemiology has not doubled in the past five years. Only 106,722 of the 315,678 journal articles were published between 2001 and 2006. It's truly a triumph of the human mind, but it is not doubling. It is a compound annual growth rate (CAGR) of 8.85%, which would result in a doubling every 9 years. Looking at the time series, we see dips and lulls and a steadily progressive trend line, with a couple of jumps. When I get the graphs prepared, it will be easier to see.

Overall, CAGR for epidemiology is 6.22%, which doubles every 11.49 years. Overall growth in patents is 17.37%, which doubles every 4.33 years.

However, backing out the 2007 patents (198) leaves a total of 8,332. Of that number, 5,444 (65%) were filed between 2001 and 2006. So while academic investigation did not double in the past five years, commercial research and development more than did so. Indeed, epidemiology patents appear to have more than doubled in every five year period since 1987, when 19 patents were filed, a CAGR of 22.76%.

I suppose, if we treated one journal article and one patent as exactly equal, we could add them up and compare the totals. But, as I certainly don't have any reason to think that might be the case, I'm not going to. The figures are below if you want to do it, though.

Click below to see the data.

How Would We Define Victory?

Well, time to define a few things here. We are trying to discover if human knowledge doubles every 5 years. To do this, we are testing three hypotheses:

1. We learn so much new information that the actual stock of knowledge is double what it was 5 years ago. We are testing this by seeing if scientific publications and patents are doubling every 5 years. We also look and see if there are twice as many scientists and researchers as 5 years ago.
2. That new and old human knowledge is accessible and usable to twice as many people as it was 5 years ago. We're looking at things like the size and reach of websites like Wikipedia, and also referring to our search for more creators and users of information.
3. We are disproving incorrect hypotheses at such a rate that we are doubling human knowledge by unlearning untruths, which corrects mistakes and frees up time for productive learning.

But this is a bit simplistic.  If I know one thing and learn another, have I doubled human knowledge or just what I have learned? Is a learned fact human knowledge before it is published?

If I know one thing and, without learning another, communicate that one thing to another human being, does that double human knowledge because two of us now know it? Or is that just repetition?

If I discover an hypothesis is not correct, does it make any difference if others continue to hold the old and incorrect belief?

And then we must make things difficult by defining knowledge. Many of you will have seen the pyramid that has data at the bottom, information on top of it, then knowledge and then wisdom. The implication is that data is a fact revealed, information is the fact in context, knowledge is is the fact absorbed and assimilated, and wisdom is right action based on knowledge as experienced. But is that an accurate or adequate definition of knowledge? Is it sufficient?

Wikipedia has a definition--it's a bit long, but the start is promising. Of course the start is a quote from Oxford's English Dictionary. "Knowledge is defined (Oxford English Dictionary) variously as (i) facts, information, and skills acquired by a person through experience or education; the theoretical or practical understanding of a subject, (ii) what is known in a particular field or in total; facts and information or (iii) awareness or familiarity gained by experience of a fact or situation." Yeah, that's it.

So does learning that the Earth is more than 6,000 years old mean anything if millions don't believe it? If creationists make learned scientists spend time in court patiently explaining evolution instead of working in the laboratory--their true 'field of the Lord,' have we learned anything? Does knowledge have to be unanimously held, or unanimously available?

Boy, where's a philosopher when you need one?