Does Human Knowledge Double Every 5 Years?

Funding Frustration

Well, the previous two posts give a fairly good picture of how the U.S. federal government funds research (note that we did not examine state and local funding, nor private sector funding of R&D, nor defense funding, for that matter).

I feel satisfied that I know what's happening in the U.S. But my real frustration is that you, dear reader (icy ironic tone, please) will quite reasonably expect me to find this out for the world.

But while it only took 7 hours to compile the information for the U.S. (due to better and more available statistics), it will take weeks to get a picture of what's happening in the world.

I think I'll put a tip jar on this blog.

Into The Jungle Of Research Funding

Oh, this hurts. I did not want to go into research funding. But now that it is fairly clear that the rate of progress varies significantly by sector, we have to examine the inputs.

The first relevant document found is a PDF from the European Commission. It states that it wants the EU to increase research 'efforts' to 3% of GDP by 2010. Ambitious, that. The document talks about the importance of 'basic research,' as opposed to directed, I assume, which is encouraging. It actually serves as a good basic primer on how funding is structured in Europe, the U.S. and Japan, but while it talks about funding for specific programmes, it doesn't talk about levels of funding at the national or international level. So we move on.

Curious Cat, a blog that bills itself as a Science and Engineering blog, posts research funding for U.S. academic R&D for the past few years (see below in the extended post). More importantly there's a link to the NSF where we find an Excel spreadsheet (also in the extended post below) that charts all federal spending on research for science and engineering between 1970 and 2003. There are other charts available for viewing and downloading here.

In 1970, combined spending by all U.S. Federal agencies for research in science and engineering totaled $4.9 billion (USD). By 2003, that had grown to $53.3 billion (USD). Not bad.

The compound annual growth rate for U.S. Federal funding for research into science and engineering is 7.28%, which means it doubles every 9.86 years.

However, adjusting for inflation to 2006 U.S. dollars, it reads a bit differently. In 2006 dollars, the spend in 1970 was the equivalent of $26 billion, while the 2003 spend rises on recalculation to a total of £58 billion. That provides a CAGR of 2.39%, which would double every 29.35 years. A bit sad, that.

But the spreadsheet breaks spending down by discipline, as well. I'll address that in the next post.

Outcomes

Advances in human knowledge should make life better. That's hard to measure. However, we can measure if human life gets longer (it does), healthier, richer (if not more rewarding), etc.

The rest of this post is philosophizing--feel free to skip. And in a previous post, I talk about the Outcomes Fallacy. But if you want thinking on the issue of human knowledge and outcomes, click to read more.

Oncology

Entrez is the integrated, text-based search and retrieval system used at NCBI for the major databases, including PubMed, Nucleotide and Protein Sequences, Protein Structures, Complete Genomes, Taxonomy, and others. It is a service of the National Library of Medicine and the National Institutes of Health. However, it is included within Scirus, so the results will be taken from Scirus.

Searching with the term 'oncology' returns 148,053 results. 57,936 were published between 2002 and 2006 (inclusive). That's only 39% of the total, so it doesn't look like human knowledge of cancer has doubled in the past five years on that basis. In the preceding five years (1997-2001), only 38,153 results were published. That means that the most recent five year period did not even double the preceding five year period.

The CAGR for academic publications for oncology is 9.23% (since 1920), which would double every 7.85 years. Patent growth since the first filing in 1973 is 28.62%, which would double every 2.75 years. Using the same base date of 1920 would yield a CAGR of 10.02%, doubling every 7.26 years. I don't know which date distorts the picture less.

And the fact is, we don't know how many times we will have to double our knowledge of cancer before we will be able to make a big difference in the lives of those who have it. I have to say, I think I'd rather see the growth rate we note in nanotechnology here--and yes, I think I am saying I would make the trade off. I'm not advocating central planning of scientific research--I'm just wondering if research incentives are correctly allocated.

Now champions of nanotechnology wll say that nanotechnology in general has many direct applications to medicine in general, and further that some nanotechnology research is specifically concerned with oncology. And it's true. My reservation comes from my suspicion (for which I have zero evidence) that this is engineering led (hey--come here and see what my new toy can do!) rather than medicine-led (can you build something that will do x, y, and z in the following sequence and make sure it does not do a, b or c?) If my suspicion is correct, this wil prolong the search for solutions rather than shorten it.

Is there a truce in the war on cancer? Best put out that cigarette. Because we see that, although oncology did see a doubling of knowledge every five years up to 1991, that hyper-growth tailed off afterwards. Note that hypergrowth started before Nixon's war on cancer. Note that growth since hypergrowth stopped has been perfectly respectable--14,000 papers published in 2006 (more in 2006 than in the entire decade of the 80's), and 4,000 patents filed. And note especially that growth in patents has continued to double every five years. Make of it what you will.

The figures are below. Click to see the numbers.

Relevant Research from 1988

It would have been very strange if I were the only one asking this question and doing some research. I'm glad to find some prior work on this, not least because it means I'm not the only one who cares.

A report on the International Conference on Fifth Generation Computer Systems in 1988, written by Jakob Nielson, refers to research by Jorg Siekmann of the University of Kaiserslauten in the then West Germany.

Nielson writes, "Siekmann reported that the number of scientific journals doubles every 15 years, the number of books in university libraries doubles every 10 years, and the number of scientific publications doubles every 5 years. The knowledge explosion has already reached a level where in chemistry it is often cheaper to conduct a possible duplication of an experiment than to search the literature to find whether the result of a previous experiment can be used." That last bit is scary--I wonder if it's still true. Let's see if we can find more about Siekmann and his research.

Google likes him--a little too much. There are 108,000 search returns on his name.

The University has some material about him: Jörg H. Siekmann is the head of several research groups working in artificial intelligence at the "Universität des Saarlandes" and at the German Research Centre for Artificial Intelligence (DFKI) in Saarbrücken. He is a professor in the department of computer science and a director at the DFKI. The five research groups are:

Automated Reasoning for Mathematics: OMEGA 
Competence Center e-Learning
Formal Methods: VSE
Multi-Agent Systems
The IT-Security Evaluation Centre

Currently he is the chairman of the collaborative research centre "Resource-adaptive cognitive Processes" (SFB 378) funded by the German Science Foundation (DFG). He is chairman of the network of excellence on computational logic (CoLogNet), and vice president of the International Federation on Computational Logic (IFCoLog) . His main research interests are Artificial Intelligence, Automated Reasoning, Multiagent Systems and e-Learning in Mathematics.

But I can't find any more information about his statement at the conference. However, I did stumble across something quite interesting that might lead to the origin of the statement  Human Knowledge Doubles Every Five Years. See next post (above).

How Would We Define Victory?

Well, time to define a few things here. We are trying to discover if human knowledge doubles every 5 years. To do this, we are testing three hypotheses:

1. We learn so much new information that the actual stock of knowledge is double what it was 5 years ago. We are testing this by seeing if scientific publications and patents are doubling every 5 years. We also look and see if there are twice as many scientists and researchers as 5 years ago.
2. That new and old human knowledge is accessible and usable to twice as many people as it was 5 years ago. We're looking at things like the size and reach of websites like Wikipedia, and also referring to our search for more creators and users of information.
3. We are disproving incorrect hypotheses at such a rate that we are doubling human knowledge by unlearning untruths, which corrects mistakes and frees up time for productive learning.

But this is a bit simplistic.  If I know one thing and learn another, have I doubled human knowledge or just what I have learned? Is a learned fact human knowledge before it is published?

If I know one thing and, without learning another, communicate that one thing to another human being, does that double human knowledge because two of us now know it? Or is that just repetition?

If I discover an hypothesis is not correct, does it make any difference if others continue to hold the old and incorrect belief?

And then we must make things difficult by defining knowledge. Many of you will have seen the pyramid that has data at the bottom, information on top of it, then knowledge and then wisdom. The implication is that data is a fact revealed, information is the fact in context, knowledge is is the fact absorbed and assimilated, and wisdom is right action based on knowledge as experienced. But is that an accurate or adequate definition of knowledge? Is it sufficient?

Wikipedia has a definition--it's a bit long, but the start is promising. Of course the start is a quote from Oxford's English Dictionary. "Knowledge is defined (Oxford English Dictionary) variously as (i) facts, information, and skills acquired by a person through experience or education; the theoretical or practical understanding of a subject, (ii) what is known in a particular field or in total; facts and information or (iii) awareness or familiarity gained by experience of a fact or situation." Yeah, that's it.

So does learning that the Earth is more than 6,000 years old mean anything if millions don't believe it? If creationists make learned scientists spend time in court patiently explaining evolution instead of working in the laboratory--their true 'field of the Lord,' have we learned anything? Does knowledge have to be unanimously held, or unanimously available?

Boy, where's a philosopher when you need one?

An Anecdote Regarding Hypothesis 2

The second hypothesis I am investigating with this weblog is that the availability of existing human knowledge is expanding exponentially. If true, I think it would reduce the burden of expectations on those who think new human knowledge is doubling every five years, if that turns out not to be the case. If a brilliant Chinese programmer can see the appropriate article online today, where he couldn't in the past, it might guide his efforts more effectively--pretty common-sensical, I think.

So, this article from the Dalhousie News is interesting. In it, we see that the Nova Scotian university now has access to 15,000 academic journals--up from 4,000 five years ago. Essentially, these common-sense Canadians formed a buyer's club and got good deals on academic journals online. Good for them.

I'm starting to like Hypothesis Number Two. Every time I search for evidence regarding the increased availability of human knowledge, I feel like I'm swimming with the river, easily finding data that seems reliable and pertinent. Every time I look for evidence about the primary hypothesis (that actual human knowledge is doubling every five years), I really feel like I'm swimming against the current.

non-Conclusions on Medical Knowledge

After writing the previous three posts, I have some thoughts about human knowledge and medicine. I reserve the right to change my mind on any of this:

• Human knowledge about matters medical has increased dramatically, but does not double every 5 years
• Medical outcomes have increased human health and longevity, but these outcomes do not 'follow the money,' nor the research
• Managing newly acquired medical knowledge is now the key issue impeding further progress

The second hypothesis I am testing is whether or not the increasing availability of knowledge to a wider population can satisfy the criteria of doubling human knowledge. I believe that medicine may offer strong evidence for this.

The invention of the polio vaccine did not eradicate polio. It was the diffusion of the knowledge about the potential of the vaccine, first to doctors, then to mothers, that made the difference. Nobody has found much of a cure for lung cancer, but in areas where public policy has pushed awareness of the link between smoking and cancer (and social activism has become more militant), smaller percentages of the population continue to smoke, and this has had a beneficial effect.

The same is roughly true regarding the benefits of mammography, checking of the colon and prostate and pap smears. It is pushing the knowledge to the people that has made all the difference. Human life expectancy has risen in the developed countries to between 75 and 84 years of age. One hundred years ago, it was between 42 and 48. Almost all of the difference seems to me to be due to increases in human knowledge, but completely conditional on this knowledge being published and affecting the behaviour of the general public.

Cancer Survival Rates

Again, a doubling of human knowledge should result in different outcomes in areas where that knowledge is applied. A lot of money is spent in improving our knowledge of certain cancers. If we double our knowledge, treatment of cancer should have a beneficial effect--improved survival rates being one effect we can measure.

A Google news search on cancer survival rates shows evidence of this. From Canada, the National Post reports that "the death rate for breast cancer for Canadian women has fallen 25% since 1986, when organized breast screening programs began being rolled out across the country." Apparently, this reduction is down to human knowledge from the early 1980's (beneficial effects of mammography) being dispersed through the wider population. Again, to me this counts. This is good news, great news. But again, if we keep a 5-year clock ticking, this news will disappoint, as it shows nothing remotely indicating a doubling of human knowledge within that timeframe.

Similarly, MedicineNet says, "Statistics (released in 1997) show that cancer patients are living longer and even "beating" the disease. Information released at an AMA sponsored conference for science writers, showed that the death rate from the dreaded disease has decreased by 3 percent in the last few years. In the  1940's only one patient in four survived on the average. By the 1960's, that figure was up to one in three, and now has reached 50% survival." So we see a doubling of the beneficial effect, but over 60 years, not five.

More encouragingly, St. Jude's Research Hospital shows 5-year survival rates for some cancers since 1962, and some of them might just qualify as 10-year doubling (pause to experiment on how to present this information... Okay, got it--click on the image to enlarge to readable size).

Again, a miracle, but it doesn't fit a 5-year window--it's 10 years for the best performing cancers.

There Were Hints And Allegations

Remember we are testing three hypotheses here:

1. The actual store of human knowledge (not, repeat not, data) has doubled in the past 5 years

2. Access to knowledge has grown so quickly that it fulfills the same function as a doubling of human knowledge

3. We are disproving previously held theories at such a rate that it frees up resources to concentrate on useful lines of enquiry resulting in an effective doubling of knowledge.

The easiest to find positive indications for is number 2. As an example, 5 years ago:

• Wikipedia had 80,000 articles and 125,000 hits per day in August 2002. On January 1, 2007, it had over 6 million articles (in 251 languages!) and in Dec. 06 it averaged more than 5 million daily unique visits. Okay, that's more than double... but this only works for hypothesis number 2, as we have no indication on how much of the knowledge placed on Wikipedia during that time period existed before 2002.