Does Human Knowledge Double Every 5 Years?

Well, I had a plan...

I have been hijacked away from this project for two months, working on a very interesting project called Blindside, which had the benefit of having a paycheck attached to it. It also is tangentially related to what I am doing here, as my next few posts will show.

No reason why you should care about any of the above (but do follow the link if you're interested in how technology relates to UK government, information assurance issues and identity management), but it has caused me to rethink my scheduling for this. I'm going to do more primary research--maybe through the summer--and then tie it up with a couple of really brilliant think pieces and try to sell it as a book.

I'd like to welcome the flood (well, trickle) of readers who have actually found this weblog--I assume it's academic assignments that lead you here. I hope you feel free to comment or ask questions.

Anyway, back to work.

The Future of Blogging

Well, I'm going off topic again. I've been doing this for a little less than two months, and I have some observations about the process I'd like to share.

First, I think a weblog is the most appropriate forum for me to conduct this research. I will probably test the waters to see if I can get a book out of this, but the ability to hyperlink, the opportunity to write conversationally as I work, the tagging feature of categories (as opposed to the rigidity of chapters), all have made this an easier job for me--something I can do nights and weekends--rather than a herculean labour.

Second, after everybody gets tired of using weblogs to recount their previous night's drinkng exploits, I think that efforts such as this will become the best use of weblogs. I'm not doing anything here that you could not do, if you are reading this. I've put up my principal sources of information and the major tools I use for reference, and if you want to do your own work or fact-check mine-or even take it further, I'll be happy to help in any way I can.

Lastly, as a non-academic (but one who has been exposed to a lot of academic publishing), I recognise that the work I'm doing here is not sufficient to be authoritative. However, I'm trying to ensure that if this stimulates the conversation on the topic adequately for institutional investigation, any researcher will be able to start from a higher point than if I had not done this. And you know what? That's enough for me.

Into The Jungle Of Research Funding

Oh, this hurts. I did not want to go into research funding. But now that it is fairly clear that the rate of progress varies significantly by sector, we have to examine the inputs.

The first relevant document found is a PDF from the European Commission. It states that it wants the EU to increase research 'efforts' to 3% of GDP by 2010. Ambitious, that. The document talks about the importance of 'basic research,' as opposed to directed, I assume, which is encouraging. It actually serves as a good basic primer on how funding is structured in Europe, the U.S. and Japan, but while it talks about funding for specific programmes, it doesn't talk about levels of funding at the national or international level. So we move on.

Curious Cat, a blog that bills itself as a Science and Engineering blog, posts research funding for U.S. academic R&D for the past few years (see below in the extended post). More importantly there's a link to the NSF where we find an Excel spreadsheet (also in the extended post below) that charts all federal spending on research for science and engineering between 1970 and 2003. There are other charts available for viewing and downloading here.

In 1970, combined spending by all U.S. Federal agencies for research in science and engineering totaled $4.9 billion (USD). By 2003, that had grown to $53.3 billion (USD). Not bad.

The compound annual growth rate for U.S. Federal funding for research into science and engineering is 7.28%, which means it doubles every 9.86 years.

However, adjusting for inflation to 2006 U.S. dollars, it reads a bit differently. In 2006 dollars, the spend in 1970 was the equivalent of $26 billion, while the 2003 spend rises on recalculation to a total of £58 billion. That provides a CAGR of 2.39%, which would double every 29.35 years. A bit sad, that.

But the spreadsheet breaks spending down by discipline, as well. I'll address that in the next post.

So, Does Human Knowledge Double Every 5 Years?

1. No,

2. But it could

3. Because we see that it does in sectors where we pour in human and financial resources.

However, the Western world is not pouring in human and financial resources the way it used to. Students are choosing law, business and web design instead of engineering and science. Government is encouraging private sector R&D (and encouraging universities to work closely with business), but it doesn't happen enough and when it does, it is focused on very short term proejcts that improve a bottom line in the next quarter.

Absent the commitment of extra resources, it looks like human knowledge doubles every 10-15 years. I'll be showing my work in the next post, and then supplementing it with information about the commitment of resources, an unwelcome next task.

But where there is a clear incentive, such as nanotechnology or global warming (where the incentive may be just getting a project funding by slapping a green label on it), we get the output that shows that our knowledge and exploitation (via patents) of a given sector can double every five years.

Continued below--click if you want to see a bit of a rant...

A Smart Science Fiction Writer Looks At The Future--And The Singularity

Charlie Stross looks at improvements in the speed of travel, information technology and bandwidth, says some very interesting things about convergence and the near future.

One idea just charms me--that our children will never have to worry about being lost, any more than we had to worry about being eaten by wolves. I suppose in about ten minutes I'll start thinking about the Big Brother implications, but these ten minutes are nice.

Hat tip Brad DeLong.

Outcomes

Advances in human knowledge should make life better. That's hard to measure. However, we can measure if human life gets longer (it does), healthier, richer (if not more rewarding), etc.

The rest of this post is philosophizing--feel free to skip. And in a previous post, I talk about the Outcomes Fallacy. But if you want thinking on the issue of human knowledge and outcomes, click to read more.

Sobering News

The post below from yesterday, which details the stately, rather than exponential, increase in granted patents in the U.S., has really got me thinking, and I think we'd better report on the state of play here.

Let's detail those areas where we have seen adequate growth to double human knowledge every 5 years:

Update: We have since found other areas where the CAGR is sufficient to double human knowledge every five years:

Update 2: A modest review revealed errors in my calculations regarding oncology. Its position has changed (and not for the better).

Prions, where academic journal publications have increased at a CAGR of 25.57%, and patents have increased by 33.76%

Software, where academic journal publications have increased at a CAGR of 22.83% and patents have increased by 33.53%

Post-graduate enrollment in China, which has increased at a CAGR of 22.38% (although see below for more discouraging information about the growth of tertiary education worldwide)

(All percentages are CAGR (compound annual growth rate.)

1. Nanotechnology, with a blistering growth in publications of 97.16% per year. Patents, 76.15%.

2. Global Warming publications, with a of 16.85%

3. Total Chinese patent applications, at 25% per annum

4. Chinese computing patents, 58.29%

5. Chinese telecommunications patents, 30.26%

(was oncology)

7. Epidemiology patents, 22.76%

So far, that's it.

Now let's look at where we did not find anything like doubling every five years (remembering that we need a CAGR of 14.87% to double in five years time):

Update: We have now found other areas where human knowledge does not appear to be growing at a rate that will double in five years time:

Worldwide university enrollment, which has a CAGR of 4.85%

Alzheimer's disease, where academic publications have a CAGR of 9.65% and patents a CAGR of 11.26%

Programming, where academic publications have a CAGR of 12.55% and patents a CAGR of 12.83%

Chinese university enrollment, with a CAGR of 9.76%

1. U.S. patents granted, 5.21%

2. Epidemiology publications, 8.85%

3. Shale oil extraction, publications 5.59%, patents, 2.61%

4. Publications on Mars, 5.78% (see also here and here)

5. U.S. Book publishing, 5.14%

6. Indian university enrolment since 1947, 5.51%

7. Climate Models, 12.66% (Very close to 5-year doubling rate of 14.87%)

8. Improvements in Climate Models, 13.76% (Also very close)

9. Growth in publication of scientific and technical papers, 1995-1999

             OECD 0.1%

             U.S., -1.8%

             Canada, -2.2%

             UK, -0.1%

             Netherlands, -0.5%

10. Publications in astrophysics, 4% (since 1970)

11. Growth in percentage of cancer patients surviving 5 years after diagnosis, 0.7% (1960-2005)

12. Drop in percentage of soldiers dying from wounds received in battle, 1.54% (1941-2006)

13. Projected growth in number of scientists (U.S.), next ten years, 7.18%

14. Patent filings in India, last five years, 8.45%

15. Increase in patent filings, World International Patent Office, 1 year, 6.4%

6. Oncology pubications, 9.23% (revised)

Considering I cherry picked areas that I thought would be most likely to show robust growth, this is scary.

          

Relevant Research from 1988

It would have been very strange if I were the only one asking this question and doing some research. I'm glad to find some prior work on this, not least because it means I'm not the only one who cares.

A report on the International Conference on Fifth Generation Computer Systems in 1988, written by Jakob Nielson, refers to research by Jorg Siekmann of the University of Kaiserslauten in the then West Germany.

Nielson writes, "Siekmann reported that the number of scientific journals doubles every 15 years, the number of books in university libraries doubles every 10 years, and the number of scientific publications doubles every 5 years. The knowledge explosion has already reached a level where in chemistry it is often cheaper to conduct a possible duplication of an experiment than to search the literature to find whether the result of a previous experiment can be used." That last bit is scary--I wonder if it's still true. Let's see if we can find more about Siekmann and his research.

Google likes him--a little too much. There are 108,000 search returns on his name.

The University has some material about him: Jörg H. Siekmann is the head of several research groups working in artificial intelligence at the "Universität des Saarlandes" and at the German Research Centre for Artificial Intelligence (DFKI) in Saarbrücken. He is a professor in the department of computer science and a director at the DFKI. The five research groups are:

Automated Reasoning for Mathematics: OMEGA 
Competence Center e-Learning
Formal Methods: VSE
Multi-Agent Systems
The IT-Security Evaluation Centre

Currently he is the chairman of the collaborative research centre "Resource-adaptive cognitive Processes" (SFB 378) funded by the German Science Foundation (DFG). He is chairman of the network of excellence on computational logic (CoLogNet), and vice president of the International Federation on Computational Logic (IFCoLog) . His main research interests are Artificial Intelligence, Automated Reasoning, Multiagent Systems and e-Learning in Mathematics.

But I can't find any more information about his statement at the conference. However, I did stumble across something quite interesting that might lead to the origin of the statement  Human Knowledge Doubles Every Five Years. See next post (above).

How Important Is All This?

Now that I've been doing this for a month, the question of how important this all is is becoming more relevant. When you're in the middle of the hard slog of gathering data, it's normal to get frustrated. It's also normal to get a little grandiose in your thinking about the worth of what you're doing, partly to motivate yourself to keep going.

So how important is all this? If I come up with the finding that human knowledge doubles every ten years instead of every five, who will care? What will it change?

I mentioned a few things in an earlier post that I am starting to think are actually relevant questions for society. If we are convinced that human knowledge doubles every five years, then we don't need to quit smoking, as cancer will be cured before it strikes us. Similarly, we don't need to worry about pollution, because we'll sort it out before the effects can truly harm society. You can adopt this lazy attitude about any of the problems our world confronts.

But it would be a mortal pity if human knowledge doubles every six or ten or twelve years and you die of cancer the year before the cure is discovered instead of the year after.

The idea that human knowledge doubles every five years is widespread. A Google search using the term human knowledge doubles every five years returns 1,680,000 results (obviously only a fraction are relevant).

If that information is wrong, and individuals are making important decisions and organisations are misallocating resources in this mistaken belief, then yes, this research is important and worth doing. Gee, I feel reassured.

One Month Progress Report

Well, this weblog has now been up for 30 days. I had actually intended to do all the basic reseach in this time frame, but life has somehow interfered. A major reorganisation at work has slowed me down.

Nonetheless, we've managed to explore the growth in human knowledge in a number of areas, so let's recapitulate:

1. We do find a doubling of the evidence of human knowledge (journal publications and patents) in some sectors, such as nanotechnology and global warming, new fields that double from a low base. I'm tempted to label this the Manhattan Project effect, but I'm not going to do that until I study that sector since 1945.
2. We do not see a doubling in areas with a longer time series, such as astrophysics or the study of Mars, although there are periods of impressive growth in each.
3. In some areas, we see different trends for patents as opposed to journal publications. For example, in epidemiology we see an 8.85% CAGR in journal publications in the recent past, while patents have grown by 22.76% annually since 1987.
4. We do see evidence that human knowledge is available to twice as many people, and that has grown at a rate that more than doubles every five years. (Just looking at the growth of Wikipedia and the numbers of people connected to the Internet makes that fairly obvious).
5. We do not see evidence that the number of scientists and researchers doubles every five years, and it is not at all obvious that emerging country growth is making up for a slowdown or actual decline in OECD numbers.

But there's a lot more work to do, and any of those five points could change. I'm having fun doing this, so we'll just carry on.