Does Human Knowledge Double Every 5 Years?

Real World Consequences of These Issues

Let's try to look at real world effects of this dangerous illusion we are discovering.

The illusion is that a rapid growth in human knowledge (doubling every five years) is happening all around us, naturally and painlessly. As we hope to have shown on this weblog, human knowledge about a particular subject can increase that fast or faster, but it is always as a result of putting money, institutional resources, interested and bright people to do the research, together. It doesn't hurt to have an urgent public need (HIV) or potential private gain (nanotechnology) as a carrot.

But where these inputs are not provided, human knowledge about a subject grows at a much slower pace. Let me explain what I mean.

Click below for more...

U.S. Research Funding by Sector

I think the real problem with this post will be presentation. I have an Excel spreadsheet that shows U.S. federal funding for reseach by sector, and I want to show the growth rates and doubling times. However, the time series for some sectors differs, for a variety of reasons. So, please be patient with me.

I will put the figures in the extended post below, as well as some analysis, and report the topline findings here. In a later post I will do the same for subcategories. I think it will look messy. The purpose is to see what choices are being made, so we can compare funding differences to outputs--the numbers of academic publications and patents in each field.

Mathematics and computer sciences have seen the greatest growth, with a CAGR of 10.54% and a doubling time of 6.92 years. Strangely, physical sciences (including astronomy, chemistry and physics) fared second worst, with a CAGR of 5.14% and a doubling time of 13.83 years. However, it did start with the second highest base total, if that's any justification... Slowest growth in federal research funding was experienced in social sciences, with a CAGR of 4.82%, which gives a doubling time of 14.72 years.

Update: I had to recalculate all the stats to account for inflation, converting all $ figures to 2006 USD. It changes everything...

Into The Jungle Of Research Funding

Oh, this hurts. I did not want to go into research funding. But now that it is fairly clear that the rate of progress varies significantly by sector, we have to examine the inputs.

The first relevant document found is a PDF from the European Commission. It states that it wants the EU to increase research 'efforts' to 3% of GDP by 2010. Ambitious, that. The document talks about the importance of 'basic research,' as opposed to directed, I assume, which is encouraging. It actually serves as a good basic primer on how funding is structured in Europe, the U.S. and Japan, but while it talks about funding for specific programmes, it doesn't talk about levels of funding at the national or international level. So we move on.

Curious Cat, a blog that bills itself as a Science and Engineering blog, posts research funding for U.S. academic R&D for the past few years (see below in the extended post). More importantly there's a link to the NSF where we find an Excel spreadsheet (also in the extended post below) that charts all federal spending on research for science and engineering between 1970 and 2003. There are other charts available for viewing and downloading here.

In 1970, combined spending by all U.S. Federal agencies for research in science and engineering totaled $4.9 billion (USD). By 2003, that had grown to $53.3 billion (USD). Not bad.

The compound annual growth rate for U.S. Federal funding for research into science and engineering is 7.28%, which means it doubles every 9.86 years.

However, adjusting for inflation to 2006 U.S. dollars, it reads a bit differently. In 2006 dollars, the spend in 1970 was the equivalent of $26 billion, while the 2003 spend rises on recalculation to a total of £58 billion. That provides a CAGR of 2.39%, which would double every 29.35 years. A bit sad, that.

But the spreadsheet breaks spending down by discipline, as well. I'll address that in the next post.

League Tables and Update

Here is the league table for 25 points of measurement of the growth of human knowledge. As shown, only 10 of the 25 grow faster than 14.87% per year, which is what is needed to double every 5 years.

Field                                                            CAGR                   Doubling Period (Yrs)
1. Nanotechnology patents                                  44.91                           1.87
2. Nanotechnology journals                                 42.03                           1.98
3. Global warming patents                                  38.62                            2.12
4. Prions patents                                                33.76                            2.38
5. Programming patents                                      33.53                            2.4
6. Stem Cells patents                                           26.47                            2.95
7. Prions journals                                                 25.57                            3.04
8. Global warming journals                                    24.71                            3.14
9. Epidemiology patents                                        17.37                             4.33
10. Stem Cells journals                                          16.63                             4.51
5 Year Doubling Rate                                          14.87                             5
11. Programming journals                                     12.55                             5.86
12. Alzheimers Disease patents                              11.26                             6.5
13. Oncology patents                                              10.02                             7.26
14. Alzheimers Disease journals                                9.65                             7.52
15. Oncology journals                                                9.23                             7.85
16. DeSolla Price estimate of world literature growth   7.0                             10.24
17. Epidemiology journals                                           6.22                           11.49
18. Mars journals                                                        5.78                           12.34
19. Shale oil journals                                                   5.53                           12.88
20. US patent grants                                                    5.21                           13.65
21. University enrollment worldwide                              4.85                           14.64

22. Publications in astrophysics since 1970                     4.0                             17.67
23. US patent applications                                             3.88                           18.21
24. U.S. Book publishing                                                3.65                           19.33
25. Shale oil patents                                                      2.58                           27.21

More discussion below--click if you want to see it.

So, Does Human Knowledge Double Every 5 Years?

1. No,

2. But it could

3. Because we see that it does in sectors where we pour in human and financial resources.

However, the Western world is not pouring in human and financial resources the way it used to. Students are choosing law, business and web design instead of engineering and science. Government is encouraging private sector R&D (and encouraging universities to work closely with business), but it doesn't happen enough and when it does, it is focused on very short term proejcts that improve a bottom line in the next quarter.

Absent the commitment of extra resources, it looks like human knowledge doubles every 10-15 years. I'll be showing my work in the next post, and then supplementing it with information about the commitment of resources, an unwelcome next task.

But where there is a clear incentive, such as nanotechnology or global warming (where the incentive may be just getting a project funding by slapping a green label on it), we get the output that shows that our knowledge and exploitation (via patents) of a given sector can double every five years.

Continued below--click if you want to see a bit of a rant...

Stem Cells

Searching for "stem cells" or "stem cell" on Scirus returns 995,702 results, of which 152,648 are journal publications with the term in the title, abstract or contents, and 62,220 patents. According to Wikipedia, Stem cells were postulated in the early 1960s and shown to exist in 1963, giving us a 40-year window to chart the progress of human knowledge. However, looking at Scirus, I find articles as early as 1916 that seem relevant, and a patent in 1947 that also looks relevant to a layman.

Starting in 1960 excludes 74 journal articles and 2 patents, so we will make that our base year. That gives us a Compound Annual Growth Rate (CAGR) of 16.63%, or enough to double human knowledge about stem cells every five years. Hooray! We have another winner. That actually would double every 4.5 years. For patents, the CAGR is higher, 26.47%, which would double every 2.95 years. (Just for fun, if I used a base year of 1947, when the first patent was filed, the CAGR is still enough to double human knowledge every five years, although it drops to 20.2%).

Perhaps the most important finding is that President Bush's restrictions on stem-cell work in the U.S.A. has not dragged progress down, with over 100,000 journal publications and over 56,000 patents filed since he took office. Of course, I'm not counting how many of each are American, but that's for domestic policy wonks to debate... or even research, if they have the time.

As in some (not all) other sectors, we see that the quickest growth for academic publications is early, while patents explode just about the time that growth slows in academia. I'm not sure that's a coincidence...

Again, we see that when the incentives are properly allocated, we can double human knowledge in a sector every five years.

Click to see the figures below.

Alzheimers and Human Knowledge

Alzheimer's disease destroys human knowledge, but perhaps more importantly, it decoheres the personality that organises this knowledge. It is only human knowledge that can defeat this destroyer of human knowledge. It has been described,  treated (unsuccessfully) and researched for a century.

Let's see what Scirus tells us about the growth of human knowledge (and the commitment of research resources) on this. (BTW, I recently had another look at the sources Scirus uses, and it includes PubMed, so I won't be replicating this search there.) (And I got a quick scare by starting with the search term 'Alzheimers,' which only returned 1,428 results--if you're fact-checking me, and I hope to God someone is, use 'alzheimer.')

A search for Alzheimer on Scirus returns 1,274,572 results on Scirus, of which 112,401 are journal publications containing the term in the title, abstract or text. The search also returned 67,900 patents.

In the extended post, I will give the results by year and by five-year period. That's a Sunday morning for you--hot coffee and a database to search.

Okay--we'll assign 1920 as the base year for journals and infer 4 journal articles published that year, giving us a CAGR (Compound Annual Growth Rate) of 9.65%. This would indicate that human knowledge regarding Alzheimer's Disease doubles every 7.52 years.

However, the first patent was filed in 1976, which gives us a CAGR of 11.26%. This would double every 6.5 years. So, although work proceeds at a good rate and human knowledge is increasing, there is nothing like the growth rate seen in areas such as nanotechnology or global warming.

Again, we see that patent growth, starting at a later date, grows much more quickly than academic publishing. I am personally mystified as to why research into prions has grown twice as quickly as research into Alzheimers. Many more people suffer from Alzheimers than CJD, and the potential for both alleviating human suffering and reaping large rewards are much greater. The overall totals for Alzheimers research are greater than prions, don't get me wrong--but why is the growth trend favoring prions?

Prions

It's back to Scirus to see trends in publications and patents regarding prions.

Prions are infectious agents composed only of proteins, and are implicated in BSE (bovine spongiform encephalopathy, AKA mad cow disease) and Creutzfeldt-Jakob Disease (CJD) which as far as I can tell is more or less a human equivalent.

Prions were hypothesized in the 60s and actually found in the 80s, winning a Nobel prize for the disoverer, Stanley Prusiner.

The human disease is quite rare. Mad cow disease is not. Both are fatal. There is a public good to be pursued in preventing CJD and private profit in preventing mad cow disease. The two together should stimulate research. With a short timeline to look at, it will be interesting to see if human knowledge doubles every five years.

Scirus returns 13,518 journal publications with 'prion' in the title, abstract or text, and 8,781 patents. Because there is a bird called the fairy prion (Pachyptila turtur), I will actually have to look at the results... darn.

We hereby proclaim a base year of 1981, when 5 journal articles and no patents were published. That gives us a CAGR (compound annual growth rate) of 25.57% and we have another winner! Human knowledge has doubled every 5 years regarding prions! And patents have grown even more rapidly, at 33.76%--huzzah!

However, as we have seen elsewhere, a note of caution--while patent growth has consistently exhibited hypergrowth characteristics, journal publications have leveled off, with the period 2002-2006 growing 'only' 66% over the preceding five-year period.

Click below to see the results.

Programming, Software and Computer Science

Computer programming has a lot of work and a lot of people centered around it. If the Singularitarians are correct, we're going to need them all.

After all, what good is it if computing speeds and memory keep doubling if we cannot program computers to do what we want them to do? And the Singularitarians want computers to do some pretty ambitious things. In fact, given their wish list, it seems clear that human knowledge of computer programming has to double every five years if we are going to get the magic box to do what we want it to do. We will need new programming languages, tighter coding, better algorithms, and quite possibly a new way of looking at programming. Is this happening?

We've seen elsewhere that the number of people engaged in computer sciences has grown to 180,000 new entrants per year in the U.S. alone--and considering the base date for this field, that is evidence of healthy growth. Let's use Scirus to see what this growth in professionals has produced in the way of patents and journal articles.

Preliminary results: between 2002 and 2006, 38,685 journal articles were published with the word 'programming' in the title or abstract. As the total is 145,404, only 26.6% of all journal articles were published in the last five years. Of the 367,971 patents noted on Scirus with 'programming' in the title or description, 201,548, or 54.7% were published between 2002 and 2006--so that has doubled. Not the first time we have seen differing growth rates for journals and patents.

For 'software,' the period 2002-2006 saw 224,696 journal articles published, 46% of the total of 486,262. For patents, as with 'programming,' the period was more robust--589,919 out of a total of 900,846, or 65% of the total.

The base year for 'programming' is 1940, when a UK patent was filed for improvements on a statistical calculating device. The compound annual growth rate for journal publications is 12.55%--good, but not good enough to double every five years (it is less than six years, to be fair). The compound annual growth rate for patents is 12.83%.

Remember that we need a CAGR (Compound Annual Growth Rate) of 14.87% to double every five years.

For 'software, the base year is 1965, when 5 journal articles were published and 1 patent was filed. And software is a winner, with CAGR of 22.83% for journal publications!  Hooray for coders everywhere! And a double huzzah to IBM, Microsoft, Sun and countless others, as patent growth since 1965 has a CAGR of 33.53%. Another winner!

As there are so many more patents than journal publications for both search terms, we should probably give more weight to patents, and be extremely pleased that human knowledge appears to have doubled in the past five years in this area, especially given the gloomy results reported in my previous post last week.

Click below to see results and analysis.

Outcomes

Advances in human knowledge should make life better. That's hard to measure. However, we can measure if human life gets longer (it does), healthier, richer (if not more rewarding), etc.

The rest of this post is philosophizing--feel free to skip. And in a previous post, I talk about the Outcomes Fallacy. But if you want thinking on the issue of human knowledge and outcomes, click to read more.