Does Human Knowledge Double Every 5 Years?

More on U.S. Federal Funding for Research

Okay, as promised--here are the growth rates and doubling periods for the subcategory disciplines for U.S. federal funding. They are taken from this spreadsheet: Download us_funding_for_research3.xls, which is found here on the NSF website, where it is labeled Table 1A.

There's a real shocker below (to me, anyways). I put the charts up twice--once using the straight figures given by the NSF, and again after adjusting for inflation.

If anyone wants to know why the U.S. research output has declined, look at the second one. In 14 of the 26 disciplines measured, inflation-adjusted growth is negative. Only two categories show growth that is not embarrassing--computer sciences and 'other medical.'

Elsewhere we have seen that the number of the student-age population has declined (although higher proportions are enrolling). We have heard that our proportion of the contribution to the increase of knowledge is bound to decline, given the growing contribution from other countries such as China.

It's all bull. The U.S.A. defunded research. Click to see the numbers. Read 'em and weep.

The Future of Blogging

Well, I'm going off topic again. I've been doing this for a little less than two months, and I have some observations about the process I'd like to share.

First, I think a weblog is the most appropriate forum for me to conduct this research. I will probably test the waters to see if I can get a book out of this, but the ability to hyperlink, the opportunity to write conversationally as I work, the tagging feature of categories (as opposed to the rigidity of chapters), all have made this an easier job for me--something I can do nights and weekends--rather than a herculean labour.

Second, after everybody gets tired of using weblogs to recount their previous night's drinkng exploits, I think that efforts such as this will become the best use of weblogs. I'm not doing anything here that you could not do, if you are reading this. I've put up my principal sources of information and the major tools I use for reference, and if you want to do your own work or fact-check mine-or even take it further, I'll be happy to help in any way I can.

Lastly, as a non-academic (but one who has been exposed to a lot of academic publishing), I recognise that the work I'm doing here is not sufficient to be authoritative. However, I'm trying to ensure that if this stimulates the conversation on the topic adequately for institutional investigation, any researcher will be able to start from a higher point than if I had not done this. And you know what? That's enough for me.

Funding Frustration

Well, the previous two posts give a fairly good picture of how the U.S. federal government funds research (note that we did not examine state and local funding, nor private sector funding of R&D, nor defense funding, for that matter).

I feel satisfied that I know what's happening in the U.S. But my real frustration is that you, dear reader (icy ironic tone, please) will quite reasonably expect me to find this out for the world.

But while it only took 7 hours to compile the information for the U.S. (due to better and more available statistics), it will take weeks to get a picture of what's happening in the world.

I think I'll put a tip jar on this blog.

U.S. Research Funding by Sector

I think the real problem with this post will be presentation. I have an Excel spreadsheet that shows U.S. federal funding for reseach by sector, and I want to show the growth rates and doubling times. However, the time series for some sectors differs, for a variety of reasons. So, please be patient with me.

I will put the figures in the extended post below, as well as some analysis, and report the topline findings here. In a later post I will do the same for subcategories. I think it will look messy. The purpose is to see what choices are being made, so we can compare funding differences to outputs--the numbers of academic publications and patents in each field.

Mathematics and computer sciences have seen the greatest growth, with a CAGR of 10.54% and a doubling time of 6.92 years. Strangely, physical sciences (including astronomy, chemistry and physics) fared second worst, with a CAGR of 5.14% and a doubling time of 13.83 years. However, it did start with the second highest base total, if that's any justification... Slowest growth in federal research funding was experienced in social sciences, with a CAGR of 4.82%, which gives a doubling time of 14.72 years.

Update: I had to recalculate all the stats to account for inflation, converting all $ figures to 2006 USD. It changes everything...

Into The Jungle Of Research Funding

Oh, this hurts. I did not want to go into research funding. But now that it is fairly clear that the rate of progress varies significantly by sector, we have to examine the inputs.

The first relevant document found is a PDF from the European Commission. It states that it wants the EU to increase research 'efforts' to 3% of GDP by 2010. Ambitious, that. The document talks about the importance of 'basic research,' as opposed to directed, I assume, which is encouraging. It actually serves as a good basic primer on how funding is structured in Europe, the U.S. and Japan, but while it talks about funding for specific programmes, it doesn't talk about levels of funding at the national or international level. So we move on.

Curious Cat, a blog that bills itself as a Science and Engineering blog, posts research funding for U.S. academic R&D for the past few years (see below in the extended post). More importantly there's a link to the NSF where we find an Excel spreadsheet (also in the extended post below) that charts all federal spending on research for science and engineering between 1970 and 2003. There are other charts available for viewing and downloading here.

In 1970, combined spending by all U.S. Federal agencies for research in science and engineering totaled $4.9 billion (USD). By 2003, that had grown to $53.3 billion (USD). Not bad.

The compound annual growth rate for U.S. Federal funding for research into science and engineering is 7.28%, which means it doubles every 9.86 years.

However, adjusting for inflation to 2006 U.S. dollars, it reads a bit differently. In 2006 dollars, the spend in 1970 was the equivalent of $26 billion, while the 2003 spend rises on recalculation to a total of £58 billion. That provides a CAGR of 2.39%, which would double every 29.35 years. A bit sad, that.

But the spreadsheet breaks spending down by discipline, as well. I'll address that in the next post.

League Tables and Update

Here is the league table for 25 points of measurement of the growth of human knowledge. As shown, only 10 of the 25 grow faster than 14.87% per year, which is what is needed to double every 5 years.

Field                                                            CAGR                   Doubling Period (Yrs)
1. Nanotechnology patents                                  44.91                           1.87
2. Nanotechnology journals                                 42.03                           1.98
3. Global warming patents                                  38.62                            2.12
4. Prions patents                                                33.76                            2.38
5. Programming patents                                      33.53                            2.4
6. Stem Cells patents                                           26.47                            2.95
7. Prions journals                                                 25.57                            3.04
8. Global warming journals                                    24.71                            3.14
9. Epidemiology patents                                        17.37                             4.33
10. Stem Cells journals                                          16.63                             4.51
5 Year Doubling Rate                                          14.87                             5
11. Programming journals                                     12.55                             5.86
12. Alzheimers Disease patents                              11.26                             6.5
13. Oncology patents                                              10.02                             7.26
14. Alzheimers Disease journals                                9.65                             7.52
15. Oncology journals                                                9.23                             7.85
16. DeSolla Price estimate of world literature growth   7.0                             10.24
17. Epidemiology journals                                           6.22                           11.49
18. Mars journals                                                        5.78                           12.34
19. Shale oil journals                                                   5.53                           12.88
20. US patent grants                                                    5.21                           13.65
21. University enrollment worldwide                              4.85                           14.64

22. Publications in astrophysics since 1970                     4.0                             17.67
23. US patent applications                                             3.88                           18.21
24. U.S. Book publishing                                                3.65                           19.33
25. Shale oil patents                                                      2.58                           27.21

More discussion below--click if you want to see it.

So, Does Human Knowledge Double Every 5 Years?

1. No,

2. But it could

3. Because we see that it does in sectors where we pour in human and financial resources.

However, the Western world is not pouring in human and financial resources the way it used to. Students are choosing law, business and web design instead of engineering and science. Government is encouraging private sector R&D (and encouraging universities to work closely with business), but it doesn't happen enough and when it does, it is focused on very short term proejcts that improve a bottom line in the next quarter.

Absent the commitment of extra resources, it looks like human knowledge doubles every 10-15 years. I'll be showing my work in the next post, and then supplementing it with information about the commitment of resources, an unwelcome next task.

But where there is a clear incentive, such as nanotechnology or global warming (where the incentive may be just getting a project funding by slapping a green label on it), we get the output that shows that our knowledge and exploitation (via patents) of a given sector can double every five years.

Continued below--click if you want to see a bit of a rant...

Stem Cells

Searching for "stem cells" or "stem cell" on Scirus returns 995,702 results, of which 152,648 are journal publications with the term in the title, abstract or contents, and 62,220 patents. According to Wikipedia, Stem cells were postulated in the early 1960s and shown to exist in 1963, giving us a 40-year window to chart the progress of human knowledge. However, looking at Scirus, I find articles as early as 1916 that seem relevant, and a patent in 1947 that also looks relevant to a layman.

Starting in 1960 excludes 74 journal articles and 2 patents, so we will make that our base year. That gives us a Compound Annual Growth Rate (CAGR) of 16.63%, or enough to double human knowledge about stem cells every five years. Hooray! We have another winner. That actually would double every 4.5 years. For patents, the CAGR is higher, 26.47%, which would double every 2.95 years. (Just for fun, if I used a base year of 1947, when the first patent was filed, the CAGR is still enough to double human knowledge every five years, although it drops to 20.2%).

Perhaps the most important finding is that President Bush's restrictions on stem-cell work in the U.S.A. has not dragged progress down, with over 100,000 journal publications and over 56,000 patents filed since he took office. Of course, I'm not counting how many of each are American, but that's for domestic policy wonks to debate... or even research, if they have the time.

As in some (not all) other sectors, we see that the quickest growth for academic publications is early, while patents explode just about the time that growth slows in academia. I'm not sure that's a coincidence...

Again, we see that when the incentives are properly allocated, we can double human knowledge in a sector every five years.

Click to see the figures below.

That Eureka Moment--University Students Worldwide

I have been searching for this for quite a while. UNESCO had it--click here to see it on their website. I include the graph here--click on the image to see a larger version.

  The number of tertiary education students has grown from 68 million in 1991 to 132 million in 2004. That's a doubling over a 14 year timespan. That's a compound annual growth (CAGR) rate of 4.85%, which would lead to a doubling every 14.64 years. It's a third of the growth rate we would need to double every five years.

This is daunting. We see the world university student population mirroring the growth rate of India since 1947, when what I think everyone assumes is a growth rate similar to China's.

I don't think this is adequate growth to sustain a required expansion of conventional infrastructure, let alone power a Singularity.

Alzheimers and Human Knowledge

Alzheimer's disease destroys human knowledge, but perhaps more importantly, it decoheres the personality that organises this knowledge. It is only human knowledge that can defeat this destroyer of human knowledge. It has been described,  treated (unsuccessfully) and researched for a century.

Let's see what Scirus tells us about the growth of human knowledge (and the commitment of research resources) on this. (BTW, I recently had another look at the sources Scirus uses, and it includes PubMed, so I won't be replicating this search there.) (And I got a quick scare by starting with the search term 'Alzheimers,' which only returned 1,428 results--if you're fact-checking me, and I hope to God someone is, use 'alzheimer.')

A search for Alzheimer on Scirus returns 1,274,572 results on Scirus, of which 112,401 are journal publications containing the term in the title, abstract or text. The search also returned 67,900 patents.

In the extended post, I will give the results by year and by five-year period. That's a Sunday morning for you--hot coffee and a database to search.

Okay--we'll assign 1920 as the base year for journals and infer 4 journal articles published that year, giving us a CAGR (Compound Annual Growth Rate) of 9.65%. This would indicate that human knowledge regarding Alzheimer's Disease doubles every 7.52 years.

However, the first patent was filed in 1976, which gives us a CAGR of 11.26%. This would double every 6.5 years. So, although work proceeds at a good rate and human knowledge is increasing, there is nothing like the growth rate seen in areas such as nanotechnology or global warming.

Again, we see that patent growth, starting at a later date, grows much more quickly than academic publishing. I am personally mystified as to why research into prions has grown twice as quickly as research into Alzheimers. Many more people suffer from Alzheimers than CJD, and the potential for both alleviating human suffering and reaping large rewards are much greater. The overall totals for Alzheimers research are greater than prions, don't get me wrong--but why is the growth trend favoring prions?