Does Human Knowledge Double Every 5 Years?

Identity Management and Human Knowledge

Identity Management (IDM) is a discipline that tries to supervise how passwords are protected and verified, how IT departments decide what files you have access to, etc.

It's a new field--they're still debating the name, they haven't really defined the limits, and they don't know where the concept should sit inside organisations. Perfect for what we're doing here. We've looked at very long time series, for things such as Mars and Alzheimer's Disease and battlefield injuries. We've looked at timelines for fairly new fields, such as nanotechnology. Here is a field that is just being born.

The first academic work starts to appear in 2000, according to Scirus, and there are only 612 academic publications with the phrase "identity management" in the title, abstract or text overall. Truly nascent.

There have been 1,172 patents filed with the phrase "identity management" in the title, abstract or text overall.

In the extended post below, I show the figures and explain a bit more about how I calculate the compound annual growth rate, and show an alternative means of calculating it.

Using the method I've employed throughout this weblog, the growth rate for academic publications is 34.93%, which would double every 2.31 years, and the growth rate for patent applications is 32.93%, which would double every 2.44 years. and I would then write 'Hooray! We have another winner!'

But the alternative means of calculating growth would look even better, showing growth in academic publications of 71.54% and a doubling time of 1.28 years, and for patents, CAGR of 71.37% and a doubling time of 1.29 years.

I'm going to stick with my method, but if you want to recalculate all of the figures, the tools to do so are here. Click to see the numbers and the explanation.

Two New Sectors To Measure

The project that called me away from this explored the growth of two new 'disciplines'--identity management and information assurance. Both are attempts to put strategic management protocols around all the work that is going on in subsectors relating to passwords, permissions, hand-offs of responsibilities between websites and organisations, etc. They're both new, and they both deal with subsectors that they want to 'control' (i.e., have oversight responsibility for). So I've been exposed to information on the growth of both of them and other sectors as well, such as biometrics.

The next three posts will look at knowledge growth in these sectors. Since two of them were essentially invented after 9/11, the time series will be blessedly short.

Human Knowledge Doubles

The question is how quickly. Here's yet another summary of what is in this weblog. Since April, we have shown that primary evidence of human knowledge, as shown in patents and academic publication, doubles at different rates for different sectors, ranging from 2 years for nanotechnology to 21 years for other sectors.

Secondary evidence, such as the number of university students and funding for research, varies by country. The OECD has reduced government funding for research (and the growth in patents and students in scientific disciplines) has dropped to zero. China and India, among others, are funding research and encouraging students to enter scientific fields, and they are seeing rapid gowth in publications, patents, research funding and students.

But the growth of Asian knowledge engines has not yet replaced the flattening of the growth curve in the developed world. This has led to uneven growth in knowledge--in sexy sectors like nanotechnology, growth is very rapid. In other areas, such as Alzheimer's Disease, it is not as rapid as one would think.

Real World Consequences of These Issues

Let's try to look at real world effects of this dangerous illusion we are discovering.

The illusion is that a rapid growth in human knowledge (doubling every five years) is happening all around us, naturally and painlessly. As we hope to have shown on this weblog, human knowledge about a particular subject can increase that fast or faster, but it is always as a result of putting money, institutional resources, interested and bright people to do the research, together. It doesn't hurt to have an urgent public need (HIV) or potential private gain (nanotechnology) as a carrot.

But where these inputs are not provided, human knowledge about a subject grows at a much slower pace. Let me explain what I mean.

Click below for more...

Funding Frustration

Well, the previous two posts give a fairly good picture of how the U.S. federal government funds research (note that we did not examine state and local funding, nor private sector funding of R&D, nor defense funding, for that matter).

I feel satisfied that I know what's happening in the U.S. But my real frustration is that you, dear reader (icy ironic tone, please) will quite reasonably expect me to find this out for the world.

But while it only took 7 hours to compile the information for the U.S. (due to better and more available statistics), it will take weeks to get a picture of what's happening in the world.

I think I'll put a tip jar on this blog.

U.S. Research Funding by Sector

I think the real problem with this post will be presentation. I have an Excel spreadsheet that shows U.S. federal funding for reseach by sector, and I want to show the growth rates and doubling times. However, the time series for some sectors differs, for a variety of reasons. So, please be patient with me.

I will put the figures in the extended post below, as well as some analysis, and report the topline findings here. In a later post I will do the same for subcategories. I think it will look messy. The purpose is to see what choices are being made, so we can compare funding differences to outputs--the numbers of academic publications and patents in each field.

Mathematics and computer sciences have seen the greatest growth, with a CAGR of 10.54% and a doubling time of 6.92 years. Strangely, physical sciences (including astronomy, chemistry and physics) fared second worst, with a CAGR of 5.14% and a doubling time of 13.83 years. However, it did start with the second highest base total, if that's any justification... Slowest growth in federal research funding was experienced in social sciences, with a CAGR of 4.82%, which gives a doubling time of 14.72 years.

Update: I had to recalculate all the stats to account for inflation, converting all $ figures to 2006 USD. It changes everything...

Into The Jungle Of Research Funding

Oh, this hurts. I did not want to go into research funding. But now that it is fairly clear that the rate of progress varies significantly by sector, we have to examine the inputs.

The first relevant document found is a PDF from the European Commission. It states that it wants the EU to increase research 'efforts' to 3% of GDP by 2010. Ambitious, that. The document talks about the importance of 'basic research,' as opposed to directed, I assume, which is encouraging. It actually serves as a good basic primer on how funding is structured in Europe, the U.S. and Japan, but while it talks about funding for specific programmes, it doesn't talk about levels of funding at the national or international level. So we move on.

Curious Cat, a blog that bills itself as a Science and Engineering blog, posts research funding for U.S. academic R&D for the past few years (see below in the extended post). More importantly there's a link to the NSF where we find an Excel spreadsheet (also in the extended post below) that charts all federal spending on research for science and engineering between 1970 and 2003. There are other charts available for viewing and downloading here.

In 1970, combined spending by all U.S. Federal agencies for research in science and engineering totaled $4.9 billion (USD). By 2003, that had grown to $53.3 billion (USD). Not bad.

The compound annual growth rate for U.S. Federal funding for research into science and engineering is 7.28%, which means it doubles every 9.86 years.

However, adjusting for inflation to 2006 U.S. dollars, it reads a bit differently. In 2006 dollars, the spend in 1970 was the equivalent of $26 billion, while the 2003 spend rises on recalculation to a total of £58 billion. That provides a CAGR of 2.39%, which would double every 29.35 years. A bit sad, that.

But the spreadsheet breaks spending down by discipline, as well. I'll address that in the next post.

League Tables and Update

Here is the league table for 25 points of measurement of the growth of human knowledge. As shown, only 10 of the 25 grow faster than 14.87% per year, which is what is needed to double every 5 years.

Field                                                            CAGR                   Doubling Period (Yrs)
1. Nanotechnology patents                                  44.91                           1.87
2. Nanotechnology journals                                 42.03                           1.98
3. Global warming patents                                  38.62                            2.12
4. Prions patents                                                33.76                            2.38
5. Programming patents                                      33.53                            2.4
6. Stem Cells patents                                           26.47                            2.95
7. Prions journals                                                 25.57                            3.04
8. Global warming journals                                    24.71                            3.14
9. Epidemiology patents                                        17.37                             4.33
10. Stem Cells journals                                          16.63                             4.51
5 Year Doubling Rate                                          14.87                             5
11. Programming journals                                     12.55                             5.86
12. Alzheimers Disease patents                              11.26                             6.5
13. Oncology patents                                              10.02                             7.26
14. Alzheimers Disease journals                                9.65                             7.52
15. Oncology journals                                                9.23                             7.85
16. DeSolla Price estimate of world literature growth   7.0                             10.24
17. Epidemiology journals                                           6.22                           11.49
18. Mars journals                                                        5.78                           12.34
19. Shale oil journals                                                   5.53                           12.88
20. US patent grants                                                    5.21                           13.65
21. University enrollment worldwide                              4.85                           14.64

22. Publications in astrophysics since 1970                     4.0                             17.67
23. US patent applications                                             3.88                           18.21
24. U.S. Book publishing                                                3.65                           19.33
25. Shale oil patents                                                      2.58                           27.21

More discussion below--click if you want to see it.

So, Does Human Knowledge Double Every 5 Years?

1. No,

2. But it could

3. Because we see that it does in sectors where we pour in human and financial resources.

However, the Western world is not pouring in human and financial resources the way it used to. Students are choosing law, business and web design instead of engineering and science. Government is encouraging private sector R&D (and encouraging universities to work closely with business), but it doesn't happen enough and when it does, it is focused on very short term proejcts that improve a bottom line in the next quarter.

Absent the commitment of extra resources, it looks like human knowledge doubles every 10-15 years. I'll be showing my work in the next post, and then supplementing it with information about the commitment of resources, an unwelcome next task.

But where there is a clear incentive, such as nanotechnology or global warming (where the incentive may be just getting a project funding by slapping a green label on it), we get the output that shows that our knowledge and exploitation (via patents) of a given sector can double every five years.

Continued below--click if you want to see a bit of a rant...

Stem Cells

Searching for "stem cells" or "stem cell" on Scirus returns 995,702 results, of which 152,648 are journal publications with the term in the title, abstract or contents, and 62,220 patents. According to Wikipedia, Stem cells were postulated in the early 1960s and shown to exist in 1963, giving us a 40-year window to chart the progress of human knowledge. However, looking at Scirus, I find articles as early as 1916 that seem relevant, and a patent in 1947 that also looks relevant to a layman.

Starting in 1960 excludes 74 journal articles and 2 patents, so we will make that our base year. That gives us a Compound Annual Growth Rate (CAGR) of 16.63%, or enough to double human knowledge about stem cells every five years. Hooray! We have another winner. That actually would double every 4.5 years. For patents, the CAGR is higher, 26.47%, which would double every 2.95 years. (Just for fun, if I used a base year of 1947, when the first patent was filed, the CAGR is still enough to double human knowledge every five years, although it drops to 20.2%).

Perhaps the most important finding is that President Bush's restrictions on stem-cell work in the U.S.A. has not dragged progress down, with over 100,000 journal publications and over 56,000 patents filed since he took office. Of course, I'm not counting how many of each are American, but that's for domestic policy wonks to debate... or even research, if they have the time.

As in some (not all) other sectors, we see that the quickest growth for academic publications is early, while patents explode just about the time that growth slows in academia. I'm not sure that's a coincidence...

Again, we see that when the incentives are properly allocated, we can double human knowledge in a sector every five years.

Click to see the figures below.