Does Human Knowledge Double Every 5 Years?

Human Knowledge Doubles

The question is how quickly. Here's yet another summary of what is in this weblog. Since April, we have shown that primary evidence of human knowledge, as shown in patents and academic publication, doubles at different rates for different sectors, ranging from 2 years for nanotechnology to 21 years for other sectors.

Secondary evidence, such as the number of university students and funding for research, varies by country. The OECD has reduced government funding for research (and the growth in patents and students in scientific disciplines) has dropped to zero. China and India, among others, are funding research and encouraging students to enter scientific fields, and they are seeing rapid gowth in publications, patents, research funding and students.

But the growth of Asian knowledge engines has not yet replaced the flattening of the growth curve in the developed world. This has led to uneven growth in knowledge--in sexy sectors like nanotechnology, growth is very rapid. In other areas, such as Alzheimer's Disease, it is not as rapid as one would think.

Identity Management and Human Knowledge

Identity Management (IDM) is a discipline that tries to supervise how passwords are protected and verified, how IT departments decide what files you have access to, etc.

It's a new field--they're still debating the name, they haven't really defined the limits, and they don't know where the concept should sit inside organisations. Perfect for what we're doing here. We've looked at very long time series, for things such as Mars and Alzheimer's Disease and battlefield injuries. We've looked at timelines for fairly new fields, such as nanotechnology. Here is a field that is just being born.

The first academic work starts to appear in 2000, according to Scirus, and there are only 612 academic publications with the phrase "identity management" in the title, abstract or text overall. Truly nascent.

There have been 1,172 patents filed with the phrase "identity management" in the title, abstract or text overall.

In the extended post below, I show the figures and explain a bit more about how I calculate the compound annual growth rate, and show an alternative means of calculating it.

Using the method I've employed throughout this weblog, the growth rate for academic publications is 34.93%, which would double every 2.31 years, and the growth rate for patent applications is 32.93%, which would double every 2.44 years. and I would then write 'Hooray! We have another winner!'

But the alternative means of calculating growth would look even better, showing growth in academic publications of 71.54% and a doubling time of 1.28 years, and for patents, CAGR of 71.37% and a doubling time of 1.29 years.

I'm going to stick with my method, but if you want to recalculate all of the figures, the tools to do so are here. Click to see the numbers and the explanation.

Two New Sectors To Measure

The project that called me away from this explored the growth of two new 'disciplines'--identity management and information assurance. Both are attempts to put strategic management protocols around all the work that is going on in subsectors relating to passwords, permissions, hand-offs of responsibilities between websites and organisations, etc. They're both new, and they both deal with subsectors that they want to 'control' (i.e., have oversight responsibility for). So I've been exposed to information on the growth of both of them and other sectors as well, such as biometrics.

The next three posts will look at knowledge growth in these sectors. Since two of them were essentially invented after 9/11, the time series will be blessedly short.

Well, I had a plan...

I have been hijacked away from this project for two months, working on a very interesting project called Blindside, which had the benefit of having a paycheck attached to it. It also is tangentially related to what I am doing here, as my next few posts will show.

No reason why you should care about any of the above (but do follow the link if you're interested in how technology relates to UK government, information assurance issues and identity management), but it has caused me to rethink my scheduling for this. I'm going to do more primary research--maybe through the summer--and then tie it up with a couple of really brilliant think pieces and try to sell it as a book.

I'd like to welcome the flood (well, trickle) of readers who have actually found this weblog--I assume it's academic assignments that lead you here. I hope you feel free to comment or ask questions.

Anyway, back to work.

Real World Consequences of These Issues

Let's try to look at real world effects of this dangerous illusion we are discovering.

The illusion is that a rapid growth in human knowledge (doubling every five years) is happening all around us, naturally and painlessly. As we hope to have shown on this weblog, human knowledge about a particular subject can increase that fast or faster, but it is always as a result of putting money, institutional resources, interested and bright people to do the research, together. It doesn't hurt to have an urgent public need (HIV) or potential private gain (nanotechnology) as a carrot.

But where these inputs are not provided, human knowledge about a subject grows at a much slower pace. Let me explain what I mean.

Click below for more...

Update and Summary

We are still 5 days short of our two-month anniversary on this weblog. I am going to try and do some more data mining before the 7th of June, but shortly after that date I am going to turn towards deeper analysis of data already collected. After the 7th, I will be doing research aimed at clarifying points brought up during the first two months' work.

What have we learned so far?

• Only 10 of 25 sectors explored are growing at a rate that would double every 5 years
• In areas that are heavily favoured by government or private industry, a sector can grow even faster than a 5-year doubling rate. But in areas that are not so favoured, growth ranges from a 10-15 year doubling rate (typically)
• In the U.S. this is correlated with an erosion of federal funding for research
• Growth in Asian knowledge engines (university enrollment, funding, patenting) does not overcome a decline in OECD states. It tracks GDP growth, and appears to be at a level that will barely sustain development in their domestic economies
• The increase in the availability of existing knowledge to more people is dramatic, and perhaps may compensate for the less-than-hoped-for growth in new knowledge.

So let's see what happens next.

More on U.S. Federal Funding for Research

Okay, as promised--here are the growth rates and doubling periods for the subcategory disciplines for U.S. federal funding. They are taken from this spreadsheet: Download us_funding_for_research3.xls, which is found here on the NSF website, where it is labeled Table 1A.

There's a real shocker below (to me, anyways). I put the charts up twice--once using the straight figures given by the NSF, and again after adjusting for inflation.

If anyone wants to know why the U.S. research output has declined, look at the second one. In 14 of the 26 disciplines measured, inflation-adjusted growth is negative. Only two categories show growth that is not embarrassing--computer sciences and 'other medical.'

Elsewhere we have seen that the number of the student-age population has declined (although higher proportions are enrolling). We have heard that our proportion of the contribution to the increase of knowledge is bound to decline, given the growing contribution from other countries such as China.

It's all bull. The U.S.A. defunded research. Click to see the numbers. Read 'em and weep.

The Future of Blogging

Well, I'm going off topic again. I've been doing this for a little less than two months, and I have some observations about the process I'd like to share.

First, I think a weblog is the most appropriate forum for me to conduct this research. I will probably test the waters to see if I can get a book out of this, but the ability to hyperlink, the opportunity to write conversationally as I work, the tagging feature of categories (as opposed to the rigidity of chapters), all have made this an easier job for me--something I can do nights and weekends--rather than a herculean labour.

Second, after everybody gets tired of using weblogs to recount their previous night's drinkng exploits, I think that efforts such as this will become the best use of weblogs. I'm not doing anything here that you could not do, if you are reading this. I've put up my principal sources of information and the major tools I use for reference, and if you want to do your own work or fact-check mine-or even take it further, I'll be happy to help in any way I can.

Lastly, as a non-academic (but one who has been exposed to a lot of academic publishing), I recognise that the work I'm doing here is not sufficient to be authoritative. However, I'm trying to ensure that if this stimulates the conversation on the topic adequately for institutional investigation, any researcher will be able to start from a higher point than if I had not done this. And you know what? That's enough for me.

Funding Frustration

Well, the previous two posts give a fairly good picture of how the U.S. federal government funds research (note that we did not examine state and local funding, nor private sector funding of R&D, nor defense funding, for that matter).

I feel satisfied that I know what's happening in the U.S. But my real frustration is that you, dear reader (icy ironic tone, please) will quite reasonably expect me to find this out for the world.

But while it only took 7 hours to compile the information for the U.S. (due to better and more available statistics), it will take weeks to get a picture of what's happening in the world.

I think I'll put a tip jar on this blog.

U.S. Research Funding by Sector

I think the real problem with this post will be presentation. I have an Excel spreadsheet that shows U.S. federal funding for reseach by sector, and I want to show the growth rates and doubling times. However, the time series for some sectors differs, for a variety of reasons. So, please be patient with me.

I will put the figures in the extended post below, as well as some analysis, and report the topline findings here. In a later post I will do the same for subcategories. I think it will look messy. The purpose is to see what choices are being made, so we can compare funding differences to outputs--the numbers of academic publications and patents in each field.

Mathematics and computer sciences have seen the greatest growth, with a CAGR of 10.54% and a doubling time of 6.92 years. Strangely, physical sciences (including astronomy, chemistry and physics) fared second worst, with a CAGR of 5.14% and a doubling time of 13.83 years. However, it did start with the second highest base total, if that's any justification... Slowest growth in federal research funding was experienced in social sciences, with a CAGR of 4.82%, which gives a doubling time of 14.72 years.

Update: I had to recalculate all the stats to account for inflation, converting all $ figures to 2006 USD. It changes everything...

Into The Jungle Of Research Funding

Oh, this hurts. I did not want to go into research funding. But now that it is fairly clear that the rate of progress varies significantly by sector, we have to examine the inputs.

The first relevant document found is a PDF from the European Commission. It states that it wants the EU to increase research 'efforts' to 3% of GDP by 2010. Ambitious, that. The document talks about the importance of 'basic research,' as opposed to directed, I assume, which is encouraging. It actually serves as a good basic primer on how funding is structured in Europe, the U.S. and Japan, but while it talks about funding for specific programmes, it doesn't talk about levels of funding at the national or international level. So we move on.

Curious Cat, a blog that bills itself as a Science and Engineering blog, posts research funding for U.S. academic R&D for the past few years (see below in the extended post). More importantly there's a link to the NSF where we find an Excel spreadsheet (also in the extended post below) that charts all federal spending on research for science and engineering between 1970 and 2003. There are other charts available for viewing and downloading here.

In 1970, combined spending by all U.S. Federal agencies for research in science and engineering totaled $4.9 billion (USD). By 2003, that had grown to $53.3 billion (USD). Not bad.

The compound annual growth rate for U.S. Federal funding for research into science and engineering is 7.28%, which means it doubles every 9.86 years.

However, adjusting for inflation to 2006 U.S. dollars, it reads a bit differently. In 2006 dollars, the spend in 1970 was the equivalent of $26 billion, while the 2003 spend rises on recalculation to a total of £58 billion. That provides a CAGR of 2.39%, which would double every 29.35 years. A bit sad, that.

But the spreadsheet breaks spending down by discipline, as well. I'll address that in the next post.

League Tables and Update

Here is the league table for 25 points of measurement of the growth of human knowledge. As shown, only 10 of the 25 grow faster than 14.87% per year, which is what is needed to double every 5 years.

Field                                                            CAGR                   Doubling Period (Yrs)
1. Nanotechnology patents                                  44.91                           1.87
2. Nanotechnology journals                                 42.03                           1.98
3. Global warming patents                                  38.62                            2.12
4. Prions patents                                                33.76                            2.38
5. Programming patents                                      33.53                            2.4
6. Stem Cells patents                                           26.47                            2.95
7. Prions journals                                                 25.57                            3.04
8. Global warming journals                                    24.71                            3.14
9. Epidemiology patents                                        17.37                             4.33
10. Stem Cells journals                                          16.63                             4.51
5 Year Doubling Rate                                          14.87                             5
11. Programming journals                                     12.55                             5.86
12. Alzheimers Disease patents                              11.26                             6.5
13. Oncology patents                                              10.02                             7.26
14. Alzheimers Disease journals                                9.65                             7.52
15. Oncology journals                                                9.23                             7.85
16. DeSolla Price estimate of world literature growth   7.0                             10.24
17. Epidemiology journals                                           6.22                           11.49
18. Mars journals                                                        5.78                           12.34
19. Shale oil journals                                                   5.53                           12.88
20. US patent grants                                                    5.21                           13.65
21. University enrollment worldwide                              4.85                           14.64

22. Publications in astrophysics since 1970                     4.0                             17.67
23. US patent applications                                             3.88                           18.21
24. U.S. Book publishing                                                3.65                           19.33
25. Shale oil patents                                                      2.58                           27.21

More discussion below--click if you want to see it.

So, Does Human Knowledge Double Every 5 Years?

1. No,

2. But it could

3. Because we see that it does in sectors where we pour in human and financial resources.

However, the Western world is not pouring in human and financial resources the way it used to. Students are choosing law, business and web design instead of engineering and science. Government is encouraging private sector R&D (and encouraging universities to work closely with business), but it doesn't happen enough and when it does, it is focused on very short term proejcts that improve a bottom line in the next quarter.

Absent the commitment of extra resources, it looks like human knowledge doubles every 10-15 years. I'll be showing my work in the next post, and then supplementing it with information about the commitment of resources, an unwelcome next task.

But where there is a clear incentive, such as nanotechnology or global warming (where the incentive may be just getting a project funding by slapping a green label on it), we get the output that shows that our knowledge and exploitation (via patents) of a given sector can double every five years.

Continued below--click if you want to see a bit of a rant...

Stem Cells

Searching for "stem cells" or "stem cell" on Scirus returns 995,702 results, of which 152,648 are journal publications with the term in the title, abstract or contents, and 62,220 patents. According to Wikipedia, Stem cells were postulated in the early 1960s and shown to exist in 1963, giving us a 40-year window to chart the progress of human knowledge. However, looking at Scirus, I find articles as early as 1916 that seem relevant, and a patent in 1947 that also looks relevant to a layman.

Starting in 1960 excludes 74 journal articles and 2 patents, so we will make that our base year. That gives us a Compound Annual Growth Rate (CAGR) of 16.63%, or enough to double human knowledge about stem cells every five years. Hooray! We have another winner. That actually would double every 4.5 years. For patents, the CAGR is higher, 26.47%, which would double every 2.95 years. (Just for fun, if I used a base year of 1947, when the first patent was filed, the CAGR is still enough to double human knowledge every five years, although it drops to 20.2%).

Perhaps the most important finding is that President Bush's restrictions on stem-cell work in the U.S.A. has not dragged progress down, with over 100,000 journal publications and over 56,000 patents filed since he took office. Of course, I'm not counting how many of each are American, but that's for domestic policy wonks to debate... or even research, if they have the time.

As in some (not all) other sectors, we see that the quickest growth for academic publications is early, while patents explode just about the time that growth slows in academia. I'm not sure that's a coincidence...

Again, we see that when the incentives are properly allocated, we can double human knowledge in a sector every five years.

Click to see the figures below.

That Eureka Moment--University Students Worldwide

I have been searching for this for quite a while. UNESCO had it--click here to see it on their website. I include the graph here--click on the image to see a larger version.

  The number of tertiary education students has grown from 68 million in 1991 to 132 million in 2004. That's a doubling over a 14 year timespan. That's a compound annual growth (CAGR) rate of 4.85%, which would lead to a doubling every 14.64 years. It's a third of the growth rate we would need to double every five years.

This is daunting. We see the world university student population mirroring the growth rate of India since 1947, when what I think everyone assumes is a growth rate similar to China's.

I don't think this is adequate growth to sustain a required expansion of conventional infrastructure, let alone power a Singularity.

Alzheimers and Human Knowledge

Alzheimer's disease destroys human knowledge, but perhaps more importantly, it decoheres the personality that organises this knowledge. It is only human knowledge that can defeat this destroyer of human knowledge. It has been described,  treated (unsuccessfully) and researched for a century.

Let's see what Scirus tells us about the growth of human knowledge (and the commitment of research resources) on this. (BTW, I recently had another look at the sources Scirus uses, and it includes PubMed, so I won't be replicating this search there.) (And I got a quick scare by starting with the search term 'Alzheimers,' which only returned 1,428 results--if you're fact-checking me, and I hope to God someone is, use 'alzheimer.')

A search for Alzheimer on Scirus returns 1,274,572 results on Scirus, of which 112,401 are journal publications containing the term in the title, abstract or text. The search also returned 67,900 patents.

In the extended post, I will give the results by year and by five-year period. That's a Sunday morning for you--hot coffee and a database to search.

Okay--we'll assign 1920 as the base year for journals and infer 4 journal articles published that year, giving us a CAGR (Compound Annual Growth Rate) of 9.65%. This would indicate that human knowledge regarding Alzheimer's Disease doubles every 7.52 years.

However, the first patent was filed in 1976, which gives us a CAGR of 11.26%. This would double every 6.5 years. So, although work proceeds at a good rate and human knowledge is increasing, there is nothing like the growth rate seen in areas such as nanotechnology or global warming.

Again, we see that patent growth, starting at a later date, grows much more quickly than academic publishing. I am personally mystified as to why research into prions has grown twice as quickly as research into Alzheimers. Many more people suffer from Alzheimers than CJD, and the potential for both alleviating human suffering and reaping large rewards are much greater. The overall totals for Alzheimers research are greater than prions, don't get me wrong--but why is the growth trend favoring prions?

University Students Worldwide

The OECD charges for their statistical publications, but sometimes you can find a browsable 'Read It' version (sort of try before you buy?). In any event, I found a freebie copy of a 168 page PDF of their Focus on Higher Education document for 2005-2006.

I'm looking for numbers for enrollment, hopefully a time series. However, as I browse through I will put other relevant information here in bullet points. Bear with me--this is a work in progress.

• There are more than 17,000 higher education institutions in the world.
• 'In the past three decades, the proportion of young people enrolled in higher education has risen from 20% of the population to just over 50% on average in the OECD.(Angel Gurria, Secretary General of the OECD.)
• In 2002, approximately 3.6 million researchers were engaged in R&D in the OECD (p. 46)
• In 2003, 2.12 million students were enrolled in universities outside their country of origin (93% in the OECD) (p. 64). This has increased by 50% since 1998.

And that's all. Big disappointment. I have seen elsewhere the figure of 88 million university students worldwide--it was from a UNESCO high muckamuck, and I'll link to it when I re-find it. But what I need are annual figures... for the world... going back 1,000 years, if possible--hey, you could send me a link...

UK Higher Education Numbers

According to this website published by Universities UK, In 2004/05, there were approximately 2.3 million students in the UK (Total undergraduate and postgraduate).

This chart shows growth from 1961 through 1995, going from 200,000 to 1 million in 34 years. It's evidently more than doubled in the past eleven years. Even looking only at this period, compound growth is 7.87%, which would double every 9.85 years. Looking at the entire period, CAGR is 7.02%.

Click on the image to see a bigger version. Courtesy of University of Leeds

With the recent addition of student fees, I don't see hypergrowth kicking in any time soon. Hope the rest of the world will compensate.

Prions

It's back to Scirus to see trends in publications and patents regarding prions.

Prions are infectious agents composed only of proteins, and are implicated in BSE (bovine spongiform encephalopathy, AKA mad cow disease) and Creutzfeldt-Jakob Disease (CJD) which as far as I can tell is more or less a human equivalent.

Prions were hypothesized in the 60s and actually found in the 80s, winning a Nobel prize for the disoverer, Stanley Prusiner.

The human disease is quite rare. Mad cow disease is not. Both are fatal. There is a public good to be pursued in preventing CJD and private profit in preventing mad cow disease. The two together should stimulate research. With a short timeline to look at, it will be interesting to see if human knowledge doubles every five years.

Scirus returns 13,518 journal publications with 'prion' in the title, abstract or text, and 8,781 patents. Because there is a bird called the fairy prion (Pachyptila turtur), I will actually have to look at the results... darn.

We hereby proclaim a base year of 1981, when 5 journal articles and no patents were published. That gives us a CAGR (compound annual growth rate) of 25.57% and we have another winner! Human knowledge has doubled every 5 years regarding prions! And patents have grown even more rapidly, at 33.76%--huzzah!

However, as we have seen elsewhere, a note of caution--while patent growth has consistently exhibited hypergrowth characteristics, journal publications have leveled off, with the period 2002-2006 growing 'only' 66% over the preceding five-year period.

Click below to see the results.

A Smart Science Fiction Writer Looks At The Future--And The Singularity

Charlie Stross looks at improvements in the speed of travel, information technology and bandwidth, says some very interesting things about convergence and the near future.

One idea just charms me--that our children will never have to worry about being lost, any more than we had to worry about being eaten by wolves. I suppose in about ten minutes I'll start thinking about the Big Brother implications, but these ten minutes are nice.

Hat tip Brad DeLong.

Programming, Software and Computer Science

Computer programming has a lot of work and a lot of people centered around it. If the Singularitarians are correct, we're going to need them all.

After all, what good is it if computing speeds and memory keep doubling if we cannot program computers to do what we want them to do? And the Singularitarians want computers to do some pretty ambitious things. In fact, given their wish list, it seems clear that human knowledge of computer programming has to double every five years if we are going to get the magic box to do what we want it to do. We will need new programming languages, tighter coding, better algorithms, and quite possibly a new way of looking at programming. Is this happening?

We've seen elsewhere that the number of people engaged in computer sciences has grown to 180,000 new entrants per year in the U.S. alone--and considering the base date for this field, that is evidence of healthy growth. Let's use Scirus to see what this growth in professionals has produced in the way of patents and journal articles.

Preliminary results: between 2002 and 2006, 38,685 journal articles were published with the word 'programming' in the title or abstract. As the total is 145,404, only 26.6% of all journal articles were published in the last five years. Of the 367,971 patents noted on Scirus with 'programming' in the title or description, 201,548, or 54.7% were published between 2002 and 2006--so that has doubled. Not the first time we have seen differing growth rates for journals and patents.

For 'software,' the period 2002-2006 saw 224,696 journal articles published, 46% of the total of 486,262. For patents, as with 'programming,' the period was more robust--589,919 out of a total of 900,846, or 65% of the total.

The base year for 'programming' is 1940, when a UK patent was filed for improvements on a statistical calculating device. The compound annual growth rate for journal publications is 12.55%--good, but not good enough to double every five years (it is less than six years, to be fair). The compound annual growth rate for patents is 12.83%.

Remember that we need a CAGR (Compound Annual Growth Rate) of 14.87% to double every five years.

For 'software, the base year is 1965, when 5 journal articles were published and 1 patent was filed. And software is a winner, with CAGR of 22.83% for journal publications!  Hooray for coders everywhere! And a double huzzah to IBM, Microsoft, Sun and countless others, as patent growth since 1965 has a CAGR of 33.53%. Another winner!

As there are so many more patents than journal publications for both search terms, we should probably give more weight to patents, and be extremely pleased that human knowledge appears to have doubled in the past five years in this area, especially given the gloomy results reported in my previous post last week.

Click below to see results and analysis.

Outcomes

Advances in human knowledge should make life better. That's hard to measure. However, we can measure if human life gets longer (it does), healthier, richer (if not more rewarding), etc.

The rest of this post is philosophizing--feel free to skip. And in a previous post, I talk about the Outcomes Fallacy. But if you want thinking on the issue of human knowledge and outcomes, click to read more.

University Enrollment in China

If we start with the premise that university students are destined to either become creators or transmitters of human knowledge, large increases in their total numbers would augur well for future trends in the growth of human knowledge. (Obviously there's plenty of room for contribution by non-graduates--at least I hope so, as a non-graduate myself.)

If you are interested in the background of education in The People's Republic of China, I highly recommend this article in Wikipedia.

Here's an article by People's Daily, written in 2000, about China's plans to expand university enrollment. They cite the number of 2.04 million in 2000, and projected an increase of 1 million by the end of the year. But the article is really bad and really is not useful, other than as a signpost for further research. It did however, have the huge advantage of being the first return on a Google search for Chinese university enrollment.

The China Education and Research Network published a 2001 article writing about a goal to increase China's university enrollment to 15% (of 18-24 year olds) by 2010 from the then level of 10%.

A link at the bottom of the article led me to this announcement, which is a bit hard to decipher:

• China's universities and colleges enrolled 2.6 million freshmen in 2001, 800,000 more than last year.
• Officials with China's Ministry of Education said that China plans to increase the number of university students to 16 million in the five years to 2005, compared with a total enrollment of over 11 million in the 1996-2000 period.
• According to a special plan for educational development, university students will account for 15 percent of school students in 2005. By that time, the number of postgraduates will reach 600,000.

So this supports, in a roundabout way, the first piece--2000 level of 10% with projected rise to 15%, but target date of 2005. Let's see if we can find out how they did...

In 1977, despite the fact that 5.7 million students took the uniform national examinations, there were only 278,000 university places available for them. What a waste... By 1985, (if I am reading the Wikipedia article referenced above correctly), that had risen to 560,000 places (there were also 550,000 places for professional and technical schools). And we see above that they enrolled 2.6 million in 2001. That's essentially a ten-fold increase in 24 years. Well done, China. But sadly, that's only a CAGR of 9.76%.

For science and engineering, we read that China is now putting out about 600,000 graduates per year. This does not appear to be a huge increase--Wikipedia tells us that there were more than 1 million applicants for science and engineering places in 1984, and we can kind of infer that there were about 200,000 accepted, a CAGR of 5.12%.

It very much resembles the growth in science and engineering found in the United States between 1950 and 2000. But it doesn't even replace that growth, let alone overcome the shortfall in U.S. (and OECD) science and engineering since 2000.

As so often happens, I find more relevant research after I've written all the above. (You get to see the work in progress, FWIW...)

The Chinese Ministry of Education does publish enrollment figures. I found out thanks to Wang Yibing, former UNESCO higher education specialist and part time professor at Zhejiang University. In this article (PDF), Wang shows the MOE figures for Institutes of Higher Learning from 1998 to 2006. I reproduce them below.

It shows that growth of post-graduate enrollment since 1998 (from 725,000 to 3,648,000) does show a CAGR of 22.38%. We have another winner! This is truly exciting--these are post graduates, after all. It almost makes up for the disappointment in U.S. patents found earlier in the week. Now we just have to hope that they don't get hijacked into designing toys for U.S. consumers...

Enrollment at Institutes of Higher Learning overall in China since 1998 also shows a 21.21% CAGR, again, more than sufficient to double every five years! Hooray, China! I am reluctant to carry this analysis back to 1977, because I believe my prior numbers separate universities and other institutes of higher learning, but we just have to root for another 5-10 years of growth.

Click to see the numbers.

Trends in U.S. Knowledge Workers

This is another grab bag post, where I park interesting numbers for further exploration at a later date.

• In the last 20 years of the twentieth century, the U.S. college-age population declined by more than 21%, from 21.6 million in 1980 to 17.0 million in 2000. The college-age population decline reversed itself in 2001, and will increase to 19.3 million by 2010 (a 13% increase over the year 2000 figure). Witty comment about how demography is destiny, please.
• The private, for-profit sector is by far the largest employer of science and engineering workers. In 1999, 74% of scientists and engineers with bachelor’s degrees and 62% of those with master’s degrees were employed in private, for-profit companies. So we might see an explanation of why patents grow more quickly than academic publications--at least in some sectors...
• Between 1991 and 2001, the number of jobs in the U.S. increased by less than 15% while science employment increased by more than 93%. Much of this can be attributed to the enormous growth in computer scientists. The number of computer systems analysts and scientists increased by 268%, or 1,135,000 people, during this period. The number of natural scientists increased by almost 33%. Within this category, medical scientists experienced the largest increase: 111%.
• Within engineering, as in the period from 1991–2001, electrical and electronic engineering is projected to have the biggest absolute employment gain, up by 31,000 jobs, or about 11% between 2000 and 2010. Larger relative gains are expected for biomedical, environmental, and computer hardware engineers. Further declines in petroleum engineers, as well as mining and geological engineers, are projected. Employment for all engineering occupations is expected to increase by only 9.4% over this period.
• The Bureau of Labor Statistics expects that the trends evident between 1991 and 2001 will persist. While the economy as a whole is expected to generate 15.2% more jobs over this decade, employment opportunities for scientists (excluding social scientists) and engineers are anticipated to increase by about 44%, amounting to an additional 2.3 million jobs.
• Approximately 88% of the increase in science and engineering jobs will occur in computer-related occupations. Overall employment in these occupations is projected to grow by almost 69% between 2000 and 2010, with more than 1.9 million jobs being added.
• Eight of the ten occupations expected to grow most rapidly between 2000 and 2010 are computer-related. The increasing demand for computer-related occupations reflects the rapid advances in computer technology and the continuing development of new computer applications, including the Internet and intranets.
• Overall, computer specialists, a component of computer and mathematical occupations, are projected to increase by 68.6%, for example. Five computer occupations — computer software applications engineers, computer support specialists, computer systems analysts, computer systems software engineers, and network and computer systems administrators — also are among the occupations with the largest projected numerical job growth. Computer and data processing services is the economy’s fastest-growing industry, and in almost all industries, employment in computer occupations is projected to grow much faster than average.

Well, they might want to rethink their forecasting on petroleum engineers, but there is no indication that anyone expects hypergrowth in the numbers of science and engineering workers.

Sobering News

The post below from yesterday, which details the stately, rather than exponential, increase in granted patents in the U.S., has really got me thinking, and I think we'd better report on the state of play here.

Let's detail those areas where we have seen adequate growth to double human knowledge every 5 years:

Update: We have since found other areas where the CAGR is sufficient to double human knowledge every five years:

Update 2: A modest review revealed errors in my calculations regarding oncology. Its position has changed (and not for the better).

Prions, where academic journal publications have increased at a CAGR of 25.57%, and patents have increased by 33.76%

Software, where academic journal publications have increased at a CAGR of 22.83% and patents have increased by 33.53%

Post-graduate enrollment in China, which has increased at a CAGR of 22.38% (although see below for more discouraging information about the growth of tertiary education worldwide)

(All percentages are CAGR (compound annual growth rate.)

1. Nanotechnology, with a blistering growth in publications of 97.16% per year. Patents, 76.15%.

2. Global Warming publications, with a of 16.85%

3. Total Chinese patent applications, at 25% per annum

4. Chinese computing patents, 58.29%

5. Chinese telecommunications patents, 30.26%

(was oncology)

7. Epidemiology patents, 22.76%

So far, that's it.

Now let's look at where we did not find anything like doubling every five years (remembering that we need a CAGR of 14.87% to double in five years time):

Update: We have now found other areas where human knowledge does not appear to be growing at a rate that will double in five years time:

Worldwide university enrollment, which has a CAGR of 4.85%

Alzheimer's disease, where academic publications have a CAGR of 9.65% and patents a CAGR of 11.26%

Programming, where academic publications have a CAGR of 12.55% and patents a CAGR of 12.83%

Chinese university enrollment, with a CAGR of 9.76%

1. U.S. patents granted, 5.21%

2. Epidemiology publications, 8.85%

3. Shale oil extraction, publications 5.59%, patents, 2.61%

4. Publications on Mars, 5.78% (see also here and here)

5. U.S. Book publishing, 5.14%

6. Indian university enrolment since 1947, 5.51%

7. Climate Models, 12.66% (Very close to 5-year doubling rate of 14.87%)

8. Improvements in Climate Models, 13.76% (Also very close)

9. Growth in publication of scientific and technical papers, 1995-1999

             OECD 0.1%

             U.S., -1.8%

             Canada, -2.2%

             UK, -0.1%

             Netherlands, -0.5%

10. Publications in astrophysics, 4% (since 1970)

11. Growth in percentage of cancer patients surviving 5 years after diagnosis, 0.7% (1960-2005)

12. Drop in percentage of soldiers dying from wounds received in battle, 1.54% (1941-2006)

13. Projected growth in number of scientists (U.S.), next ten years, 7.18%

14. Patent filings in India, last five years, 8.45%

15. Increase in patent filings, World International Patent Office, 1 year, 6.4%

6. Oncology pubications, 9.23% (revised)

Considering I cherry picked areas that I thought would be most likely to show robust growth, this is scary.

          

Human Knowledge and Patents

It used to be that, when an academic found something new in her or his research, she or he would write a paper and submit it to an academic journal. And it used to be that when corporate research and development departments (or some geezer in a garage) made a discovery, they would rush to apply for a patent.

Now, academics are quite likely to file a patent, or the institution they work for might--or both, in combination. So it would not be surprising to see a growth in patent applications.

The United States Patent and Trademark Office has been issuing patents since 1790. As of March, 2006, the USPTO had issued more than 7 million patents.

And wonder of wonders, miracle of miracles, they have a statistics page that shows patent activity by year since 1790. Wow.

We'll have to juggle a bit here. The applications for patent stops at 2005 (390,733), but the patent grants time series goes to 2006 (173,771). It's kind of important which year to choose, because in 2006 patent grants returned to growth after 2 years of decline. Not coincidentally, the Patent Office hired 1,100 new patent examiners in 2006 to deal with a backlog of applications.

Going to 2006, patent grants have shown a CAGR (compound annual growth rate) of 5.21%, which would cause a doubling of patent grants every 13.65 years. Patent applications started being counted in 1840, and as I said, we don't have 2006 figures. For the 165 years available, patent applications have shown a CAGR of 3.88%, which would double applications every 18.21 years.

Well, perhaps the rate of growth has been higher recently. That's what the five-year comparison is supposed to show. We will focus on patent grants only, and statistics are in the extended post below.

There are only two five year periods where the number of patents granted is double that of the preceding five year period. After the Civil War (1867-1871), when patents climbed from 26,613 to 61,646, and between 1807 and 1811, when patents rocketed from 366 to 898.

There are many five year periods where the numbers of patents declined, the most recent being the period between 1977 and 1981.

The figures below look very much like the real world. As the population increased and the environment became more conducive to technological innovation, human knowledge grew--but not exponentially and not, indeed, in a straight line. Recent performance is good--but no better in percentage terms than many periods preceding it. The number of US patents granted has not doubled in the past 15 years, let alone the past 5. We'll take a look at other countries' patent statistics later--maybe they are making up for the U.S. shortfall.

Click if you want to see the numbers.

Oncology

Entrez is the integrated, text-based search and retrieval system used at NCBI for the major databases, including PubMed, Nucleotide and Protein Sequences, Protein Structures, Complete Genomes, Taxonomy, and others. It is a service of the National Library of Medicine and the National Institutes of Health. However, it is included within Scirus, so the results will be taken from Scirus.

Searching with the term 'oncology' returns 148,053 results. 57,936 were published between 2002 and 2006 (inclusive). That's only 39% of the total, so it doesn't look like human knowledge of cancer has doubled in the past five years on that basis. In the preceding five years (1997-2001), only 38,153 results were published. That means that the most recent five year period did not even double the preceding five year period.

The CAGR for academic publications for oncology is 9.23% (since 1920), which would double every 7.85 years. Patent growth since the first filing in 1973 is 28.62%, which would double every 2.75 years. Using the same base date of 1920 would yield a CAGR of 10.02%, doubling every 7.26 years. I don't know which date distorts the picture less.

And the fact is, we don't know how many times we will have to double our knowledge of cancer before we will be able to make a big difference in the lives of those who have it. I have to say, I think I'd rather see the growth rate we note in nanotechnology here--and yes, I think I am saying I would make the trade off. I'm not advocating central planning of scientific research--I'm just wondering if research incentives are correctly allocated.

Now champions of nanotechnology wll say that nanotechnology in general has many direct applications to medicine in general, and further that some nanotechnology research is specifically concerned with oncology. And it's true. My reservation comes from my suspicion (for which I have zero evidence) that this is engineering led (hey--come here and see what my new toy can do!) rather than medicine-led (can you build something that will do x, y, and z in the following sequence and make sure it does not do a, b or c?) If my suspicion is correct, this wil prolong the search for solutions rather than shorten it.

Is there a truce in the war on cancer? Best put out that cigarette. Because we see that, although oncology did see a doubling of knowledge every five years up to 1991, that hyper-growth tailed off afterwards. Note that hypergrowth started before Nixon's war on cancer. Note that growth since hypergrowth stopped has been perfectly respectable--14,000 papers published in 2006 (more in 2006 than in the entire decade of the 80's), and 4,000 patents filed. And note especially that growth in patents has continued to double every five years. Make of it what you will.

The figures are below. Click to see the numbers.

So Where Did This Crazy Idea Come From?

We saw below that President Clinton said that human knowledge doubled every five years in at least two different speeches. We saw below that a respected German professor in Artificial Intelligence said that the number of scientific publications doubled every five years. But neither of them cited a source. I doubt if President Clinton did the research, and I can't find anything published by the German professor that deals with this. So where did it come from?

On several searches I've seen reference to UNESCO as a source of this information. And indeed, at the 11th General Conference titled Universities--Gateway to the Future, in 2000 Mahdi Elmandjra said, "Richard Knight estimates that the total knowledge of mankind doubles every seven years and adds,"90% of the present knowledge of mankind has been produced in the course of the last thirty years. But if we define knowledge as the capacity to survive on earth in a sustainable manner then 90% of human knowledge has been lost during the last 30 years."

But Richard Knight (referenced in other papers of the conference) is a professor of architecture at the University of Genoa in Italy. I cannot find anything published by him--and the quote says seven years, not five.

Where does this idea come from?

Relevant Research from 1988

It would have been very strange if I were the only one asking this question and doing some research. I'm glad to find some prior work on this, not least because it means I'm not the only one who cares.

A report on the International Conference on Fifth Generation Computer Systems in 1988, written by Jakob Nielson, refers to research by Jorg Siekmann of the University of Kaiserslauten in the then West Germany.

Nielson writes, "Siekmann reported that the number of scientific journals doubles every 15 years, the number of books in university libraries doubles every 10 years, and the number of scientific publications doubles every 5 years. The knowledge explosion has already reached a level where in chemistry it is often cheaper to conduct a possible duplication of an experiment than to search the literature to find whether the result of a previous experiment can be used." That last bit is scary--I wonder if it's still true. Let's see if we can find more about Siekmann and his research.

Google likes him--a little too much. There are 108,000 search returns on his name.

The University has some material about him: Jörg H. Siekmann is the head of several research groups working in artificial intelligence at the "Universität des Saarlandes" and at the German Research Centre for Artificial Intelligence (DFKI) in Saarbrücken. He is a professor in the department of computer science and a director at the DFKI. The five research groups are:

Automated Reasoning for Mathematics: OMEGA 
Competence Center e-Learning
Formal Methods: VSE
Multi-Agent Systems
The IT-Security Evaluation Centre

Currently he is the chairman of the collaborative research centre "Resource-adaptive cognitive Processes" (SFB 378) funded by the German Science Foundation (DFG). He is chairman of the network of excellence on computational logic (CoLogNet), and vice president of the International Federation on Computational Logic (IFCoLog) . His main research interests are Artificial Intelligence, Automated Reasoning, Multiagent Systems and e-Learning in Mathematics.

But I can't find any more information about his statement at the conference. However, I did stumble across something quite interesting that might lead to the origin of the statement  Human Knowledge Doubles Every Five Years. See next post (above).

How Important Is All This?

Now that I've been doing this for a month, the question of how important this all is is becoming more relevant. When you're in the middle of the hard slog of gathering data, it's normal to get frustrated. It's also normal to get a little grandiose in your thinking about the worth of what you're doing, partly to motivate yourself to keep going.

So how important is all this? If I come up with the finding that human knowledge doubles every ten years instead of every five, who will care? What will it change?

I mentioned a few things in an earlier post that I am starting to think are actually relevant questions for society. If we are convinced that human knowledge doubles every five years, then we don't need to quit smoking, as cancer will be cured before it strikes us. Similarly, we don't need to worry about pollution, because we'll sort it out before the effects can truly harm society. You can adopt this lazy attitude about any of the problems our world confronts.

But it would be a mortal pity if human knowledge doubles every six or ten or twelve years and you die of cancer the year before the cure is discovered instead of the year after.

The idea that human knowledge doubles every five years is widespread. A Google search using the term human knowledge doubles every five years returns 1,680,000 results (obviously only a fraction are relevant).

If that information is wrong, and individuals are making important decisions and organisations are misallocating resources in this mistaken belief, then yes, this research is important and worth doing. Gee, I feel reassured.

One Month Progress Report

Well, this weblog has now been up for 30 days. I had actually intended to do all the basic reseach in this time frame, but life has somehow interfered. A major reorganisation at work has slowed me down.

Nonetheless, we've managed to explore the growth in human knowledge in a number of areas, so let's recapitulate:

1. We do find a doubling of the evidence of human knowledge (journal publications and patents) in some sectors, such as nanotechnology and global warming, new fields that double from a low base. I'm tempted to label this the Manhattan Project effect, but I'm not going to do that until I study that sector since 1945.
2. We do not see a doubling in areas with a longer time series, such as astrophysics or the study of Mars, although there are periods of impressive growth in each.
3. In some areas, we see different trends for patents as opposed to journal publications. For example, in epidemiology we see an 8.85% CAGR in journal publications in the recent past, while patents have grown by 22.76% annually since 1987.
4. We do see evidence that human knowledge is available to twice as many people, and that has grown at a rate that more than doubles every five years. (Just looking at the growth of Wikipedia and the numbers of people connected to the Internet makes that fairly obvious).
5. We do not see evidence that the number of scientists and researchers doubles every five years, and it is not at all obvious that emerging country growth is making up for a slowdown or actual decline in OECD numbers.

But there's a lot more work to do, and any of those five points could change. I'm having fun doing this, so we'll just carry on.

Epidemiology

Well, Scirus, the database of Scientific knowledge I've been using frequently on this weblog, has a lot of material relating to epidemiology. I feel a bit vulnerable relying on this source so heavily--if anybody has suggestions for freely available information, I will happily (well, grumblingly and slowly and swearing occasionally) replicate these searches on it.

Searching for epidemiology on Scirus returns 1,500,983 results. 315,678 of these are journal articles. 8,530 are patent results and the other million plus are 'other' web results. So, focusing on journal articles and patent results since 1920, let's see if the numbers double every 5 years.

Human knowledge (as expressed by journal articles published) about epidemiology has not doubled in the past five years. Only 106,722 of the 315,678 journal articles were published between 2001 and 2006. It's truly a triumph of the human mind, but it is not doubling. It is a compound annual growth rate (CAGR) of 8.85%, which would result in a doubling every 9 years. Looking at the time series, we see dips and lulls and a steadily progressive trend line, with a couple of jumps. When I get the graphs prepared, it will be easier to see.

Overall, CAGR for epidemiology is 6.22%, which doubles every 11.49 years. Overall growth in patents is 17.37%, which doubles every 4.33 years.

However, backing out the 2007 patents (198) leaves a total of 8,332. Of that number, 5,444 (65%) were filed between 2001 and 2006. So while academic investigation did not double in the past five years, commercial research and development more than did so. Indeed, epidemiology patents appear to have more than doubled in every five year period since 1987, when 19 patents were filed, a CAGR of 22.76%.

I suppose, if we treated one journal article and one patent as exactly equal, we could add them up and compare the totals. But, as I certainly don't have any reason to think that might be the case, I'm not going to. The figures are below if you want to do it, though.

Click below to see the data.

How Would We Define Victory?

Well, time to define a few things here. We are trying to discover if human knowledge doubles every 5 years. To do this, we are testing three hypotheses:

1. We learn so much new information that the actual stock of knowledge is double what it was 5 years ago. We are testing this by seeing if scientific publications and patents are doubling every 5 years. We also look and see if there are twice as many scientists and researchers as 5 years ago.
2. That new and old human knowledge is accessible and usable to twice as many people as it was 5 years ago. We're looking at things like the size and reach of websites like Wikipedia, and also referring to our search for more creators and users of information.
3. We are disproving incorrect hypotheses at such a rate that we are doubling human knowledge by unlearning untruths, which corrects mistakes and frees up time for productive learning.

But this is a bit simplistic.  If I know one thing and learn another, have I doubled human knowledge or just what I have learned? Is a learned fact human knowledge before it is published?

If I know one thing and, without learning another, communicate that one thing to another human being, does that double human knowledge because two of us now know it? Or is that just repetition?

If I discover an hypothesis is not correct, does it make any difference if others continue to hold the old and incorrect belief?

And then we must make things difficult by defining knowledge. Many of you will have seen the pyramid that has data at the bottom, information on top of it, then knowledge and then wisdom. The implication is that data is a fact revealed, information is the fact in context, knowledge is is the fact absorbed and assimilated, and wisdom is right action based on knowledge as experienced. But is that an accurate or adequate definition of knowledge? Is it sufficient?

Wikipedia has a definition--it's a bit long, but the start is promising. Of course the start is a quote from Oxford's English Dictionary. "Knowledge is defined (Oxford English Dictionary) variously as (i) facts, information, and skills acquired by a person through experience or education; the theoretical or practical understanding of a subject, (ii) what is known in a particular field or in total; facts and information or (iii) awareness or familiarity gained by experience of a fact or situation." Yeah, that's it.

So does learning that the Earth is more than 6,000 years old mean anything if millions don't believe it? If creationists make learned scientists spend time in court patiently explaining evolution instead of working in the laboratory--their true 'field of the Lord,' have we learned anything? Does knowledge have to be unanimously held, or unanimously available?

Boy, where's a philosopher when you need one?

The Pot of (Black) Gold

Oil Shale Extraction could bring access to 1 trillion barrels of oil in the United States. Canada has even more. If some biblical figure could strike his staff against the stone and bring forth oil, he (or she) would be a very rich biblical figure, indeed.

But it's a bit tougher than that. An interesting article about the state of oil today notes that "companies developing tar sand production in Alberta announced that their costs of production were rising substantially."

Sounds like a job for science and technology... especially as biblical figures are a bit short on the ground these days. With rumors of oil production about to decline and continued resistance to nuclear power, someone who can bring this large scale energy source to our gas tanks will be heavily rewarded.

So what is happening in this subsector of a subsector? There are academic institutions that focus on engineering and others that focus on energy. There are patents to be filed, alliances to be made, deals to be done and advances to be discovered. Again, like nanotechnology and global warming, this is one area where we should expect to find a doubling of human knowledge every five years--at least.

Better yet, from our point of view, we've been extracting oil from shale since 1847 (those canny Scots at it again), so there should be a time series of publications for us to study--almost as long as our study of studying Mars.

We did find one, at Scirus again. It goes back to 1920, but only gives results for the decade up til 1950.

Taking an average of 4 journal articles for 1920 and 115 patents for the same year, we get CAGR (Compound Annual Growth Rate) of 5.59% for journal articles, and 2.61% for patents. As shown below, you can easily tell when oil prices rose.

But comparing 2002-2006 with the five years preceding, we can quite clearly see the effects of commerce on knowledge seeking. Journal articles during this period rose by 13.94% for the entire 5-year period. But patents--they rose by 99.19%, doubling within 5 years.

We see similar effects when measuring output in the 70's. Journal articles grew more strongly--at 245.86% for the five year periods. Patents grew strongly, but only rose from 1,098 to 1,730 or 57.56%. In fact, it makes me wonder if the oil shocks of the 70's drove academic interest immediately, which led to a knowledge base being available for patent seekers when prices rose again after 2001.

The story of oil shale and human knowledge looks much more like punctuated equilibrium than the steady advance of human knowledge, and what punctures the equilibrium is economics.

Results below...