What's the ROI of the NIH?
The NIH frequently boasts of the enormous rewards that the American taxpayer reaps from scientific investment. Supported by United for Medical Research (UMR), a short report is updated and published annually. 2024’s report is here, and it looks like this.
Look at the arrow going up and to the right!
Every dollar of input that goes into the NIH, so stated the ex-NIH director recently on 60 Minutes, results in $2.46 of economic activity, a return of 146%! Wow!
These are not new claims; I’ve always wondered where these figures come from but it’s never been top of mind enough for me to actually investigate until now when the NIH faces its first real political threat to its budget.
I’m broadly very sympathetic to this rosy view of scientific spending. Modern technology cannot exist without scientific research and technology is necessary but not sufficient for any economic growth we’ve experienced since the stone age. Technology is the only weapon available for humans to increase their productivity. Science is the embryonic precursor of that technology, and science has this other weird property: it’s entirely made of ideas.
Scientific ideas can be copied and mimicked and in a real sense, they’re free. This makes measuring the return on scientific progress difficult, and it makes the plausible societal value of scientific ideas potentially very large. That’s why it’s actually a bit difficult to dismiss out of hand the extraordinary claims about the 146% return that the NIH makes.
RIMS II
The report linked to above states in a footnote that
UMR’s analysis of the employment and economic activity attributable to NIH extramural research spending in the 50 U.S. states and the District of Columbia, relies on the RIMS II model maintained by the Bureau of Economic Analysis, part of the U.S. Department of Commerce.
RIMS II is the Regional Input-Output Modeling System II. This idea harkens back to ideas about how an economy works from the 60s and 70s. The economy is modeled as a collection of independent regions and interdependent sectors within those regions that make things; in a given region, a sector then turns input from another sector into its own output. Because everyone in government still works under Keynesian assumptions, each of these transfers between regional sectors has a multiplier and this multiplier is determined by local scarcity of the thing being transferred. Let’s look at a stylized example of how this might work.
A science administrator at NIH, let’s call him Antonius Faustus, makes an extramural grant to a lab in Cambridge, Massachusetts at MIT. With the grant money, the PI hires a lab manager, pays MIT overhead from the grant, and buys beakers. The BEA has taken a look at Cambridge with census data and other survey methods and noticed there are a lot of labs and lab managers there. Since the lab manager spends his wages in Cambridge, this money turns over through the local economy and produces more “economic activity” than what originally got paid to the lab manager in wages.
The same thing happens with the MIT overhead. MIT is embedded in Cambridge with many-long standing business relationships. The BEA figures the overhead capital will turn over through the HVAC company that keeps the lab cool or the pizzeria that the biology department hires to deliver pizza for journal clubs. So just by virtue of giving the lab in Cambridge money, so UMR reasons, Antonius Faustus has produced a return, a surplus, on top of the original grant money.
Something else happens with the beaker money though. Suppose the tariffs haven’t yet worked and there’s no reshored beaker factory in Cambridge. That money leaks out of the Cambridge economy to Corning, NY or wherever beakers are made. This money will get a lower multiplier, since the money can’t turn over in the regional economy like the lab managers’ wages or the MIT overhead expenses.
This all gets formalized into a gigantic matrix of numbers that the BEA maintains and updates annually. You do this for all the extramural grants in all the regions the NIH funds, invert the matrix, add up the numbers, and you get a total return on the NIH’s budget of 146%. Magic!
Where’s the science?
I’m going to charitably give the RIMS II multiplier logic the benefit of the doubt here even though multipliers never made any sense to me even when I first learned about them in college. The careful reader will note that the above stylized story about Faustus and the lab in Cambridge has nothing whatsoever to do with science. The story would be exactly the same if the lab manager played solitaire all day, the lab never did any experiments, and the PI never published any papers. The RIMS II accounting that Francis Collins and the feverish defenders of the NIH budget parrot has absolutely nothing to do with knowledge accumulation or the returns of scientific progress. By this accounting, scientific grant making is simply a jobs program. Even if the numbers were completely correct, I’d feel pretty embarrassed as a scientific funder if I excluded scientific progress when accounting for my final work product.
Maybe making comparisons of science grant making to a jobs program is a little too harsh. The slightly more charitable way of thinking about it is that scientific work is like making steel or roasting coffee. These things are scarce, tangible, intermediate products that work their way through an economy. Other sectors in the RIMS II model use them as inputs. Some have larger multipliers than others, and are therefore more locally valuable.
This too undersells what’s going on with good scientific research, which is inherently non-local and non-scarce. Scientific ideas can be sent around the world and used by anyone. In the RIMS II terminology, they have inherently tiny local multipliers, which makes this methodology particularly inappropriate for accounting for the returns of an organization like the NIH.
The actual returns to scientific funding
Frontier Economics put together this detailed meta-analysis claiming the social returns from R&D investment were twice that of the private returns and were probably at least 40%. The long and variable temporal lags that accumulate on the returns of ideas make these estimate tricky, but it’s still an enormously good deal if true. A very recent NBER paper concurs and states
returns to federally funded R&D appear to be substantially higher than the returns to other forms of federal investment such as physical infrastructure.
They favorably cite Fieldhouse and Mertens, 2023 who claim the extraordinary range of 150 to 300% after WWII. Jones and Williams, 2017 think the return is probably north of 30%.
Caroline Ellison claimed Alameda Research produced 110% percent returns.
However, Alameda Research was a scam and Ellison was a liar. Why does the NIH seek to mimic these types of dubious and unsupported claims? I have no idea. They have the truth on their side, and as it turns out the truth makes them look good. Scientific ideas, properly conceived, are not a scam, but their returns are difficult to measure.
Coda
The NIH and scientific funding more broadly need serious reform and part of reform means cuts. Many annoying people on Twitter have reported the $2.46 statistic as if it’s gospel.
One of the reasons this is so annoying is that these people are broadly correct: science is made of ideas, ideas are a public good, and public goods are by definition under-invested in. Amid a replication crisis and what looks like a new and daunting scientific race with CCP, I don’t envy the new NIH’s director’s job balancing these reforms with protecting his budget.