Math in a Time of Excel: Economists' Error Undermines Influential Paper

Getty Images Harvard University professors Carmen Reinhart and Kenneth Rogoff
Carmen Reinhart and Kenneth Rogoff are at the top of the economics profession, with influence inside and outside the academy. Both have worked in senior positions at the International Monetary Fund; both have chairs at Harvard. Rogoff was a chess grandmaster in his mid-twenties, before giving up the royal game to focus on economics. "I'm not a great mathematician," he told the Financial Times, explaining the relation between his interests, "but game theory really clicked for me." This method of analysis offers insight into government behavior during a debt crisis, Rogoff suggested: "One of the reasons that Carmen Reinhart and I hit it off, is that we are both incredibly cynical about governments."

Now the most influential product of their collaboration – an argument widely considered to have helped lay the basis for the West's recent shift toward austerity economics – has been shown to rest in large part on a simple error, a spreadsheet coding mistake discovered by a 28 year-old graduate student at the University of Massachusetts Amherst named Thomas Herndon.

"I almost didn't believe my eyes when I saw the basic spreadsheet error," Herndon told Reuters. "I was like, am I just looking at this wrong? There has to be some other explanation." Herndon asked his girlfriend for confirmation that he wasn't missing something. He was right, she said.

The story begins with a large-scale research project carried out by Reinhart and Rogoff, investigating hundreds of years of financial crises around the world. The result was a well-received 2009 book, This Time Is Different: Eight Centuries of Financial Folly. They also published a paper based on a selection of their data, titled "Growth in a Time of Debt" (2010), which claimed to show that economic growth slows appreciably once a country's public debt to GDP ratio gets above 90 percent.

It was an argument bound to receive attention from the press and policymakers, because, according to some projections, the U.S. was set to pass that purported 90 percent red line in the coming decade. And the Reinhart-Rogoff result achieved notoriety on both sides of the Atlantic: Reinhart testified before the National Commission on Fiscal Responsibility and Reform; Paul Ryan cited the study in his 2013 budget, "The Path to Prosperity: A Blueprint for American Renewal"; and European Union Economic and Monetary Affairs Commissioner Olli Rehn spoke in February of "serious academic research" that indicated an economic danger zone above 90 percent debt-to-GDP. (The G20 countries have said they will soon consider a proposal "to cut public debt over the longer term to well below 90 percent of gross domestic product," The Fiscal Times reports.)

The Washington Post editorial board even presented the "Growth in a Time of Debt" finding as established economic fact. Writing about the risks of supposedly overly rosy economic growth and spending projections, the paper cautioned, "debt-to-GDP could keep rising – and stick dangerously near the 90 percent mark that economists regard as a threat to sustainable economic growth."
Growth in a Time of Debt

Needless to say, this being an academic question, there was disagreement all along. Several economists argued for "reverse causation," suggesting that high levels of debt are in fact caused by slow economic growth, rather than conversely. Others questioned the data that Reinhart and Rogoff chose to focus on, saying the authors excluded evidence that didn't match their take-home result.

But the fatal blow came when Herndon attempted to replicate the eminent professors' findings for a class assignment. He thought he would arrive at the same results but offer a different explanatory conclusion; instead he found that the numbers weren't right. When Reinhart and Rogoff provided their data spreadsheet, Herndon saw why: In the words of a paper he wrote with his teachers, "a coding error in the RR working spreadsheet entirely excludes five countries, Australia, Austria, Belgium, Canada, and Denmark, from the analysis." This spreadsheet error "is responsible for a -0.3 percentage-point error in RR's published average real GDP growth in the highest public debt/GDP category."

error in Spreadsheet

So that slightly negative average growth rate for countries with debt-to-GDP ratios was illusory, the product of an Excel error; 2.2 percent is the actual GDP growth rate for countries above the 90 percent mark.

Herndon and his co-authors found other errors, as well; Mike Konczal summarizes their findings at Next New Deal. Confronted with Herndon et. al's evidence, Reinhart and Rogoff released a response which essentially said "this changes nothing." They later conceded the coding mistake, saying, "It is sobering that such an error slipped into one of our papers. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper." They don't accept the other criticisms, which concern data manipulation (selective exclusions and unconventional weighting).

Herndon's teacher and co-author Michael Ash told Bloomberg that the UMass-Amherst team's new calculations do show "a modest diminishment of growth" in high-debt countries, but not "the stagnation or decline" claimed by Reinhart-Rogoff. The point, in Konczal's summation, is that "there's no magic number out there":

The debt needs to be thought of as a response to the contingent circumstances we find ourselves in, with mass unemployment, a Federal Reserve desperately trying to gain traction at the zero lower bound, and a gap between what we could be producing and what we are. The past guides us, but so far it has failed to provide evidence of an emergency threshold. In fact, it tells us that a larger deficit right now would help us greatly.

But an identifiable threshold was apparently alluring to the Harvard academics, and advantageous to political elites. The tide of opinion among policymakers may be shifting, however: Olli Rehn told Reuters that the euro zone will pivot back towards attempting to promote growth, rather than cut budget deficits. Whether this decision was influenced by the Reinhart-Rogoff affair was not discussed.

Increase your money and finance knowledge from home

Timing Your Spending

How to pay less by changing when you purchase.

View Course »

How much house can I afford

Home buying 101, evaluating one of your most important financial decisions.

View Course »

Add a Comment

*0 / 3000 Character Maximum


Filter by:

It's funny that now the Liberals, Dems and Left *don't* cite Warren Buffet (he's long fretted about government debt being too high).

So, if austerity is not the answer, what do you guys imagine is? Going deeper into debt?

April 27 2013 at 11:22 AM Report abuse rate up rate down Reply

conservative math, should have known..................

April 25 2013 at 11:31 PM Report abuse +1 rate up rate down Reply
1 reply to josephautomotive's comment

What a grounless comment. It was an error. Errors are not political.

April 26 2013 at 3:51 PM Report abuse rate up rate down Reply

What difference, at this point, does it make?

April 25 2013 at 9:21 PM Report abuse rate up rate down Reply
AnTique Geex

...meanwhile the global economy continues in experimental mode until the theory is proven or disproven by empirical data.

April 25 2013 at 6:37 PM Report abuse rate up rate down Reply

I can't help but wonder if those making the spreadsheat have a new job. I tend to think they may have been performing compilations at the Republican Headquarters relating to polls on Romney's state by state numbers last election. Hummm na couldn't be them.

April 22 2013 at 2:59 PM Report abuse rate up rate down Reply

If they were serious people, they'd be thanking the guy for finding a mistake in their research and calculations.

April 22 2013 at 1:13 PM Report abuse +3 rate up rate down Reply
1 reply to Mike's comment

Still wiping the egg off their faces afrer the breakfast of Crow

April 25 2013 at 11:32 PM Report abuse rate up rate down Reply
1 reply to josephautomotive's comment

Actually they are still trying to ram austerity down the throats of the world's nations, regardless of the deeply and criminally incorrect nature of the "research".
Criminal because millions of lives have been wrecked by nations and elites bent on inflicting the austerity program on everyone except themselves, come hell or high water, because it increases profits and plays the trojan horse for privatization.

April 26 2013 at 9:29 PM Report abuse rate up rate down
Normie Baby

With all their denials concerning the data, I believe they both would be perfect candidates for the Global Warming scam. The should start calling themselves climatologists.

April 22 2013 at 1:09 PM Report abuse -1 rate up rate down Reply

It's no longer about the actual data, or even the end result. It's how powerful we feel about the compilation, construction and final submission of the report- and all of the computer wizardry it takes to get there. (Look what I did!!) Gotta love corporate type finance, in the end It's really about the presentation and having all the right fonts and colors. Come on, be real- how many times have you reworked a report because it wasn't aesthetically appealing on the handout or power point etc- now compare that number to how many times you reworked it in order to recheck your data result?
Data? What data? Isn't that the computer's job? Oops!

April 22 2013 at 12:14 PM Report abuse +1 rate up rate down Reply
1 reply to lbpurchasing's comment

Sometimes I don't like the truth. Especially when it makes me look sooooooooo bad. Nail on the head truth. God I hate all the reports.

April 22 2013 at 2:56 PM Report abuse rate up rate down Reply

Ah yes. Point out one stupid move to try to change the conversation. The reality is if you look at the trend it still points clearly as a support of their conclusion that is that increased debt reduces GDP. But stupid mistake so I think we should burn them at the cross

April 22 2013 at 11:43 AM Report abuse rate up rate down Reply
2 replies to kevgtwin's comment

Actually, the R&R report shows correlation, not causation. It's just as possible that it was the decreasing GDP that led to higher debt. In fact, it's very likely. What happens when GDP decreases? Tax revenue decreases as well? If all things are the same, then this leads to higher debt. What also happens when GDP decreases? It usually leads to layoffs right? As long as there is a social safety net within the country, then this requires additional safety net spending. If all things are the same, then this also leads to higher debt.

As GDP decreases, the government gets both a reduction in tax revenue, and an increase in safety net spending. All things being the same, this leads to higher debt.

That's not even getting into whether the government has spent money while attempting to stimulate the economy! Even higher debt, which may not have a 1:1 impact on GDP in the first year.

Now.. what conclusion was R&R attempting to come up with again, save for a very obvious correlation?

Their idea of a substantial GDP cliff dropoff at 90% debt, for which Republicans have referenced multiple times as a reason for cutting spending in the US, was found to be 100% false. In fact, one could say that this 90% dropoff was the only reason the R&R report garnered as much fame as it did.

April 25 2013 at 2:09 PM Report abuse +1 rate up rate down Reply

More likely it is the (mistaken) belief that debt slows the economy, that is creating a self fullfilling prophecy. Any one who looks at it will tell you that the economy and the markets are ruled largely by emotion, and fear is a very stong emotion. Now we find out that this fear is the result of faulty data and manipulation to make a study fit a hypothesis. IMO alot of these problems were created by the party that lost most of their political power about 6 years back in an effort to regain that power. One more reason to make sure they never get into power ever again

April 25 2013 at 11:39 PM Report abuse rate up rate down Reply

Ah, computers... raising simple human error to a whole new level.

April 22 2013 at 11:24 AM Report abuse +2 rate up rate down Reply