Why Reinhart and Rogoff Aren’t the Real Villains

Update (May 2): We’re finally seeing more fair-minded commentators taking a balanced view of Reinhart’s and Rogoff’s research. After investigating data discrepancies for New Zealand (pointed out by reader Margaret below), Bloomberg’s Matthew C. Klein called out the three University of Massachusetts Amherst authors for not recognizing that the so-called data “omissions” were suspect data points as explained by Reinhart and Rogoff all along (h/t Marginal Revolution). ECONJEFF would have liked to have seen more “professional courtesy” and called for economists to establish “norms of good behavior” for replications/critiques. Also, John Mauldin applied his usual clear thinking to the topic in his weekly “Outside the Box” article. And blogger George Washington looks at the bigger picture in the austerity debate, noting Reinhart’s and Rogoff’s advocacy of debt write-downs in Europe (which would have meant less austerity in countries like Ireland and Portugal) and stressing that until governments stop saving “the big banks instead of the little guy” and other basic problems are fixed, “neither stimulus nor austerity can ever work” (h/t Angry Bear).

For those surfers who haven’t already lost interest in the spirited debate over a 2% calculation discrepancy in an historical average, please read on.

I’ll share some thoughts on the academic paper that was released last week by three University of Massachusetts Amherst scholars and the Internet frenzy that followed. My goal is to clear up a handful of fallacies that have taken hold in the media and blogosphere, while also throwing in my editorial comments and “scores” on each of the parties involved.

The three Massachusetts authors – Thomas Herndon, Michael Ash and Robert Pollin, now known on the Internet as “HAP” – argued that a heavily cited 2010 paper by Carmen Reinhart and Kenneth Rogoff (RR) contained fatal errors. In a critique that was released to the public last Tuesday, they revealed a calculation error in one of RR’s spreadsheets. And they also argued that RR omitted data points without justification and used an unconventional weighting method in their statistical averages. In response, RR acknowledged the calculation error but defended their data set and weighting methods.

This is more than an obscure academic debate only because RR’s conclusions are well-known in both academic and political circles. In their 2010 paper and a second paper released last year (with Vincent Reinhart), they suggested that economic growth tends to slow after government debt rises above 90% of GDP.

They’ve backed their thesis with calculations covering a few different country groupings and time periods, and also by summarizing the work of a handful of other researchers who’ve reached similar conclusions. It’s also clear that they truly believe that excessive government debt leads to lower growth – on a conceptual basis – as do many other people.

It’s always politics. Never personal.

But their beliefs don’t sit well with HAP, who argue that RR have too much influence over public policy decisions in both the U.S. and Europe. HAP suggest that policy could be less austere in both regions. They don’t like to hear politicians repeat RR’s warnings about the dangers of high debt, and they hope to discredit RR to bring an end to their perceived role in current policies.

HAP’s paper concludes with the statement: “RR’s findings have served as an intellectual bulwark in support of austerity politics. The fact that RR’s findings are wrong should therefore lead us to reassess the austerity agenda itself in both Europe and the United States.”

But HAP haven’t challenged the full breadth of RR’s thinking and research on fiscal policy matters, at least in the critique they issued last week. Instead, they focus on just one of the calculations in one of RR’s papers – arithmetic average returns for 1946 through 2009 in the first paper.

Here are the competing views on these average returns:

rr paper

From this simple chart, pundits and bloggers launched a days-long game of “whisper down the lane.” We’ve been fed a succession of incomplete, exaggerated, misleading and erroneous reports, as I’ll explain in these six observations:

  1. For all the public focus on RR’s spreadsheet calculation error, it didn’t have a meaningful effect on their results. As reported by HAP in their paper (see page 7), it changed the arithmetic average in the >90% bucket on the right hand side of the chart by 0.3%. That’s pocket change. But the error’s insignificance was remarkably emphasized in only two of the many accounts I’ve read (kudos to Justin Fox of the Harvard Business Review and Brad Plumer of the Washington Post, with apologies to those I didn’t read). In a couple of the very earliest reports on the paper (see here and here), the authors eventually backtracked by adding a mix of clarifications, corrections and updates to their original posts, presumably after recognizing that they overstated the error’s significance. But they both left their prose written in a way that continued to emphasize it. And their later clarifications didn’t stop other commentators from reporting that the return differences shown in the chart are explained entirely by the error, which is simply untrue. Nor did they prevent sensationalistic titles such as these: How an Excel error fueled panic over the federal debt (LA Times), FAQ: Reinhart, Rogoff and the Excel Error That Changed History (BusinessWeek), Math in a Time of Excel: Economists’ Error Undermines Influential Paper (DailyFinance)
  2. Much of the reporting extended beyond the 2010 paper, leading readers to believe that HAP’s critique invalidates RR’s other work, including their 2009 bestseller, This Time is Different. An LA Times report, which was also picked up by RealClearPolitics, even claimed that RR “popularized” the 90% threshold in their book. In fact, the book did no such thing, nor did RR publish any similar results before 2010.
  3. The dispute centers on the slope and significance of the line in the chart above, and particularly the last segment leading to the 90% bucket, not whether it’s rising or falling. But that didn’t stop pundits from writing their accounts in ways that suggested disagreement about the line’s direction. Moreover, RR pointed out that they placed more emphasis on medians than averages in their discussions (which is entirely consistent with a reread of their papers), and the medians escaped HAP’s critique without comment (more on this below). The fact that HAP’s average return calculations yield similar results to RR’s medians has received almost no attention in the public discussion.
  4. RR never presented 90% as a magic number – where 89.9 is a clear, sunny day and 90.1 a class 5 hurricane – nor did they neglect to recognize that correlation is not causation. I’ve cited the paper on several occasions and never saw it in the way that their critics claim it was presented. Marginal Revolution – probably the most heavily trafficked economics blog – recently republished a 2010 post that likewise didn’t consider 90% to be either “sacred” or “stable.” I doubt that any of those who read RR carefully saw 90% as more than the upper limit on one of their buckets and a reasonable marker to use in their conclusions.
  5. The austerity push in Europe wasn’t triggered in any way, shape or form by RR’s research. It’s based on northern Europe’s struggle to limit the potential damage to their own economies from fiscal crises in the peripheral countries, as explained by OpenEurope here.
  6. Similarly, RR aren’t the puppet masters controlling Republican budget strategies in the U.S., notwithstanding Paul Ryan’s reference to their research, which was discussed by HAP in their paper and repeated many times in articles last week. I’m not aware of any public comments from Ryan on the matter, but it seems unlikely that we’ll wake up tomorrow and read all about his conversion to the “debt doesn’t matter” school based on HAP’s critique.

Keeping score

And now for the scorecard that I promised. I’ll start with RR:

-0.5 for an Excel error that should have been caught before publication. But this is a minor issue, as I pointed out above.  I’ve reread the paper to check the effect, and the error didn’t change a single word. We all make mistakes, and this one wasn’t even a factor. It’s like the stumble that costs a distance runner a fraction of a second but doesn’t change his position in the race.  I’ll say it again: It didn’t change a single word.

Dead-even on the debate over the weighting method. RR have a clear and logical defense for their approach, while HAP offered a reasonable criticism. This happens all the time in academia. People think and act differently, and they also approach research differently.

Dead-even on the data omissions raised by HAP. I have no reason to doubt RR’s defense that their data set wasn’t complete when they wrote the paper. I’ve used their data on several occasions and seen it evolve over time, with significant additions to their government defaults in 2011, for example. And it’s not easy to build such a large data set that you can use with confidence, let alone share with your peers as RR have done graciously. One of the disputed countries (New Zealand), in particular, didn’t surprise me in the least, since earlier this year I collected significantly different debt and growth figures from different sources.  Unless you choose to be careless, these types of discrepancies take time to sort out.

-1 for the interactional effects of their various methods. Based on the mix of methods that RR chose, HAP pointed out that their average return for the >90% bucket assigned a 14% weight to a single year’s growth in New Zealand. The year happened to be 1951, when New Zealand’s economy contracted by 7.6%. This seems too much weight for such an extreme result and it would have been helpful to report its effect on the results.  But it’s not the intellectual travesty that many have made it out to be.  Empirical work is always vulnerable to outlying data.  The important thing is not to make your methods perfect, which is impossible, but to recognize their limitations.

+10 for their contribution to their field. Yes, I’m biased in that I believe they’ve built the world’s most comprehensive history of the types of risks that are most threatening to us today. Their data set and book are tremendous accomplishments. And remember, they operate in the field of macroeconomics. If you were to review all of the published papers in this field for the last, say, 100 years, and weigh them up against real life events, the vast majority could be shown to have major shortcomings. Many have done real damage, leading policymakers to adopt views that are hopelessly disconnected from reality. It’s no exaggeration to say that the foundations of conventional macroeconomic theory have been discredited repeatedly in the last century. And it’s the papers that rely on unrealistic, abstract theories that should concern us most, not a 2% disagreement in an historical average. If you’re interested in an example of the type of paper that deserves to be stamped out, see this post from earlier this year. By comparison, HAP vs. RR is pretty ho-hum to me.

Here are my scores for HAP:

+2 for delivering a thoroughly researched critique on one aspect of RR’s paper.

-1 for the way it was done. I don’t know this for sure, but it seems as though RR gave HAP their data and spreadsheets and yet didn’t even receive an advance copy of the critique. Their initial response suggests they saw it for the first time last Tuesday afternoon, well after blogger Rortybomb had already read the critique, interviewed the authors, examined their spreadsheets and written his article. Because of this apparent ambush, many people formed their opinions without seeing both sides of the story.

-1 for failing to even acknowledge the majority of RR’s results on the empirical relationship between growth and debt. They had no comment whatsoever on the very first result cited in RR’s 2010 paper – the finding that the median return is about 1% lower when debt rises above 90% of GDP. And they also failed to comment on the first result cited in RR’s 2012 paper, which also referenced a return difference of about a percent. Instead, they focused exclusively on arithmetic averages over a single time period and calculated a revised return difference of, well, about 1%.  In other words, we’re being asked to cross out RR’s 1% and replace it with the more “accurate” 1% that’s been reported by HAP (see chart below).  Can someone please remind me what we’re arguing about?

rr paper 2

Overall, HAP certainly offered some analysis for people to consider, while pointing out weaknesses in the 2010 paper, as should be expected of a critique.  But they’ve just as certainly failed to disprove RR’s thesis that high debt tends to be associated with lower growth. (Note that I’ve not claimed causation, and nor did RR in their papers.)

And lastly, let’s assign a score to the pundits

In the meantime, pundits with a predisposition toward loose fiscal policy have launched a character assassination of remarkable force.  One only needs to read a few of the more critical essays and comment threads to see RR subjected to a treatment normally reserved for crooks and felons.

And the most amazing thing about the past week may be how many people became instant experts on exactly how RR described their research to policymakers all over the world.  I must have been the only one who missed the nightly Reinhart and Rogoff Hour on national tv.  Since I follow their work and use their data more than most, it seems odd that I was the guy left out of the loop.

Which brings me to the scoring for the pundits who unleashed this frenzy. To me, their contribution isn’t so much a number but a smell. They’ve left a stench of hypocrisy and a strong whiff of political trickery, by using sensationalistic language and misrepresenting the real issues.

It’s easy to see why they sided with HAP – they’re philosophically opposed to RR’s policy advice.

And it’s easy to see why they emphasized the insignificant spreadsheet errors – no-one would have paid attention if they hadn’t.

But in the process of using the critique as an opportunity for political chest banging, they made clear errors in articles that were written to heap scorn on someone else’s errors. And they should own up to their mistakes exactly as RR did.

Bookmark and Share
This entry was posted in Uncategorized. Bookmark the permalink.

37 Responses to Why Reinhart and Rogoff Aren’t the Real Villains

  1. HAP should be criticized for not reporting a test of the hypothesis that mean growth rates are equal for countries with debt ratios from 60%-90% and debt ratios of 90%-120%. Had they reported this test they would have rejected the hypothesis of equal growth rates and concluded (using their own data and methods) that the negative correlation between debt ratios and growth was statistically significant over this range. Instead, in HAP’s Table 4, they report results of a joint F-test that growth rates in countries with debt ratios of 30-60%, 60-90% and 90-120% are not statistically distinguishable (p-value =0.11). The latter test reported by HAP does not address the key issues raised by RR. RR acknowledge that the correlation between debt and growth is weak for debt ratios from 30-90% and say that growth begins to drop off after debt ratios exceed 90%.

    • bv says:

      interesting that one of their main gripes was RR’s weighting, in light of that.

    • jk says:

      Not the proper test: If you allow for data inspection before dividing into temporal brackets you would not obtain valid p-values: the general F-Test for all time intervals is the better test and avoids data fishing…. unless you believe in this miraculously precise a priori hypothesis…

  2. Hojat says:

    From reading this article, the only thing that I got, and I don’t know you at all, is that you yourself are against higher debt for no good reason.

    Best

  3. ffwiley says:

    Thanks SB, excellent point.

    As far as Hojat’s suggestion that there’s no reason to be concerned about higher debt, I began a series of articles last month on this topic – “Answering the Most Important Question in Today’s Economy,” for example – and will continue to add to it.

  4. RP says:

    In regards to paragraphs 4 and 6 above, I point you to the following:

    http://www.theatlantic.com/business/archive/2013/04/forget-excel-this-was-reinhart-and-rogoffs-biggest-mistake/275088/

    You are correct that RR’s paper itself did not sell the 90% threshold as a magic number, but RR did that themselves all to often afterwards as the link clearly cites.

    Also, it does seem that for blog based on a cynical viewpoint towards economic theory, you have a not so cynical viewpoint of the behavior and motivation of our elected officials. Hard not to understand how two sort-of-famous Harvard economists, and authors of a well know paper detailing the dangers of higher debt, would have some sway over politicians:

    But their biggest overselling didn’t come in the media. It came behind closed doors — in Congress. Tim Fernholz of Quartz flagged the following passage from Senator Tom Coburn’s recent book about the time R-R briefed members of Congress in April 2011, a few months before the debt ceiling debacle:

    Johnny Isakson, a Republican from Georgia and always a gentleman, stood up to ask his question: “Do we need to act this year? Is it better to act quickly?”

    “Absolutely,” Rogoff said. “Not acting moves the risk closer,” he explained, because every year of not acting adds another year of debt accumulation. “You have very few levers at this point,” he warned us.

    Reinhart echoed Conrad’s point and explained that countries rarely pass the 90 percent debt-to-GDP tipping point precisely because it is dangerous to let that much debt accumulate. She said, “If it is not risky to hit the 90 percent threshold, we would expect a higher incidence.”

    • ffwiley says:

      RP,

      Are you really suggesting that RR shouldn’t say what they truly think when they’re speaking with policymakers? And by that logic, presumably Krugman, Delong and Summers shouldn’t say what they think? (Would have been kind of hard for Summers, come to think of it, since he was the policymaker not too long ago.)

      There’s a huge difference between giving your opinion and producing erroneous research. RR clearly believe that excessive debt leads to lower growth, as I said in the article. We all know they’ve shared their beliefs, as they should be expected to do. And last week’s critique certainly didn’t show that they’ve lost their right to speak freely. On the contrary, it confirmed that RR’s thesis hasn’t been disproven. And apart from the error that didn’t matter, the research was not “wrong” as reported by HAP and others.

      BTW, you may have seen the Wash Post came out and debunked the idea that they were meaningfully influenced by the RR paper in their editorials, as so many pundits insinuated last week. No-one is denying that RR doesn’t speak with policymakers. But their influence was greatly exaggerated (misrepresented, really) and that’s the backlash that you’re hearing now. You may think that exaggerating RR’s influence amounts to cynicism. In my book, it’s nonsense. And using the same two examples that I used in the article above (Ryan and Europe), the Wash Post called it preposterous.

      • Al says:

        It’s difficult to weight the post-HAP claims of politicians that RR did not have a great effect. Once committed to a position, admitting that your intellectual foundations contained major flaws (the non-existant “tipping point”) is like baring your belly. A quick google search shows that the 90% figure made appearances in many political speeches, although you say that it doesn’t mean anything, and so perhaps it doesn’t.

      • Cas says:

        ““Absolutely,” Rogoff said. “Not acting moves the risk closer,” he explained, because every year of not acting adds another year of debt accumulation. “You have very few levers at this point,” he warned us.

        Reinhart echoed Conrad’s point and explained that countries rarely pass the 90 percent debt-to-GDP tipping point precisely because it is dangerous to let that much debt accumulate. She said, “If it is not risky to hit the 90 percent threshold, we would expect a higher incidence.”

        “”Are you really suggesting that RR shouldn’t say what they truly think when they’re speaking with policymakers?”

        No, but is also seems a tad confusing to the overall tone of your blog entry to be shocked that some folks have pointed out that RR actually oversold their paper’s conclusions to policy makers and legislators. If you grant (and you don’t have to) that RR told politicians et al that 90% was a critical threshold, and we need to stop accumulating debt based on their research, I am at a loss to understand why you think that they can do this BASED on their research. Based on their research, they can say: “Certainly, you can read it that way, Senator–but to be fair, the 90% is not an actual threshold that we should see as a hard and fast rule. More importantly, it is absolutely possible that causation goes exactly the other way–economies that are doing poorly accumulate debt. The other side of the debate could be right about that. Our collective gut feeling is that debt is bad, and we need to keep it under 90%. That is our personal opinion.”

        • ffwiley says:

          Cas,

          I agree that RR promoted 90%. This was based on both their reading of the research and their conceptualizing of the factors that link debt to growth. But your suggested language seems a bit OTT to me. Does your doctor add this many qualifiers to his recommendation to keep cholesterol below 200 (or substitute whatever lab readings you’ve been advised on), to reflect every source of imprecision in the link between health and cholesterol?

          Also, the quote doesn’t reference the means, medians or any of the conclusions of the 2010 paper. Nor can you be sure that it conveys the whole discussion. Have either Conrad or RR elaborated beyond this one account that clearly summarizes a longer discussion? What else was said? Was it the type of discussion that called for detail, or the type in which whomever RR were speaking with just wanted to know what they thought? I’ve presented research for many years, and most of the time your audience has little patience for a list of qualifiers – they just want your answer (unless you’re presenting to other researchers).

          Maybe there’s a recording out there somewhere of RR truly misrepresenting the statistical meaning of their numbers, but I haven’t heard it.

          • eigenman says:

            I don’t know if this is relevant, but the slope in question does make a difference in policy makers. If doubling your debt makes your GDP divide in half, then policy makers would have to be very wary compared to if it shaves off 2% off of your GDP. The answer of “Absolutely. Not acting moves the risk closer” is much more consistent with their original findings than “We need to reduce the debt eventually, but right now we have other very important problems that austerity can exaggerate.” Similarly, very few of the results in their paper suggested that “we need to move now” approach. The one that did, is the one that HAP is focusing on.

            BTW, thank you for such a wonderful blog. It’s hard to find insightful blog posts on such politicized topic, so I am very grateful that I found yours.

  5. I reached much the same conclusions in my analysis of this at Econbrowser

    • ffwiley says:

      Thanks James,
      I thought your article was excellent and makes some important points that I didn’t consider. If anyone missed it, I recommend checking out Econbrowser.

  6. perfectlyGoodInk says:

    What about the reverse causality issue? It is known that lower growth results in lower revenues and worsens debt positions. Furthermore, Arindrajit Dube points out that debt is much more highly correlated with past growth rates than future growth rates.

  7. Margaret says:

    The New Zealand figure, combined with the fact that it figured large in the RR work, requires some explanation.

    New Zealand in the 1950s was dominated by the wool industry. It represented 51% of our exports and while I can’t put my fingers on the numbers it was probably something like 20% of the GDP. This meant that the Korean War wool boom had a major impact. The prices received in 1951 have gone down in legend in New Zealand.

    I have obviously got a different GDP series than RR used as the numbers don’t match. For the record, my one is from here
    http://www.stats.govt.nz/browse_for_stats/economic_indicators/NationalAccounts/long-term-data-series.aspx

    It shows:
    1950 4.97
    1951 15.60
    1952 -5.50
    1953 -0.11
    1954 3.01

    So to pick out the negative number (what I presume was 1952 – it makes absolutely no sense for it to be 1951) as indicative of the growth of the period is to ignore the fact it was a correction from an unnatural and politically-driven high experienced in the previous year. To then use that number in isolation from its context, as an input into an analysis of the relationship between debt and growth seems to be — well, let us call it “strange”.

    • ffwiley says:

      This is interesting, Stats N.Z. does seem to contradict the whole discussion about the 1951 real growth figure. And I doubt this is explained by the negative growth in 1952 – RR’s debt/GDP data is above 90% in 1951 but not 1952.

      • Margaret says:

        Again I don’t know where they got their numbers from, but using the same source the ratio of debt/GDP (my numbers it is only government debt) are
        1950 101.9
        1951 83.0
        1952 77.9
        1953 76.7
        1954 73.1

        so it was definitely declining from a post-war high over this period.

        • perfectlyGoodInk says:

          Based on this site, the New Zealand numbers appear to be from Maddison/Statistics New Zealand (albeit with odd discrepancies).

        • ffwiley says:

          Margaret,
          In case you missed it, Bloomberg has more info on the N.Z. data discrepancy that you pointed out (it was linked on Marginal Revolution this afternoon). They didn’t rerun all the figures, but they called it “Advantage, RR,” faulting HAP for jumping to the conclusion that there wasn’t an explanation for the data “omissions.”

          http://www.bloomberg.com/news/2013-04-30/reinhart-and-rogoff-were-right-about-new-zealand.html

          • Klein in the Bloomberg piece and Reinhart on her website are both apparently wrong, and RR 2010, HAP and Maddison (on whom they are based) still seem to be correct.

            The discrepancy is due to the fact that the Maddison data is for calendar years while the Statistics NZ series is for fiscal years ending March 31 (personal communication Brian Easton, who compiled the data originally). So a StatsNZ 1952 entry is for April 1 1951 – March 31 1952 and thus lags by one year. Maddison’s is still the best calendar-year series and Reinhart is again wrong (besides having made transcription errors even with the Maddison data for 1946 and 1947 – see my post at http://silverberg-on-meltdown-economics.blogspot.co.at/2013/04/reinhart-rogoff-vs-new-zealand-1951.html).

            Since the NZ growth gyrations had absolutely nothing to do with the debt rate, this debate should be irrelevant to the general issue of debt vs. growth anyway, and highlights yet again how defective the RR methodology was.

          • ffwiley says:

            GS,

            Per my article, I haven’t defended the NZ data point and its effect on the results, but I find it odd that you would jump to the conclusion that RR made transcription errors on 1946-47. Contrary to your web post, RR do link to the Maddison data on your site (linking directly to the Groningen Growth and Dev. Center), and it matches the figures that you’ve attributed to RR (unless it’s been changed since my download some time ago). The discrepancy that you’ve pointed out may be exactly the uncertainty in the post-WWII data that RR pointed out and Klein seconded when he called out HAP for not accepting or even seeking out RR’s reason for not using the data. I’ve found the same problem with 1930s NZ data, which is all over the place, depending on the source. And RR’s 1950-51 start date for this particular calc happens to be consistent with standard data sets that begin at that time for GDP (the IMF as reported on RR’s site and Maddison’s for many countries).

            RR didn’t make a transcription error. Why would you announce on your site and here that it was an error without even looking for other explanations? You should post a correction and acknowledge that you got it wrong.

  8. john byrne says:

    “And the most amazing thing about the past week may be how many people became instant experts on exactly how RR described their research to policymakers all over the world. I must have been the only one who missed the nightly Reinhart and Rogoff Hour on national tv. Since I follow their work and use their data more than most, it seems odd that I was the guy left out of the loop.”

    the conditional clause that introduces the last sentence led me to discount your preceding critique/’analysis” as anything but unbiased .

  9. Maurice Schiff says:

    Below the first figure, you mention the difference in growth rates between HAP and RR when Debt/GDP > 90% is 0.3%, which would be the case if the comparison was between 0.2% and -0.1%. However, the HAP growth rate is NOT 0.2% but 2.2%. Hence, the difference is not 0.3% but 2.3%. Though you defended some of RR’s errors, I wonder how you would defend yours, and how you would amend your article in light of this error.

    • ffwiley says:

      MS,
      I certainly make errors, but this isn’t one of them. Your 2.2% calculation is based on all of HAP’s calculations – it doesn’t isolate the Excel error. I’ve lifted the 0.3% figure directly from their paper (page 7) and you’ll find the same figure in the more honest accounts of the debate.

  10. Kenneth Stanley says:

    I’ve been following this discussion from the perspective of a non-economist.

    The numbers and methods being debated seem to be close enough that any little tweak in the assumptions will yield a slight difference in the measured outcome.

    So it seems fair to say that: “RR found that debt levels above 90% result in a large reduction in GDP growth. But upon review, and looking at the numbers differently, HAP finds that the reduction in GDP growth is not as bad as RR initially found”

    Fair enough??

  11. NapaDenny says:

    Also as a non-economist, I find it interesting that a 1 point difference in GDP is considered trivial. If lower debt economy A grows at 3 % and Higher debt economy B grows at 2%, then economy A has grown 50% faster than B. over time that is a huge difference.

  12. smr says:

    Your graph looks very nice, but is misleading: the Y-axis label says “average GDP growth”, but you plot RR’s median results. There is a clear difference between median and mean (as I gather you know very well). It would have been better had you redrawn Figure 3 of Herndon et al. in a neat figure, and published that one. That Figure is much more informative than the one you show here, since it allows the reader to assess that the numbers presented in the ‘stylized’ representations have a large spread, and that there is no clear threshold at the level of 90% debt/GDP ratio. +10 for HAP for enabling inquisitive people to make up their own mind.

    • ffwiley says:

      smr,
      I removed the axis label, thank you for pointing this out. I agree HAP’s Fig. 3 is a helpful chart, and that I could have added an extra positive score for their graphing (also Figures 1 & 2).

  13. jorge says:

    You say that RR don’t claim causation. FYI, here’s a direct quote from the 2010 paper: “The nonlinear response of growth to debt as debt grows towards historical boundaries is reminiscent of the “debt intolerance” phenomenon developed in Reinhart, Rogoff, and Savastano (2003). As countries hit debt tolerance ceilings, market interest rates can begin to rise quite suddenly, forcing painful adjustment.”

    • ffwiley says:

      To clarify, what I actually wrote is that RR did suggest causality (paragraph 4 – “they suggested that economic growth tends to slow after debt rises above 90%”), based on the obvious conceptual relationships that fair-minded people have long acknowledged (paragraph 5 – “truly believe … on a conceptual basis”), but they don’t confuse that causation with purely statistical inferences (presumably the paragraph that you’ve referenced, which said that they didn’t “neglect to recognize that correlation isn’t causation”).

      You’ll actually find the kind of language that you highlighted in thousands (millions?) of papers that don’t really “prove” causality. Believe it or not, this is commonly done and it’s only further evidence that this is a witch hunt. A handful of intellectually dishonest pundits are whipping up a whole stew of misconceptions about “right” and “wrong” in economic research.

      One of the appeals of RR’s work is that it’s not full of statistical inferences, because these inferences are always inherently wrong in economics. Whenever an economist makes a purely statistical inference (using the “parametric statistics” that dominate research), she’s assuming that some characteristic of the economy doesn’t change. In the real world, you won’t find such a thing. RR stripped away these pretenses – they just built a database, shared it with everyone else, reported a few statistics, and then told us what they thought these statistics meant and why.

      • John Mauldin suggested I read your analysis. You have done a good job.

        I agree that RR have created a great data base but unfortunately they have fallen short on utilizing it. The attempt to use macro averages or medians was probably a serious mistake in approach. Analysis of each data base entry individually and then classifying the cases into a number of categories based on common characteristics would probably have been much more productive (see below).

        As one who has fallen victim to making too broad generalizations from large data sets I can identify with the situation RR now find themselves in. It is also a shortcoming of HAP because they have focused on trying to replicate the RR analysis process.

        The alternate approach mentioned above has been proposed by Arindjarit Dube (http://www.nextnewdeal.net/rortybomb/guest-post-reinhartrogoff-and-growth-time-debt) and has been investigated by two economics graduate students (no relation to HAP). See http://econintersect.com/wordpress/?p=35802.

        I think the great database compiled by RR may yet find a renewed life with alternative uses beyond the original RR work and the replication assessment of the RR methodologies by HAP.

        If we are lucky the useful and productive life of the RR monumental data compilation may be just beginning.

        • ffwiley says:

          JL,

          I’m a huge fan of simple stats and charts – e.g. the medians in RR’s work, scatter plots in HAP – especially when the data’s available online and I can check it out myself. The reason being that statistical relationships are never constant, as you know. They vary from time to time and place to place, so I try to be careful not to read too much into the data. You’ll see how I approach data if you have a chance to read “Answering the Most Important Question in Today’s Economy” – all that I did was to investigate every instance of debt-to-GDP > 150% and report on the outcomes, with some interpretation along the way.

          That said, I had a quick look at your link last week and will look more closely in time, even though I’m highly skeptical that Dube (who is a colleague of HAP) has rendered means and medians obsolete with some regressions and a distributed lag model. Will also spend some time on your site – I can see you have strong content and an impressive list of contributors.

          I appreciate your comment – thank you for reading and weighing in.

  14. Unanimous says:

    A few points:

    “over 90%” is a silly category to use to reach conclusions about 90%. It inlcudes debt levels way different to 90% including 150% and 200%. It is not a category that should be compared with 0-30%, and 30-60% because those categories are bounded while the over 90% category isn’t. 90-120% is a category that can be used as a comparison, and as HAP showed via an F-Test there is no statistical significance to that category.

    An analysis that does not depend on categorising into a small number of buckets is needed if you want to look for critical levels or reach any conclusions about critical levels, otherwise you will end up picking a limit of one of your buckets as a critical level. 90% is more a side effect of the buckets than a result present in the data. It could just as easily have been 80% or 100% if they chose different buckets.

    The underlying data is scattered, and there is little significance to any relationship. There are many examples of debt higher than 90%, and higher than 120%, and even one over 200% that didn’t result in any catastrophy (and we are still waiting on Japan, but there are demographic problems there). Australia after WWII is a good example of a long period of high growth and unemployment under 2% in the presence of paying down 160% government debt, but you exclude that because, well, WWII, yeah that’s it, WWII.

    The fact that examples get excluded because of some particular properties of their debt or economy rather than suspected failings in gathering the data should be taken as an acknowledgement that debt itself is not a problem.

    History may have happened to included slightly more badly performing governments and economies that did also have high debt than well performing ones with high debt, but so what. You follow the good examples and not the bad ones, and then after a while history will end up with different examples in it.

    The surprising thing is that it requires a lot of complicated analysis and judgement calls before people will feel comfortable reporting a result. This is an indicator that there probably isn’t a clear realtionship. If there was a clear relationship, you would simply be able to plot some points and the relationship would stand out, but that’s not the case. So we’ve got a difficult to find relationship, which even if it did exist would relate only to the past, and would be useless for predicting future outcomes.

    To reach any conclusions on debt as an indicator of future outcomes from this data is plainly wrong and deceptive. The only safe thing that can be said is that there is very little, if any, relationship between government debt and subsequent performance of an economy. But then you wouldn’t have much of a paper to publish, let alone one that gets cited and publicised.

    The whole thing is a good example of how the academic system produces rubbish under a cloak of rationality. There is good work too in the academic system, but more rubbish than not. It’s also a good example of rubbish being picked up by particular politicians because it supports the message they want to convey.

  15. I am an investor, not a politician, economist, pundit.. or some other manipulator of the facts. As an investor, I am acutely aware of the variability of future outcomes which derive from present decisions, and the humility that in turn derives from that experience. After all, “experience” is what we get when we don’t get what we want, and loss of capital can be a particularly painful “experience”. With this in mind, I am much more concerned about directional trends than the accuracy of those trends. If I can commit 10-year horizon capital with a reasonable sense of confidence that a given exit environment will prevail, then the information driving that confidence has high value. If, on the other hand, the information that I gather leads me to uncertainty, my capital remains inert. This does not argue for an increase participation in the public debate by persuasive charlatans. Far from it. But this argument, combined with the obvious biases reflected by the various commentators leaves me cold, and proves my thesis that economists and academics tend to make lousy investors. The one notable exception to this thesis is the protagonist in J.K. Galbraith’s classic novella “The Tenured Professor”. And we all know what happened to him.

    Although I find this debate of high entertainment value, it is of virtually zero consequence relative to the discipline of capital allocation. Why? Because both parties argue essentially the same thing: high debt is associated with periods of low growth. Correlation not proving causation, I don’t really care from which direction that association comes. It is enough, for me, to know that the association exists, and for me to add that factor when considering making investment decisions.

    However, I want to congratulate this blog site for stripping – to whatever extent doing so is possible – the discussion herein of political bias. It is tedious beyond words to read the exaggerations, slander, character and intellect assassinations, and posturing for political points on issues that have real weight. With a nod to the poster who recognized that a 1% difference in GDP growth may seem insignificant in nominal terms, it can have a massive difference in application… particularly at a time of persistent low GDP growth and crushing unemployment. If blogs like this can contribute to a rational debate on matters such as this, even if I may disagree with the conclusions, I applaud the effort.

  16. Mark Fisher says:

    Academic study has always been about explaining our intuition about events that are pushed and pulled by any number of outside agencies. The paper by RR and their subsequent book sought to provide a workable “benchmark” against which economic policy could be measured. Benchmarks are by their very nature imprecise and intended for use in making the preliminary “rough” cuts. They are not intended to be measuring sticks capable of atomic molecule precision.

    While some may have misused their findings to justify public policy shifts by citing it as an absolutist measure, I never did. I merely read it as being support for my intuitive view that misuse of any tool, and debt is one, can produce poor results.

    The problems we face are extreme and continued deficit spending cannot be the solution. While I have seen no proposed exit strategy that does not produce poor economic outcomes for our nation, this does not mean that we continue to add more poison to the system just to avoid the immediate hangover that will likely follow. Intuitively we all know that this is merely postponement and that it is unlikely that the delay will decrease the pain. Rather it will likely make it more severe and disruptive than a measured policy that meaningfully stops the insanity.

  17. The discrepancies I mentioned between RR’s growth rates and the Maddison growth rates as provided by StatisticsNZ do exist as I represented them, for 1946 and 1947. However, you are right that RR’s rates correspond to those from Maddison’s original database as available from http://www.ggdc.net/maddison/Historical_Statistics/vertical-file_02-2010.xls

    In the blog I state quite explicitly that I am using the “Maddison” real GDP data downloaded from StatisticsNZ, Why their “Maddison” values for 1945-7 differ from Maddison’s original ones is an interesting question: transcription error on their part or data revision?

    In any event I have made a postedit annotation to my post to this effect. So RR are not guilty of a transcription error. Why they now claim the original Maddison data they used in 2010 are false, however, is not at all clear, since all evidence does point to the fact that a (at least nominal) GDP recession did occur in NZ in 1951, not 1952, due either to the waterfront lockout/strike, falling wool prices, or both.

Comments are closed.