Posted by I found the hype surrounding the Reinhart-Rogoff ‘error’ really confusing — mostly due to the fact that there was no break-down of the differences between the HAP and R+R results.
The HAP paper has the breakdown in a table at the back.
So what’s the low down? There’s basically nothing in it — it’s a beat up. An excel error lowered high debt growth by 30bps, and boosted low debt growth by 10bps to 20bps (see p7).
“This spreadsheet error, compounded with other errors, is responsible for a -0.3 percentage- point error in RR’s published average real GDP growth in the highest public debt/GDP category. It also overstates growth in the lowest public debt/GDP category (0 to 30 percent) by +0.1 percentage point and understates growth in the second public debt/GDP category (30 to 60 percent) by +0.2 percentage point.”
Correcting a transcription added another 10bps to ‘high debt’ growth. Correcting these two errors yields a low debt-high debt growth spread of 370bps, down from 420bps in RR AER 2010.
The other ‘differences’ amount to a difference of opinion about the best summary measure. RR say that it’s okay to take the average of averages, and HAP say that’s odd, and not allowed.
The WSJ has a handy table that allows you to see right away that it basically comes down to what you think the best measure of central tendency is — mean or median.
What all studies agrees is that high debt countries grow more slowly than low debt countries. The difference in the median estimates (and the non averaged means in HAP) all tell the same story — more debt means less growth (probably because it means more taxes and intrusive regulation — the productivity ‘miracle’ is 10bps per month, a gain that’s easily crushed by economic and political noise).
ricardian ambivalence 
This will be in tomorrow in a special Rogoff & Reinhart section of Around the Traps.
you are in a few categories.
psst don’t agree with your stuff though
Which bit do your disagree with?
It’s very fair minded of you to link to it anyways. Thanks :)
It is out now. The special Rogoff&Reinhart edition of Around the Traps featuring our own RA with plenty of others.
I favour Andrew Gelman’s explanation
Just read that and he doesn’t actually say anything about the magnitude or importance of the error – just that they should have been more professional in admitting it . Ricardo was commenting on the first issue, not the second. Academics generally aren’t inclined to be gracious, especially the ones who’ve built up a public reputation. As a layperson, the climate science area seems to me to be the most egregious for unwillingness to concede error.
Thanks for cutting through, Ricardo. I’ve never read R&R, although I wish I had when it came out. My sense is that high debt is more likely to reflect bad policy decisions (eg excessive spending on transfers or politically-motivated projects), which is what causes sub-par performance, rather than being a direct cause of bad economic outcomes per se.
I had the feeling Gelman was trapped between his politics and his sympathy with the difficulty of publishing error-free technical papers.
The ‘they should have read keynes’ meme of others is annoying me — the general theory is a polemic, not economic theory (Hicks added that). The contribution of R+R in assembling the data-set is substantial, and commands respect.
Everything is measured with error – folks ought to prefer medians as they are more robust. RnR presented both. The HAP result is consistent with their medians. To be honest, given the obvious skew (mean so far below median) i am surprised they didn’t go looking for errors themselves.
No worries mate. I wrote the blog as i was very confused about what each of the errors were worth – the way some commentary is written, you’d think that the excel error was the whole difference, when it is in fact trivial. Glad you benefited from my toil.
Whether or not it changed the substance of their claims is irrelevant don’t you think. The type of analysis in question makes many tiny assumptions at many different points that can sway the results to fit ones own perceptions / bias. What this whole episode demonstrates is the academic dishonesty of the two authors involved. It also reflects in my opinion that the two had a preconceived notion of what they would find and thus ‘tinkered’ with the data to better fit their ‘theory’. They didn’t question their own findings because they had no need to…it was exactly what they thought they would find.
Economics, especially marco, is in no way rigorous or scientific . It is used by people to make a political point, in a lot of ways its a more intellectual form of opinion writing…intellectual in that you have to have the smarts to be able to use statistics to mask your own biases.
I think the fact that they published the medians in RR 2010 and that RRR JEP 2012 had similar means to HAP suggests that this was probably an honest mistake.
The robust result across the three papers is that more debt means less growth. We agree on the sign of the parameter. Now we are arguing about size. What are the std errors? Are these arguments about results that the statistically very distinct?
Probably not. This is now a political bun fight.
My argument is more of a general one, that is the profession of economics isn’t what it is portrayed to be, a ‘scientific’ approach to explaining economic trends. It is basically a political tool, there is no truth in economics as there is in real sciences. Instead it is a tool used in battles between competing dogmas. The present argument is a great example of this. There are other schools of thought that have equally convincing arguments about the perils of austerity and would discredit the RR2010 and RRR JEP 2012 for ignoring a litany theories about blah. The political bun fight comes from the fact neither side can absolutely prove the other to be incorrect, so when academics make honest or deliberately ignorant mistakes, it gives fuel to the other side to whack them over the head with it.
We as a society would be better off if economists arguments were treated for what they actually were, considered opinions and not gospel truths.
If this is an argument for more ‘open’ science, i agree.
It is, but I don’t see it happening. The field is too set in its ways and this latest episode is just a classic example of all the failings of economics as a field. If this had been a hard science the careers of the two authors would be over (regardless of the implications to the results). Instead its just viewed as an honest mistake.
I think you would agree it was sloppy and unprofessional. The sad thing is that this still happened after a peer review process. Whats worse is that is that this will al wash over in a few months and the same environment will continue.
I disagree, it can happen anywhere — rememberthe faster-than-light neutrinos?
agree that the same thing happens in hard sciences but the difference is that it normally encounters scepticism straight away which was the case for the faster than light neutrinos.
The other difference is that science comes from the point of disproving proposed theories, whereas economists look to prove their theories by making data fit. This is a fundamental difference in approach and something I don’t think is easily changed.
I’m not sure the hard sciences are any different. Kuhn didn’t have economics in mind when he wrote The Structure of Scientific Revolutions. In recent times, the apparent ad hocism of climate scientists has disabused me of any remaining idealism about how hard scientists conduct themselves.
Having worked in both scientific and economic areas I have to say the two fields are worlds apart. I’m not saying that the hard sciences are faultless. I’m saying that economics as a field lacks rigour in its approach at a fundamental level.
As for climate science not being rigorous well not having worked in the area I can only guess that this is because it is a new field and its politically charged. Also the field of climate science is made up of many different fields including economists, which may explain why it may not be as rigorous as the traditional sciences.
I guess it depends on what you mean by ‘rigour’. If you mean predictive power, then it’s certainly true that physics can predict planetary motions better than macroeconomists can predict GDP. But macroeconomics can predict GDP better than geologists can predict earthquakes. Comparing the predictive success of sciences or disciplines if you like only makes sense if they are trying to predict the same phenomenon.
I’m not talking about predictive power. Science is not about prediction its about understanding, prediction is a byproduct of understanding. This is exactly my point when it comes to Science and economics.
Rigour from a scientific perspective is about objectivity. In science there are no positive results because the central premise is of science is to disprove a concept / theory (not prove it). If the results do not disprove it then it is not proven, there is simply not enough evidence to say that it is false.
Economics as it currently stands approaches the problem in the opposite fashion. Most economists have a point of view and they meld data to fit this view. This may be consciously or sub-conciously but the point is that most economists are not objective and don’t truly try to disprove their theories. Whats more is that most economists aren’t even aware of this fallibility in their approach.
I agree with Katy, but not because economists per se are not rigorous or not objective, but because “Economy” is mainly a tentative description of human behavior which is still (luckily!) very mysterious in many ways. For instance, economist should talk about people culture and history, their fear and greed, but they never do. As if one economic model fits all.
1 + 1 + feelings = ?
Even the very goal of economic systems are unclear. Is a “good economic system” about making everyone live longer? Healthier? With more food? Owning more things? Or is it about having more friends? Being happier? A safer environment? Feeling included? Sometimes we say we live in a wonderful country, GDP is growing at % every year, then we look at our self and ask: are we happy? You travel to one of those “very troubled economy” and you see they seem to be having much more fun than we are.
Take this public debt example. Well, it would depends on who is managing that debt and how, how it’s invested, etc. as if 90% would be that magic number, it just does not make sense before you even start your calculations. So you fit a very limited set of numbers in your magic spreadsheet, you come out of with “a result” and magically make it a UNIVERSAL conclusion. Sure. :) Then you go with your nice spreadsheet to Greece and say: you do not need austerity, you need to spend, spend, spend. However, public money has been spent and wasted by politicians for years and years, and we want them to waste more?
It’s interesting that you take that view because I would have thought that most hard scientists would say that science is about prediction and hence economics is not much of a science because it doesn’t predict as well as physics. I personally think predictive power is the key to science; its just that i think economics does a reasonable job at predicting compared to other social sciences. Certainly, Friedman thought predictive power was very important. By comparison, I’m not sure what understanding can be had without predictive success. That sounds more like a humanities way of thinking.
I don’t agree that hard scientists are any more pure or objective than academic economists. Wherever there is ego and prestige at stake, human beings will try to make themselves look good.
@Rajat
While I agree predictive power is a very useful outcome, it is derived from first understanding the system / subject matter you are studying. This is my point, macro economists, generally, apply a model to predict, and assume in the construction of these models that they understand all the interactions fully or adequately enough to provide a powerful prediction / explanation. But we know that most macro economic models are not good at this and very very rarely provide good predictive power. If instead of focusing on predicting an outcome, economists focussed on first better understanding the systems they were studying they might be able to construct better models / theories.
Friedman would think predictive power was important, he was an economist / statistician and thus that was his entire focus, which is exactly what I think is wrong with economics. Einstein, Mendel, Pasteur, Fleming, Watson and Crick certainly didn’t see prediction as the focus of their science, it was about understanding. If the benefit of prediction followed all the better but it wasn’t the focus.
Yes I agree with you that scientists are probably not more objective than economists, we are all human. But the difference is the environment in which the two operate in is very different. Economics is not a science and maybe it shouldn’t be viewed that way because of the nature of the beast that is being studied. But when economists ‘opinions’ are treated as ‘scientific’ fact well thats a problem.
If Einstein couldn’t predict better than Newton, I doubt many people would know his name today. It goes without saying that economists do try to understand all sorts of relationships; it’s just that these relationships are extremely complicated so properly ‘understanding’ the entire economy is very hard. Just like predicting earthquakes or climate.
I think economists is a science in that many/most/some(?) economists try to make falsifiable predictions about human behaviour. I think economists is superior to sociology in this regard. I also think it misses the point to suggest that a discipline produces either ‘opinions’ or ‘facts’. R&R produced a study showing that high debt was associated with slower growth. They made some mistakes, but the result more-or-less holds according to Ricardo. You suggest this is all just their opinion. But it’s not, it’s more than that. How do you treat a published study showing a link between, say, olive oil consumption and a low incidence of heart disease? Opinion or fact? Science doesn’t work in this way.
Even if you say social sciences are not science, people still want to understand social and economic phenomena. So how best to understand – adopt a scientific approach, as economics seeks to do. And you’re back to where we started.
Yes understanding complex systems is hard, but just because it’s hard doesn’t mean that over-simplifying it in order to make over-generalized statements about relationships is right.
This is exactly my point macroeconomics over-simplifies very complex phenomena. For instance RRs paper on debt and growth suggests that growth and debt are inextricably linked. This is the problem with economics i.e. Popper: “…no matter how many instances of white swans we may have observed, this does not justify the conclusion that all swans are white.”. RR’s paper is right according to all the little assumptions and adjustments they made during their analysis. This does not change the fact that 10 different economists could perform similar analysis 10 different ways and come up with 10 different conclusions. See my point?
The problem I have is that RR made mistakes (minor or otherwise) in their analysis and it wasn’t immediately obvious. The essence of scientific research is that anyone anywhere can repeat exactly what you did and come up with the same results, i.e. the results are reproducible. RR’s methods could not be, in part because they hid parts of their analysis and or did not adequately explain their methodology, not scientific. I think this reflects the difference between science and economics. Science relies on results being reproducible and in the fields I’ve worked in you make dam sure that your results hold up because you know as soon as you publish people will try and reproduce it. My experience of economics is that reproducibility is not priority number one, but proving your ‘opinion’ is right is.
Anyway I don’t think I’m going to convince you otherwise so lets just agree to disagree.
I am not sure where Katy is going with this – is it an appeal against reductionist economics (the models are unrealistic?) or against statistics in general? I think we have learnt a lot about the economy over the past 50 years thanks to both ‘tools’.
R+R have done their part. They put together a data set covering 200 years, and made it generally available. That’s a huge contribution. I know of only three papers using the data, and in all of them the sign of the co-efficient on debt is negative (more debt = less growth).
We might have asked them to make their data available earlier, and to be more transparent in their statistical method, but they have made a huge contribution to the field.
If you disagree with their method, you can check your logic against the data yourself. Anyone can do so. That’s the gold standard. Making data and method public is a good reason to push everyone toward R (or some other open source platform for reproducible analysis).
The next step is understanding why the sign is negative.
This present R+R debate seems to me to be mostly politically motivated – many of the arguments are from those who were anti-austerity to begin with.
Sure, be anti-austerity if you like — but if you think Keynes 1936 is more robust than R+R you are kidding yourself.
Probably not explaining myself clearly but basically yes I think economics is too reductionist and economists don’t admit to the shortcomings of economic analyses clearly enough. Instead findings that should be delivered with significant caveats are delivered as if they are gospel truths that can be applied in all circumstances. It would be great if economists clearly pointed out the weakness of analysis and were more honest about the shortcomings of the data they use. For instance, does the R+R paper clearly explain the weaknesses of the data set they constructed? Do they clearly acknowledge and point out the sources of error in the data set? I don’t think they do.
There is also the question of relevance but thats probably a different debate.
cheers
Hamilton has postedon topic at econbrowser.
Actually RA I think you are in a ‘cochrane’ mood.
to me Konzal and then his guest post nail it. Gelman says it all in R&R attempted reply which is ( highly euphemistic) disappointing.
The results change.
Bad RA
Well, i do like Cochrane a lot.
My view is that everything is measured with error, and that basically the story remains the same as all the the signs in all the studies are the same. I expect that as data and nations are added that the point estimates will change — and probably by a lot.
The excel error and transcription error are embarrassing, the other method stuff is neither here nor there. Their average-average approach isn’t all that novel. If they didn’t do it, other criticisms would be valid (like ‘you are only measuring Japan and Greece’).
If you had an issue with it, you’d look at the medians. They are generally agreed to be more robust measures of central tendency.