103% of Spreadsheets have Errors

The appearance of the words "spreadsheet" and "comedian" together in political headlines suggests something is changing about the way policy gets created.  We welcome an increasing role for data in the debate -- if it is used well.  Unfortunately, right now, that seems to be a big "if."

For more than a month, politicians, economists and other macroeconomic policy-talkers (which is not to say policy-writers) have been batting back and forth a three-year old research paper so brief that even with page breaks, generous margins, seven charts and a couple of data tables it is barely two dozen pages long.  The research had previously served as a key reference for many deficit hawks until a grad student brought to light a series of errors in the work that indicated not only a seeming lack of rigor by the authors but also a systemic sloppiness which enabled the paper's questionable "insights" to influence both press and politicians.

Most of the Twitter-level attention has understandably been directed to the easiest to comprehend (and most snicker-worthy) of the paper's mistakes: an Excel formula which failed to reference all of the applicable cells.  While we don't know exactly what the original spreadsheet looked like, there is no question that the re-creation of the math makes the error seem obvious.  (You can see that graphic and read a summary of the critique or read the original in its entirety,  And soon we were treated to articles about the prevalence of spreadsheet errors (including one which cited a study that suggested that 88% of spreadsheets have errors -- although you have to wonder who checked the spreadsheet about the spreadsheets).  In an era when more data is being both captured and used by and on all of us, clearly even small spreadsheet errors are going to have big impacts.

As a variety of commentators have pointed out, several other errors and potential errors exist in the austerity study.  The authors used unconventional methods for weighting the data (and failed to explain why).  They selectively excluded some of the data (and failed to explain that, too).  They provided no compelling reason for the initial parameters they used for the study (and have subsequently gone back to work with a larger data set.  Perhaps they simply should have revisited the decision to publish altogether.  As Matthew O'Brien suggests, the boring reality is that the relationship between public debt and growth isn't clear, and trying to say anything dispositive about debt and growth more broadly is near-impossible because there simply isn't enough data.

In short, the Excel error may be the most widely-discussed weakness of the paper, but it is by far the least important in the long run (and, as the math turns out, least impactful as well).  Sure, you need proper analytical execution.  But you need the right approach in front of that.  And the right parameters in front of that.  Most importantly, however, you need to ensure you're working on a problem that can be solved with data in the first place.  Yes, many policies can and should be guided by thoughtful data analysis.  But the most important skill for using any tool is an awareness of its proper uses.m And not all problems can be solved with spreadsheets -- even error-free ones.