Outside the Beltway

Interpretation, Art, and Analysis

The analyst actually wants to understand and be correct far more than he or she wants their preferences to prevail in the analysis

Steven L. Taylor · Saturday, November 3, 2012 · 11 comments

Jay Cost is supposed to be the Weekly Standard‘s answer to Nate Silver, at least after a fashion.  Cost has training in political science (MA from Chicago) including stats and frequently writes about the polls.  He has been bullish on Romney for some time now.

Cost notes in his recent column in which he predicts a Romney win:

When I started making election predictions eight years ago, I had a very different perspective than I do today. I knew relatively little about the history of presidential elections or the geography of American politics. I had a good background in political science and statistics. So, unsurprisingly in retrospect, I focused on drawing confidence intervals from poll averages.

Since then, I have learned substantially more history, soured somewhat on political science as an academic discipline, and have become much more skeptical of public opinion polls. Both political science and the political polls too often imply a scientific precision that I no longer think actually exists in American politics. I have slowly learned that politics is a lot more art than science than I once believed.

Accordingly, what follows is a prediction based on my interpretation of the lay of the land. I know others see it differently–and they could very well be right, and I could be wrong.

Now, I agree that polling is often presented as if it were perfect, which is the wrong way to look at it without a doubt.  However, I think that when people start falling back on “art” and “interpretation” then one is often (although not always, I will grant) on shaky ground.  Still, I would note that not all systematic arguments require quantitative analysis and therefore just because a number isn’t based on numbers doesn’t mean that one is ranging into art.  It is quite possible to make sound, rigorous, and systematic arguments (all of which are hallmarks of scientific inquiry) without having a quantitative measure undergirding the position.  As such, I see Cost’s range into “art” and “interpretation” to be less about any flaws of political science or statistics (which certainly exist) but, rather, about adopting the modes of punditry because the systematic approaches don’t produce his preferred outcome.

Amusingly (to me, at least) is that while Cost starts his piece talking about the limitations of polling, he predicates the first part of his argument on, well, polling:

Romney’s advantage on the economy. This to me is pretty straightforward. Take the recent NPR poll, which was a bipartisan survey conducted by Resurgent Republic and Democracy Corps. It found Obama’s job approval rating on the economy to be underwater, 47-52. The poll also found Mitt Romney to be more trusted on the economy over Obama, 50 to 46 percent.

Poll after poll, I generally see the same thing. Romney has an edge on the economy. That includes most of the state polls.

All well and good, I suppose.  However, the thing that is most striking to me here is that Cost’s poll skepticism and foray into the interpretative arts is not to eschew the polling, but rather to simply ignore the polls he doesn’t like (the ones that show Romney losing in the electoral college) while pointing to the ones he likes.  In other words, it would seem that “art” in this context means ignoring data one does not like, even if it is the decisive bit of information (like topline candidate v. candidate information), and then elevating some secondary bit of information because it fits one’s “interpretation of the lay of the land.”

He goes on to cite some more polling data on the economy to bolster his point.

The rest of the piece is about Cost’s interpretation of the Democratic Party coalition and specifically about independent voters (a good bit of which also is based on polling data about voter ID).  The analysis misses, however, the degree to which we ought to look at independents and the degree to which many of them are solid partisan leaners.  Instead we get a clichéd note that independents decide elections.  I guess that is the “art” part.

He concludes:

this is a different approach than the poll mavens will offer. They are taking data at face value, running simulations off it, and generating probability estimates. That is not what this is, and it should not be interpreted as such. I am not willing to take polls at face value anymore. I am more interested in connecting the polls to history and the long-run structure of American politics, and when I do that I see a Romney victory.

The thing is, he is right:  his approach is not like the “poll mavens” but it is standard partisan punditry, which is to say that one starts with one’s preferences and then finds information to back that preference.  I still do not see a reason to not take the data more or less at face value especially when we can look at multiple polls over time.  Indeed, aren’t elections one of those areas at which the “poll mavens” have general been quite competent over the last half century or so?

Ultimately, I must confess: it takes some serious pretzel logic to start a column about the limitations of polling and then dig to find polling data that you can tout as the foundation of your argument.

All of this reminds me of something I read this week that Conor Friedersdorf wrote in a piece entitled Why Nate Silver’s Gambling Streak Makes Me Trust Him More:

America is filled with people who think its okay to lie, bullshit, or otherwise misrepresent the truth in order to advance the electoral prospects of a politician or the cause of a governing coalition. Let’s call them shills. Other people aren’t necessarily aware that they’re misrepresenting the truth, but their work is so shaped by what would advance the causes of a candidate or governing coalition that it’s often indistinguishable from the shills. We’ll call them hacks. In a better world, journalists would be sworn enemies of shills and hacks, and the best are. Unfortunately, the press, especially the political press, has more than its share of shills and hacks.

Indeed all around (and emphasis mine).

Here’s the real problem I have with all of this, and the thing that separates the real analyst from the shills and the hacks:  the true analyst actually wants to understand and be correct far more than he or she wants their preferences to prevail in the analysis.

Perhaps I am being uncharitable to Cost, although perhaps not.  On the one hand, everyone is free to make whatever arguments they like, but on the other I have lost patience for “analysis” that is based on preference over evidence (and I don’t think that claiming one is simply taking a foray into art and interpretation is sufficient cover).

Indeed, that foray reminds me of the following comparison made by Scott Galupo at The American Conservative (Nate Silver and the Nerd-Haters):

When Michael Lewis’s baseball book Moneyball was published in 2003, ex-player-turned-commentator Joe Morgan angrily trashed it. When the movie adaptation came out in 2011, manager Tony La Russa claimed to have been “offended” by it. The old-guard recoil from sabermetrics ultimately boils down to sentiment: You can’t quantify intangibles like hustle and pluck — the “human element.” You can’t account for sheer luck.

Sabermetricians, of course, do not disagree. The “search for objective knowledge about baseball,” as Bill James put it, does not downplay or discount the human element. On the contrary, it tries to isolate with ever-more precision the things we can know about players so that we can recognize luck when we see it— and then make meaningful, though hardly infallible, predictions about future performance.

In the last week or so, sabermetrician-turned-polling-analyst Nate Silver has had to endure the political universe’s equivalent of the Joe Morgan/Tony La Russa beatdown.

To me, claiming to go “interpretative” in the face of actual data is do what the critics of sabermatrics did: claim that the gut knows better than the data.  However, as Galupo notes: “I’ll take Silver’s transparent model over Scarborough’s hunch-level insider bluster any day of the week.”*

Let me conclude by noting that my critique is about method, not about outcomes.  It is entirely possible that Romney will win next week, as Cost believes.  The problem is that his analysis is driven by his belief, not by the evidence (which describes a lot of “analysis” that I read these days).

*Galupo was writing in regards to a comment by Joe Scarborough, amongst others.  However, the key my usage of the quote if the reference to “hunch-level bluster.”