Interpretation, Art, and Analysis
The analyst actually wants to understand and be correct far more than he or she wants their preferences to prevail in the analysis
Jay Cost is supposed to be the Weekly Standard‘s answer to Nate Silver, at least after a fashion. Cost has training in political science (MA from Chicago) including stats and frequently writes about the polls. He has been bullish on Romney for some time now.
Cost notes in his recent column in which he predicts a Romney win:
When I started making election predictions eight years ago, I had a very different perspective than I do today. I knew relatively little about the history of presidential elections or the geography of American politics. I had a good background in political science and statistics. So, unsurprisingly in retrospect, I focused on drawing confidence intervals from poll averages.
Since then, I have learned substantially more history, soured somewhat on political science as an academic discipline, and have become much more skeptical of public opinion polls. Both political science and the political polls too often imply a scientific precision that I no longer think actually exists in American politics. I have slowly learned that politics is a lot more art than science than I once believed.
Accordingly, what follows is a prediction based on my interpretation of the lay of the land. I know others see it differently–and they could very well be right, and I could be wrong.
Now, I agree that polling is often presented as if it were perfect, which is the wrong way to look at it without a doubt. However, I think that when people start falling back on “art” and “interpretation” then one is often (although not always, I will grant) on shaky ground. Still, I would note that not all systematic arguments require quantitative analysis and therefore just because a number isn’t based on numbers doesn’t mean that one is ranging into art. It is quite possible to make sound, rigorous, and systematic arguments (all of which are hallmarks of scientific inquiry) without having a quantitative measure undergirding the position. As such, I see Cost’s range into “art” and “interpretation” to be less about any flaws of political science or statistics (which certainly exist) but, rather, about adopting the modes of punditry because the systematic approaches don’t produce his preferred outcome.
Amusingly (to me, at least) is that while Cost starts his piece talking about the limitations of polling, he predicates the first part of his argument on, well, polling:
Romney’s advantage on the economy. This to me is pretty straightforward. Take the recent NPR poll, which was a bipartisan survey conducted by Resurgent Republic and Democracy Corps. It found Obama’s job approval rating on the economy to be underwater, 47-52. The poll also found Mitt Romney to be more trusted on the economy over Obama, 50 to 46 percent.
Poll after poll, I generally see the same thing. Romney has an edge on the economy. That includes most of the state polls.
All well and good, I suppose. However, the thing that is most striking to me here is that Cost’s poll skepticism and foray into the interpretative arts is not to eschew the polling, but rather to simply ignore the polls he doesn’t like (the ones that show Romney losing in the electoral college) while pointing to the ones he likes. In other words, it would seem that “art” in this context means ignoring data one does not like, even if it is the decisive bit of information (like topline candidate v. candidate information), and then elevating some secondary bit of information because it fits one’s “interpretation of the lay of the land.”
He goes on to cite some more polling data on the economy to bolster his point.
The rest of the piece is about Cost’s interpretation of the Democratic Party coalition and specifically about independent voters (a good bit of which also is based on polling data about voter ID). The analysis misses, however, the degree to which we ought to look at independents and the degree to which many of them are solid partisan leaners. Instead we get a clichéd note that independents decide elections. I guess that is the “art” part.
this is a different approach than the poll mavens will offer. They are taking data at face value, running simulations off it, and generating probability estimates. That is not what this is, and it should not be interpreted as such. I am not willing to take polls at face value anymore. I am more interested in connecting the polls to history and the long-run structure of American politics, and when I do that I see a Romney victory.
The thing is, he is right: his approach is not like the “poll mavens” but it is standard partisan punditry, which is to say that one starts with one’s preferences and then finds information to back that preference. I still do not see a reason to not take the data more or less at face value especially when we can look at multiple polls over time. Indeed, aren’t elections one of those areas at which the “poll mavens” have general been quite competent over the last half century or so?
Ultimately, I must confess: it takes some serious pretzel logic to start a column about the limitations of polling and then dig to find polling data that you can tout as the foundation of your argument.
All of this reminds me of something I read this week that Conor Friedersdorf wrote in a piece entitled Why Nate Silver’s Gambling Streak Makes Me Trust Him More:
America is filled with people who think its okay to lie, bullshit, or otherwise misrepresent the truth in order to advance the electoral prospects of a politician or the cause of a governing coalition. Let’s call them shills. Other people aren’t necessarily aware that they’re misrepresenting the truth, but their work is so shaped by what would advance the causes of a candidate or governing coalition that it’s often indistinguishable from the shills. We’ll call them hacks. In a better world, journalists would be sworn enemies of shills and hacks, and the best are. Unfortunately, the press, especially the political press, has more than its share of shills and hacks.
Indeed all around (and emphasis mine).
Here’s the real problem I have with all of this, and the thing that separates the real analyst from the shills and the hacks: the true analyst actually wants to understand and be correct far more than he or she wants their preferences to prevail in the analysis.
Perhaps I am being uncharitable to Cost, although perhaps not. On the one hand, everyone is free to make whatever arguments they like, but on the other I have lost patience for “analysis” that is based on preference over evidence (and I don’t think that claiming one is simply taking a foray into art and interpretation is sufficient cover).
Indeed, that foray reminds me of the following comparison made by Scott Galupo at The American Conservative (Nate Silver and the Nerd-Haters):
When Michael Lewis’s baseball book Moneyball was published in 2003, ex-player-turned-commentator Joe Morgan angrily trashed it. When the movie adaptation came out in 2011, manager Tony La Russa claimed to have been “offended” by it. The old-guard recoil from sabermetrics ultimately boils down to sentiment: You can’t quantify intangibles like hustle and pluck — the “human element.” You can’t account for sheer luck.
Sabermetricians, of course, do not disagree. The “search for objective knowledge about baseball,” as Bill James put it, does not downplay or discount the human element. On the contrary, it tries to isolate with ever-more precision the things we can know about players so that we can recognize luck when we see it— and then make meaningful, though hardly infallible, predictions about future performance.
In the last week or so, sabermetrician-turned-polling-analyst Nate Silver has had to endure the political universe’s equivalent of the Joe Morgan/Tony La Russa beatdown.
To me, claiming to go “interpretative” in the face of actual data is do what the critics of sabermatrics did: claim that the gut knows better than the data. However, as Galupo notes: “I’ll take Silver’s transparent model over Scarborough’s hunch-level insider bluster any day of the week.”*
Let me conclude by noting that my critique is about method, not about outcomes. It is entirely possible that Romney will win next week, as Cost believes. The problem is that his analysis is driven by his belief, not by the evidence (which describes a lot of “analysis” that I read these days).
*Galupo was writing in regards to a comment by Joe Scarborough, amongst others. However, the key my usage of the quote if the reference to “hunch-level bluster.”
Here’s Jay Cost in 2010. There’s nothing wrong with the polls in 2010, no need for his gut…
And that’s the difference between him and Nate Silver. In 2010, Silver, while he personally probably would have wanted a different outcome, believed in the polls and predicted a major win for the Republicans. In 2012, Cost won’t believe in the polls because they show something that he doesn’t want, so he goes for his gut…
I think maintaining epistemological honesty is a damned hard thing to do. I struggle with it like a Baptist struggles with demon rum. (And lose about as frequently.)
That said I found the reactions to the presidential debates very telling. Obama lost the first one badly. And every democrat said so. Obama won the second one handily. And very few Republicans were willing to admit it. Ditto the third.
One of the things that makes Democrats weak in confrontations with Republicans is that we do honesty. The reason we don’t have our own Fox News (no, MSNBC is not yet that low) is that we don’t much like being lied to or harangued – even by people we fundamentally agree with.
Anything from the Weakly Standard is to be considered seriously?
The print version of Fox News?
I once saw an article in that rag that claimed oil was a renewable resource.
The Weakly Standard endorsed the invasion and occupation of Iraq.
Jan believes things the Weakly Standard prints.
Christ…what a joke of a rag.
Fish won’t be wrapped in that garbage.
I like Matt Labash.
Or something like that.
I’d like to see even a single honest analysis done in The Weekly Standard. Not an honest analyst, just a single honest analysis.
I get what Jay Cost is saying about his gut and the economy. But there are two things I think he’s missing. Thing 1: the economy is, in fact getting better. Thing 2: Over half the country doesn’t blame Obama for the economy – they blame Bush. Still.
And in swing states, the margin between those who blame Bush and those who blame Obama is even higher.
@Alex Knapp: ah, but that sounds dangerously like a data-driven analysis 🙂
I remember taking a class in High School that had something to do with the Social Sciences.
The instructor told us about how, long ago, there was a group of wise old Monks who needed to know how many teeth were in the mouth of a horse.
They examined many of the ancient scrolls and texts and sacred volumes but there was no information on the number of teeth of a horse.
After a time the wise old Monks were joined by a novice to the order who was presented with the problem.
“We have studied the sacred books of knowledge but can not find an answer to our query. You can assist our task as there are many more volumes to search.”
“But Brothers” the young Monk asked “Why don’t we just count the teeth in the horses mouth?”
“Aggh!” “Blasphemy!” cried the wise old Monks. “All knowledge resides in the Ancient Texts!”
And the young Monk was banished from the order.
I’d say since Tetlock and Expert Political Judgment, no one should assert expert prediction in any social realm. It’s that easy. The studies are in. Experts do worse than informed generalists.
And of course that is in fuzzy non-numeric problem domains. When you have a lot of data … LOL … of course that will trump Expert Political Judgment.
I have always believed that the “undecided” count in polls represents those potential voters who want to vote for the perceived winner.