Those Awful Polls

Beldar, commenting on my recent analysis of Fred Thompson’s chances in tomorrow’s South Carolina’s primary, observed, “you have far more faith in polls than I do, or than their results (especially recently) would merit.”

There’s certainly reason to suspect the polls, given Hillary Clinton’s surprise win in New Hampshire and Mitt Romney’s larger-than-projected win in Michigan. But let’s look at the overall picture.

Iowa Poll Results vs. Iowa Vote

Caucuses are notoriously difficult to project because of their dynamic nature. That’s especially true under the rules the Democrats used in Iowa, where those who voted for candidates who get very little support were then asked to chose from the more viable candidates. But look at the final pre-caucus RealClear Politics average* versus the final results:


Iowa Poll Results vs. Iowa Vote Democrats

They got the Biden-Richardson order wrong; then again, they were marginal candidates and it’s incredibly hard to predict what their supporters will do to begin with, let alone on a second ballot. Otherwise, only Obama’s vote total was outside a +/-3 margin.


Iowa Poll Results vs. Iowa Vote Republicans

The razor-thin order of finish between Thompson and McCain was reversed; then again, that’s well within the margin of error.* Otherwise, the polls were uncannily accurate.

New Hampshire Poll Results vs. New Hampshire Vote

Here’s where the pollsters got it fantastically wrong, right? Well, let’s look.


New Hampshire Democratic Primary Poll Results

The polls showed Barack Obama winning handily and had the pundits, myself included, speculating as to whether Hillary Clinton could survive two straight losses in the traditional bellweather states. Instead, Clinton got nine percent more than expected and won rather easily. Oops.

It’s worth noting, though, that the numbers for Obama, John Edwards, and Bill Richardson were spot-on. The undecideds simply broke overwhelmingly for Clinton, for a variety of reasons that we’re still guessing about.


New Hampshire Republican Poll Numbers

A few more undecideds broke for McCain than predicted. Otherwise, no surprises here.

Michigan Poll Results vs. Michigan Vote

Another odd one. The Democratic primary was uncontested, with everyone but Hillary Clinton dropping out after the party stripped the state of its slate of delegates for voting early. On the GOP side, a favorite son was facing a guy who’d won the contest eight years earlier and was coming off a bounce from New Hampshire.


Michigan Democratic Primary Polls

More people bothered to show up and vote Uncommitted than expected. But Clinton’s numbers are amazingly accurate in a contest that didn’t matter and where there was enormous incentive to cross over to the Republican side.


Michigan Republican Primary Polls

The pollsters got the order of finish exactly right but were again off by nine points in predicting the winner’s total. But it’s worth noting, too, how inconsistent the polls were here. Let’s move beyond the average and look at the individual polls:

Michigan Republican Primary Polls Macro

For whatever reason, the pollsters had an incredibly difficult time predicting who would win here; the McCain and Romney numbers were wildly disparate from poll to poll. The speculation is that Democrats and Independents who were leaning McCain ultimately didn’t bother to turn out but there are other theories, too.

That’s always the trickiest part in polling: predicting whether behavior will follow attitude. The best are very, very good at screening for likely voters in general elections. It’s much harder in primaries because every season has a different dynamic. We’re in the first two party wide-open contest in memory, so weighing results by what has happened in the past is dicey.

South Carolina

So, what does this tell us about tomorrow’s Republican primary in South Carolina (the Democrats don’t vote there for another week)? Close races between viable candidates can surprise us but nobody’s going to get twenty points more than expected and shock the world, either.

Let’s look at the most recent polls, then:

South Carolina Republican Primary Polls

It’s quite reasonable to conclude that we have no idea whether McCain’s 4.4 percent average margin will translate into victory tomorrow. Were Huckabee to win, it wouldn’t be any shock whatsoever.

At the same time, I think we can say with very high confidence that Mitt Romney, Fred Thompson, Rudy Giuliani, and Ron Paul have no shot at winning tomorrow. If anything, their numbers are likely overstated somewhat because some of their supporters are likely to find better things to do on a Saturday while others will likely decide to cast a vote for Huckabee or McCain.

*Technically speaking, averaging polls using different methodologies, as RCP does, is problematic and it’s improper to speak of a “margin of error.” For the purposes of our discussion, though, it’s close enough.

FILED UNDER: 2008 Election, Public Opinion Polls, , , , , , , , , , , , , , , , , , ,
James Joyner
About James Joyner
James Joyner is Professor and Department Head of Security Studies at Marine Corps University's Command and Staff College. He's a former Army officer and Desert Storm veteran. Views expressed here are his own. Follow James on Twitter @DrJJoyner.


  1. Beldar says:

    I’m flattered to have inspired this impressive collection of tables, but I’ll stick with Mark Twain: There’s lies, there’s damned lies, and then there’s statistics. “Margin of error” actually means “license to get away with guessing wrong.” And the biggest lie is that the margin of error itself is not just another guess.