How To Evaluate Polls
Nate Silver offers some helpful advice on how to properly evaluate a poll result:
1. How does the poll compare to other recent surveys of the state? This is a simple one, and in the Quinnipiac poll, this comparison was favorable for Mr. Obama. He led by six points, better than his average lead of one point in the other Ohio surveys conducted so far in August.
2. How does the poll compare to the polling firm’s previous surveys in the state? Note my usage of the plural, surveys. You should be looking at how a new poll compares to the prior one that a survey firm released in a state — but also in comparison to its entire body of work there.
3. How does the survey compare with the polling firm’s surveys in other states? A related question is whether the polling has a strong“house effect.” That is, does it consistently show greater numbers for a particular candidate, no matter where it polls? If so, then it’s not really a big deal when it ventures into a new state and print another one.
4. How does the poll compare with the national trend? This is the trickiest of the seven concepts, but one of the more important ones. When comparing a poll with prior surveys of the state, it’s important to look at exactly when each survey was conducted, and what the national environment looked like at the time.
5. How does the poll compare with the historical trend in a state? A state poll can be used to make an inference about where the national race stands. Ohio, for example, is typically very slightly Republican-leaning relative to the country as a whole. So when Mr. Obama gets a poll showing him 6 points ahead there, that speaks favorably to his position in the national race.
6. How does the poll relate to the electoral calendar? Any time that a candidate is leading in a state, and he simply maintains that lead, that counts as a modest positive for him because time is running off his opponent’s clock. A three-point deficit in April is very easy to overcome. In August? Still plenty of time, but there’s room for a touch of concern. On election eve? Oops — it’s probably too late.
7. How does the number of undecided voters compare with prior renditions of the survey? In general, I focus on the margin between the candidates (“Mr. Obama’s up by three points”) rather than their absolute numbers (“Mr. Obama 47 percent, Mr. Romney 44 percent”), or the number of undecided voters in the state.
Further details on each item at the link, but the overall point is that looking at the topline number, or even the crosstabs as many political bloggers have become obsessed with doing this election cycle, isn’t really enough. Truly understanding a poll, and perhaps recognizing when it might be an outlier, requires a little more thought than that.