How To Evaluate Polls
Nate Silver offers some helpful advice on how to properly evaluate a poll result:
1. How does the poll compare to other recent surveys of the state? This is a simple one, and in the Quinnipiac poll, this comparison was favorable for Mr. Obama. He led by six points, better than his average lead of one point in the other Ohio surveys conducted so far in August.
2. How does the poll compare to the polling firm’s previous surveys in the state? Note my usage of the plural, surveys. You should be looking at how a new poll compares to the prior one that a survey firm released in a state — but also in comparison to its entire body of work there.
(…)
3. How does the survey compare with the polling firm’s surveys in other states? A related question is whether the polling has a strong“house effect.” That is, does it consistently show greater numbers for a particular candidate, no matter where it polls? If so, then it’s not really a big deal when it ventures into a new state and print another one.
(…)
4. How does the poll compare with the national trend? This is the trickiest of the seven concepts, but one of the more important ones. When comparing a poll with prior surveys of the state, it’s important to look at exactly when each survey was conducted, and what the national environment looked like at the time.
(…)
5. How does the poll compare with the historical trend in a state? A state poll can be used to make an inference about where the national race stands. Ohio, for example, is typically very slightly Republican-leaning relative to the country as a whole. So when Mr. Obama gets a poll showing him 6 points ahead there, that speaks favorably to his position in the national race.
(…)
6. How does the poll relate to the electoral calendar? Any time that a candidate is leading in a state, and he simply maintains that lead, that counts as a modest positive for him because time is running off his opponent’s clock. A three-point deficit in April is very easy to overcome. In August? Still plenty of time, but there’s room for a touch of concern. On election eve? Oops — it’s probably too late.
(…)
7. How does the number of undecided voters compare with prior renditions of the survey? In general, I focus on the margin between the candidates (“Mr. Obama’s up by three points”) rather than their absolute numbers (“Mr. Obama 47 percent, Mr. Romney 44 percent”), or the number of undecided voters in the state.
Further details on each item at the link, but the overall point is that looking at the topline number, or even the crosstabs as many political bloggers have become obsessed with doing this election cycle, isn’t really enough. Truly understanding a poll, and perhaps recognizing when it might be an outlier, requires a little more thought than that.
Some think it important to distinguish between results for likely voters vs. registered voters or, at a minimum, not to compare a poll of the former with a poll of the latter.
But, and this ia a major hot button for me, the margin of error – while widely reported – is rarely interpreted properly even by professional pollsters on the tele (no doubt because they’ve long since realized it is useless).
The MOE represents the sampling error, the chance that the results would have been different simply by the luck of the draw. So if, for example, this week a poll shows A ahead of B by 50% to 48% and next week behind by 48% to 50%, and the MOE is 4%, nothing has changed. A has not lost momentum, B has not gained momentum. So talk about what this “means” for A or B is foolish.
Also the MOE works on both sides, so if the MOE is 4% in a poll with two options (A and B), it means that A or B could be either 4% higher or 4% lower (well, actually, I believe there is some legerdemain here, way past my grade level, so that the upside/downside on each side is actually a little less than the MOE).
Trends, I think, do matter but I don’t know at what point, or under what circumstances, a lead over time shoutd outweigh in one’s analysis the MOE of the individual polls.