How To Read Polls
The Washington Examiner’s Timothy Carney offers some good practical advice on how to read all the polls that are going to be coming out over the next 11 months:
1) Never put too much weight on any one poll. If the latest poll tells you something different from the previous ones, wait to see if the latest poll is an aberration, or the beginning of a trend.
2) Compare any new poll against previous polls by the same pollster to get a notion of trends. That’s what Byron York does in his blogpost on Gingrich’s drop. Comparing a Rasmussen poll today to a University of Iowa poll last week might tell you more about the different pollsters’ methodology than it tells you about trends in public support.
3) Don’t pay attention to polls that are of “adults” or “registered voters.” Only “likely voter” polls are apt to be very meaningful. And in Iowa and New Hampshire, the sample should be at least 500 likely voters.
4) Most polls have a margin of error, and most of those margins ar somewhere between 2 or 5 points. A 2.5% margin of error, for instance, usually means that the pollster is 95% certain that if they had polled the entire population in question, no candidate’s numbers would move up or down more than 2.5%.
As a related point, if Romney is leading Gingrich by 1 point, it’s probably more accurate to say that “Romney and Gingrich are within the margin of error” than to say that Romney is leading.
5) Finally, remember polls can’t directly measure enthusiasm or softness. And they’re often simply wrong.
So, here’s the general gist: use polls of gauges of trends and ballpark figures. Don’t ever use a single poll as a real prediction of election results. That’s one reason I love RealClearPolitics, which aggregates polls, so you can see a bigger picture.
All good advice, and I second Carney’s recommendation of RealClearPolitics, which is once again doing an excellent job of staying on top of the polls at the national and state level.
Great Doug Does this mean that you are no longer going to be trumpeting single, outlying pols that represent “new lows for Obama”, or “Obama numbers keep declining”, removed from any context, or recognition of the meaninglessness of a few points difference?
I would take issue with #3. “Likely voter” polls are quite tricky and should be approached with caution, because they fundamentally represent a manipulation of the raw data using criteria that are far from straightforward. Using a LV screen means you are eliminating some of the responses you get, because you are guessing that some people will not vote. If you are very close to an actual election, and you directly ask the respondents about the likelihood that they will vote, then you are probably ok. But more than a few weeks away from the election, the reliability of the answer drops off.
And some pollsters do not even ask. They presume, based on demographic categories, what percentage of certain classes of people will vote, and they modify their results accordingly. This is far less accurate, and opens the door to all manner of unintended errors, or intentional abuse.
Thus one should not just look for the “LV” sign and assume it is a better poll than registered voters or adults. Rather the contrary, a well-sampled poll of all adults or registered voters has fewer factors which can screw up the results.
??? All polls have a margin of error.
This is WRONG.
A margin of error of 2.5% means that IF YOU HAVE AN UNBIASED SAMPLE of the general population, then you can be confident that 95 times out of 100 times that you repeat this poll with the exact same methodology, your numbers will be within 2.5%, more or less.
The margin of error is totally a function of the sample size – bigger sample size, smaller margin – because the only thing the margin really measures is whether your sample is big enough to overcome the random errors that could arise with a very small sample.
THe margin says NOTHING about the quality of the sample. It does NOT give one any confidence that the real state of the population is within 2.5% of the result shown.
For example, if you do a poll to assess Obama’s reelection chances, and you use a well-designed methodology, with a good. random sample, but confine the pool of respondents to those living in Harlem, you will get a result, and a margin of error that has nothing whatsoever to do with the larger reality. Most national polls would not, of course, make such an egregious error, but selecting a sampling pool that perfectly mirrors the overall population is a lot harder than you might imagine, and relatively small distortions can skew the results enough to make the results meaningless. That is why competent, legitimate polling firms get different results.
The “margin of error” does not address this question of accuracy at all. So no, it does not measure the likelihood that the results are accurate relative to the numbers you would get if you polled everyone in the population. It only speaks to the likelihood of getting the same numbers if you repeated your poll, using the same sampling methodology.
I don’t agree with this, at least as far as registered voters. Likely voter screens tend to be pretty ad hoc and can bias a sample. The RV screen is much more reliable and arguably as valid as many LV screens.
In addition to my objections noted above, let me offer an additional recommendation
Actually, don’t ever use ANY poll, or number of polls as predictors. Polls are DESCRIPTORS, not predictors. They only have predictive power if they are very well done in the first place, and then nothing changes before the event they are predicting happens.
I know this is rather obvious, but people seem to forget it all the time. Polls are a snapshot of where the population is today. Any of an infinite number of things can happen tonite to change opinions in the population, and render the poll outdated by tomorrow morning.
@Tano:
Given the lag time, they are a snapshot of where the population was 4-5 days ago, give or take a day or two.