How Confident Should We Be in the Polls?
Two prominent media pollsters are hedging their bets.
Washington Post polling director Scott Clement and his colleague Emily Guskin explain “How polling works and how it’s changed since 2016.” OTB readers are likely up to speed on the basics of survey methodology, sampling error, and the like but the nuance is constantly evolving.
First off, while any competent polling firm can pretty accurately portray public opinion, predicting behavior is much harder. So, while we have a great idea about how Americans feel about Donald Trump’s performance as President, it’s harder to know who will actually show up to vote. Second, because of the odd way in which we elect Presidents in particular, the problem is compounded by having to predict how people in a handful of key states will behave.
A 2017 report by the American Association for Public Opinion Research (AAPOR) found that while national polls had an average error of about two points in presidential elections since 2000 and were fairly accurate in 2016, the error in state polls was double that, at four points. The level of state polling error ranged from 3.2 points in 2004 to 5.1 points in 2016.
It is almost impossible to predict the direction in which polls will miss in a given year. In 2012, some supporters of Republican nominee Mitt Romney criticized surveys during the campaign for overrepresenting Democrats. But final state polls actually underestimated Barack Obama’s vote margin by 2.3 percentage points on average, according to the AAPOR.
The 2012 errors were largely forgotten because most polls showed Obama leading, just by a smaller margin. Yet the 2016 election demonstrated how systematic errors in a variety of states can lead to a surprising outcome. National polls missed by just a single percentage point on average, yet state surveys underestimated Trump’s vote margin by three points in Michigan and Pennsylvania, as well as seven points in Wisconsin.
Even though my late first wife was an executive with Public Opinion Strategies, the leading Republican polling firm in the country (they’re also half of the team that does the NBC/WSJ Poll and the CNBC All-America Economic Survey), I don’t fully understand why polling at the state level is harder. Part of it, though, is simply that it’s essentially as expensive to conduct a state poll as it is a national one and there are therefore many fewer state-level polls available to the public. Additionally, since the Presidential election hinges on a handful of “swing” states, it’s quite possible that those particular contests are intrinsically harder to predict. (Meanwhile, I can confidently predict which party will take Alabama in 2024 without conducting a single poll or having any idea who the party nominees will be.)
Pollsters have made two significant changes this year aimed at improving accuracy over 2016.
One is simply conducting more polls, a costly decision but one that may contribute to greater precision overall. From the start of September to last week, RealClearPolitics tracked 105 polls in Pennsylvania, Michigan, Wisconsin and Arizona, nearly double the 54 polls over the same period in 2016 in these states. The number of nonpartisan polls conducted with live telephone interviews — a more expensive but historically more accurate method — grew from 24 in 2016 to 36 in 2020.
In addition, more polls also appear to be weighting samples by educational attainment, something that wasn’t done in 2016 and that the AAPOR’s post-election report found contributed to polls underestimating Trump’s support.
Polls are routinely weighted to match estimates of population demographics including race, age and other factors that are correlated with voting. In 2016, there was an especially strong correlation between education and support for Trump and Clinton in key states, with Trump winning by wide margins among White voters with some college or less. Current polls show that correlation remains strong in this year’s election between Trump and Biden. College graduates have long been more likely to participate in surveys, and states that did not weight samples to correct for that — by weighting down the share of responses from people with degrees to their accurate share of the population — were at greater risk of underestimating Trump’s support.
But, of course, that’s dangerous. Any weighting takes what is a purely mathematical exercise and adds human judgment. That’s required; indeed, the difference in a “good” poll and a “bad” one is the likely voter screen. If those with college degrees and those without vote at the same levels, then it obviously makes sense to correct one’s sample to reflect that fact. But my guess is that college-educated folks are more likely to turn out. If so, making an adjustment based on what may well be a one-off in the previous election could well skew the polls. (Although, in this case, it would likely understate Biden’s margin if 2020 turns out to be a reversion to the norm.)
They close with the standard disclaimers:
Polls in 2016 were more accurate than many people think, and in the long run, there’s reason to have confidence in them. Still, there are enough wild cards this year — the coronavirus pandemic, extraordinary political interest, an election like none before and adoption of new voting methods — that could affect the accuracy of pre-election polls.
Take the pre-election vote estimates as advertised: a tally late in the game, with voters in charge of the final score.
That’s both good advice and a cop-out. I’d be willing to bet a good deal of money on Biden winning the popular vote. Further, I’m reasonable confident that he’ll win in enough states to get to 270 Electoral votes given the steadiness of the polls. But I’m not at all confident in how Pennsylvania or Wisconsin will turn out.
Uncertainty is inherent in polling. A sample only reflects the whole in part. I sure that there were good reasons to pull Blake Snell in game six, but those reasons didn’t pan out. Categorical certainly is not achievable. I don’t think that polls should influence my vote; I’ve voted for people who were extremely unpopular with the rest of the electorate and don’t think that I was wrong to do so.
538 now has Biden winning in 93 out of 100 scenarios.
Sam Wang has Biden winning 357 to 181 electoral votes.
The Economist has it at <1% chance of Trump winning.
I am only confident that they are all totally and tragically wrong.
No polling or forecasting takes into account the Justice Boof Court, over 200 new and unqualified Federal Judges, Baghdad Bill Barr, the Turtle-Face McConnell Senate, corrupt Republican State Legislators and Election Officials, or who knows what other crooked nonsense is inevitably coming our way?
Already the Bush v. Gore Decision, allegedly "limited to the present circumstances" of the election of 2000, is being used as justification to suppress votes, again, in 2020.
I am braced for a second term of President Trump, and with it, the end of Presidential Elections in the US.
Even if Biden wins both the Popular and Electoral vote, going away.
So much corruption has been, and continues to be, allowed to go completely unchecked…why would any sentient being believe for a second that it’s going to magically stop on November 3rd or 4th?
Biden winning is the sunshine filled happy ending to a dark evil tragedy that only happens in the movies.
I hope you are paying attention. You are witness to the end of the USA as we know it.
Dave Wasserman at Cook Political Reporter has been quoted this week by 2-3 columnist and hiw own writings at Cook, about he has been granted access to Congressional district poling being done by congressional campaigns in what are considered key districts in the presidential race. While the congressional campaigns are interested in that race, they do ask about other races. That polling is generally not available to the press, but it is the polling that in 2016 that set off the alarm bells in WI, MI and PA that Clinton was in trouble, she chose to ignore that polling and trust the analytics. Sort of like Kevin Cash.
The large polling operations typically don’t have this granular view and use algorithms to compensate, sometimes they’re right and sometimes…
Oh, Wasserman says the the CD polling indicates that Biden is a shoe in, with a wide enough EV margin to make the SC machinations moot.
I’m not sure why you consider the weighting dangerous, though it seems to me you’ve reached a different conclusion on what they mean than what I did. I didn’t interpret that as being mostly about the likely-vs-registered voter adjustments but about the even more fundamental demographic weight adjustments. e.g. If you have a “middle-aged white male” demographic in part of a poll and call 100 college educated, middle-aged white males and 60 non-college educated, middle-aged white males but the State’s real split is 2 college for every 3 non-college, without re-weighting your sample you’ll over-represent the opinions of the college educated.
@NBH:
Sure. That’s basic math. But pollsters were pretty good at math in 2016 and were already weighting to match population characteristics. It sounds like they’re trying to also weight the likely voter screen based on what happened in 2016; that’s more dangerous because it’s making a judgment rather than just looking at the data.
@Daryl and his brother Darryl:
538 on my internet says 11% chance for Trump, not 7%. I’d also like to disagree with the rest of your very dark thesis. I’d really, really, like to disagree with your dark thesis. I want so badly to disagree with your thesis. But I don’t.
Biden, riding on a wave of revulsion over Trump, may have a ratfrack proof edge this year, but we have a federal election every two years and the Justices are in for life. The GOP’s plutocratic policies cannot win an honest election. They have to change their policies or eliminate honest elections. And they really don’t want to change their policies.
“It’s hard to make predictions, especially about the future.”
@gVOR08:
Ah…you are correct…I read the wrong page.
Doesn’t change my outlook in the least.
@Daryl and his brother Darryl:
I don’t think even the Trump-packed court or the lackey Senate can overthrow an election that is not close, or that doesn’t hinge on or two results.
As for:
Elections will remain, the’ll just be fixed so firmly no one will be able to win except the GOP candidates. Mexico did this for decades.
But it took us two extended dictatorships (Juarez and Díaz), plus a foreign invasion and a period of monarchy (France and Maximilian I), and a civil war between 1910 and 1921 to get to that point.
@Kathy:
To put on my Bill Clinton hat, that depends on what your definition of “close” is.
If a Biden “landslide” is dependent on vote-by-mail, Trump may appear to be ahead on Election Night. He’ll declare victory then go after the “fake” mail ballots that subsequently come in. And the right-wing courts could easily side with him at least part of the time. There ain’t gonna be a landslide if the votes on which that landslide rests aren’t counted in the first place.
Remember Obama’s 2008 slogan “Yes we can”? The GOP’s 2020 slogan should be, “Because we can.”
@Kylopod: I wonder if the opposite might be true: so many mail-in votes already counted on election day give Biden the edge. How will Trump react then? Will he insist on counting military absentee ballots that have yet to arrive, thinking that they’ll break for him, and make the same false distinction he’s been making all along between mail-in and absentee ballots?
@Monala:
Some states start counting mail ballots on election day, but some well before then.
Some states count ballots postmarked by election day, others require ballots to be received by election day to be counted.
We should know who carried AZ, FL, NC on Nov.3. If Biden carries any of those, Trump will not be looking good.
(AZ changed the law after Sinema/McSally in 2018, now they start counting 14 days before the election, my ballot is already counted).
@Monala:
It could be. But no matter what the outcome, it’s already a virtual certainty that numerous votes aren’t going to be counted, and it’s likely those will be disproportionately Democrat. The only question is whether that’s sufficient to deny Biden a victory. Those ballots that are already being discounted as we speak in PA, FL, TX and elsewhere aren’t cheap. We can’t just take for granted that Dems will overwhelm the vote so thoroughly those losses won’t matter. That could be what happens, but it’s not inevitable.
That’s an excellent point–but the courts don’t need to accept every element of Donald’s ongoing tantrum in order to achieve the ends he desires. We’ve already seen the pattern throughout his presidency. He doesn’t win all the legal battles. But he wins a lot more than he deserves to.
I’m not as pessimistic about next week as Daryl and his brother Darryl here is. I think when all is said and done, the likeliest scenario is that Biden comes out of this the decisive winner. But I think the nightmare scenario is a very real possibility, and most people have no idea how close we are to seeing the end of American democracy. If you’ve got cancer with an 80% chance of survival, the odds against you are still way, way too high.
@charon: i voted in Florida on Saturday. After voting I had to feed my ballot into a machine. It threw an error on one of the votes and spit it back out. I got a pen and filled in the problematic circle a little better. The machine accepted it and tallied the votes.
The latest polls today for Florida have Biden up by 3, 4, and 6 points. I expect by Tuesday evening we will know he won the state.
@Teve: I got my NY absentee ballot weeks ago, but I didn’t mail it, and when the polling stations opened earlier this week, I turned mine in. It saved me having to wait in line if I had simply done regular, in-person voting.
I remember going to vote in MD in 2000 and thinking to myself how incredibly easy the ballots were–a thought that kept coming back to me as I watched the subsequent shitshow in FL. It’s amazing the lengths our country seems to go to to make voting harder and more complicated than it needs to be. Some of it is outright targeted suppression of course, but FL seems to have its own history of just being incredibly inept at the process.
I’m sticking to my prediction that Biden wins in a blowout, with FL, GA, NC, IA, WI, PA, AZ, NV, MI, and MN all not even close. Texas is still a possibility, as well.
In Jan 2019, I said Biden/Harris. I’m sticking to that.
Oh, and I predict IA, NC, GA (Ossoff), CO, ME, AZ, and SC are flipped in the Senate, with Jones (AL) losing. MT is also a possibility for a flip.
@charon:
EddieInCA says:
Thursday, October 29, 2020 at 17:20
@charon:
If Biden wins PA, or GA, or FL, or NC, Trump has no path to 270. If any of those are called early on Tuesday, it’ s over.
@EddieInCA:
I’m getting twitchily obsessive about PA polls.
Every time I look online, I end up going to 270-to-Win and/or FiveThirtyEight and look at PA.
If Biden takes PA then assuming there are no massive upsets in the upper Midwest (or maybe South West?) that’s the got to be the game right there.
It’s still worryingly close, but at least latest polling takes Biden back over 5% in PA.
I could be wrong, as an ill informed UK-anian, but I really don’t think so.
Florida etc nice to have; but PA is the keystone.
@JohnSF: “Pa is the keystone.” And I hope I don’t come across as condescending but you know that the state is in fact the “Keystone State” in their official motto.
Interesting; didn’t know that. Why would that be? Position between northern and southern states?
@JohnMcC: I believe Wisconsin is the Keystone for Team Orange. I believe his odds go up to 35% should he win there–otherwise he has extremely limited and low percentage options to piece together 270. If he loses Florida–its over.