2012 Pollster Rankings: Rasmussen And Gallup Among The Least Accurate
There are some expected and unexpected results in Nate Silver's review of pollster accuracy in 2012.
Nate Silver is out with his traditional review of the performance of prominent pollsters and, as with the survey I cited in the aftermath of Election Day, Rassmussen Reports and Gallup ended up faring the worst:
Among telephone-based polling firms that conducted a significant number of state-by-state surveys, the best results came from CNN, Mellman and Grove Insight. The latter two conducted most of their polls on behalf of liberal-leaning organizations. However, as I mentioned, since the polling consensus underestimated Mr. Obama’s performance somewhat, the polls that seemed to be Democratic-leaning often came closest to the mark.
Several polling firms got notably poor results, on the other hand. For the second consecutive election — the same was true in 2010 — Rasmussen Reports polls had a statistical bias toward Republicans, overestimating Mr. Romney’s performance by about four percentage points, on average. Polls by American Research Group and Mason-Dixon also largely missed the mark. Mason-Dixon might be given a pass since it has a decent track record over the longer term, while American Research Group has long been unreliable.
It was one of the best-known polling firms, however, that had among the worst results. In late October, Gallup consistently showed Mr. Romney ahead by about six percentage points among likely voters, far different from the average of other surveys. Gallup’s final poll of the election, which had Mr. Romney up by one point, was slightly better, but still identified the wrong winner in the election. Gallup has now had three poor elections in a row. In 2008, their polls overestimated Mr. Obama’s performance, while in 2010, they overestimated how well Republicans would do in the race for the United States House.
Among the other pollsters that ranked low in accuracy this cycle were American Research Group, which has had a spotty reputation for years now, and Mason-Dixon, which has generally been well regarded but seemed to always come back as an outlier in 2012 for some reason. The fact that Rasmussen is on the list is no surprise, of course. The company had generally done well in 2004, and in 2008 it was among the most accurate pollsters out there. By 2010, though, its Republican “house effect” became far more pronounced to the point where it was at the bottom of Silver’s accuracy ratings. The same thing happened this year, as anyone observing the polls could have told you. The main reason for this, of course, is Rasmussen’s decision to weight its polls for Party ID based on a model that assumes, contrary to the findings in most polls, that there are more Republicans in the country than Democrats. As long as the polls operate with that assumption, they are going to be consistently unreliable.
Interestingly, Silver’s review of the polls found a surprising degree of accuracy from a form of polling that most of us have been dismissing for years, online polling:
[S]ome of the most accurate firms were those that conducted their polls online.
The final poll conducted by Google Consumer Surveys had Mr. Obama ahead in the national popular vote by 2.3 percentage points – very close to his actual margin, which was 2.6 percentage points based on ballots counted through Saturday morning.
Ipsos, which conducted online polls for Reuters, came close to the actual results in most places that it surveyed, as did the Canadian online polling firm Angus Reid. Another online polling firm, YouGov, got reasonably good results.
Among the nine polling firms that conducted their polls wholly or partially online, the average error in calling the election result was 2.1 percentage points. That compares with a 3.5-point error for polling firms that used live telephone interviewers, and 5.0 points for “robopolls” that conducted their surveys by automated script. The traditional telephone polls had a slight Republican bias on the whole, while the robopolls often had a significant Republican bias. (Even the automated polling firm Public Policy Polling, which often polls for liberal and Democratic clients, projected results that were slightly more favorable for Mr. Romney than what he actually achieved.) The online polls had little overall bias, however.
The difference between the performance of live telephone polls and the automated polls may partly reflect the fact that many of the live telephone polls call cellphones along with landlines, while few of the automated surveys do. (Legal restrictions prohibit automated calls to cellphones under many circumstances.)
It’s worth noting that the type of online polling we’re talking about here is far different from the “polls” that you often seen getting thrown up on news sites and blogs where anyone who happens by can respond, sometimes more than once, if the site isn’t thorough enough in preventing multiple responses from a single person. These are polls conducted online where the pollsters are trying to come up with the same kind of representative samples that telephone pollsters have worked at putting together. The fact that they ended up being more accurate than telephone pollsters suggests strongly that the skepticism that I and others have expressed about this form of polling may no longer be warranted and that these types of polls deserve to be taken more seriously in the future.
One implication of this findings regarding online polls could be the fact that it provides a possible future solution to the problems that many pollsters have reported in recent years regarding their declining response rate. People may be more willing to participate in an online polling panel than they would be to spend 15 or 20 minutes answering questions about politics while their trying to get dinner ready in the evening. That doesn’t mean that we’ll see an end to telephone polling altogether. As Silver notes, such polls continue to serve a purpose and will continue to do so in the future, at least as far as live telephone polls that call both landlines and cell phones are concerned. The numbers above make clear, though, that automated polls such as those conducted by Rasmussen and Public Policy Polling tend to be among the least reliable polling methods. While it’s unlikely that pollsters will abandoned robo-dialing entirely because of the fact that its far cheaper than live interviews, it’s possible that we’ll see some of these robo-pollsters make a transition to online polling as that field becomes more and more refined. Until then, it seems rather clear that we need to discount the reliability of such polls, and to start taking those scientifically conducted online polls far more seriously.
They kept republicans hopeful til the end, making election night far more painful for them.
Thanks for that!
The polling firms must be very worried that there are folks who analyze polls who are far more accurate than they are. Nate Silver went 50-for-50, but he wasn’t the only one to do that. There were a number of models that went 49-for-50 or 50-for-50, though without the big megaphone that Silver has at NYT.
It’s analogous, in a way, to blogs. Lots of people are getting zillions of page views commenting on what the ground-level news gatherers (NY Times, WaPost, etc.) are writing, without actually gathering any news themselves.
At then end of the day, reporting and polling are expensive, and they’re being turned into secondary sources of information, with the commentary/analysis that sits on top of them becoming more and more prominent.
I wonder how Bill Kristol was as an anti-predictor. There’s some utility in a weathervane that points in the opposite direction of the prevailing winds.
To the author – good article but it has a fatal flaw. You mention that Rasmussen was one of the most accurate pollsters in 2008. While nationally it was ok (just about every poll had Obama winning by a good margin), Rasmussen was one of the worst when you look at its final state by state battleground polls right before the election (check realclearpolitics.com). All the 2008 key battleground state polls were off and favored McCain from between 2 and 7 points. Several were outside the margin of error. Personally, I find this to be horrendous polling.
For 2012 it was pretty much a repeat state performance, except that Rasmussen screwed up the national poll. The bias toward Romney in ALL the battleground states toward Romney was between 2 and 8 points, very similar to the bias in 2008. But in the end, was grossly off and horribly biased.
Finally someone notices PPP GOP lean. People claim they have Democratic house effect but the lack of cell phones undermine that claim.
PPP’s last polls all got it pretty much right. Ironic that their house effect was Republican leaning.
Gallup needs to junk their LV model. It simply doesn’t work. Their RV results were quite accurate.
You’re still saying this even though I showed you that you are wrong. As Tom also pointed out.
@EMRVentures: There is no reason the polling firms can not duplicate what Silver does. Silver treats each poll like the results from a single game and his poll of polls is like seasonal stats (there is more to it but not as much as people think). There is no reason a polling firm could not run various polls itself, some online, some robo, different ones with different likely voter assumptions and then aggregate the results.
Well, nothing except cost.
I am not sure how much money there is in polling but if there is enough, I would buy up a couple of the smaller firms and write Nate a fat check and setup a super polling firm.
Then again, there is google already out there with good polling. And google can afford to write Nate a very fat check. Google is a world leader in capturing and processing big data. Google does not need Nate, they can easily reproduce what he does, but the branding would be worth a lot and google has plenty of cash lying around earning 0% interest.
@EMRVentures: I don’t think pollsters have much to worry about, since all the best number crunchers – Silver, Wang, Linzer, Jackman – relied heavily on polling data. They need numbers to crunch, and econometric variables have a horrible track record in comparison.
Dude, It looks more and more like Obama stole the election with MASSIVE VOTER FRAUD and you’re here attacking pollsters.
Danm I hope you guys wake up one of these days….
Thank god you’re back, GA. We’ve had superdestroyer bringing the obsessive racism, and Tsar bringing teh stupid, but we haven’t had anyone bringing the complete batsh!t insanity. (Sadly I don’t think we’ll see much of the Jan and Drew show again.)
@G.A.: Tell us G.A., how far left do Republicans have to move before they lose your vote? Allow abortion in cases of rape? Amnesty for immigrants?
I’ll tell you my theory. It’s obvious that Mr. Romney won the states where livestock outnumber humans. Mr. Obama won the states with few livestock as a percentage of the population. I think the only possible conclusion is that in places like North Dakota, Nebraska, Kansas, Idaho, etc… cows were allowed to vote.
Try to disprove that. The logic is unassailable.
I would like to see a poll conducted on Obama’s job approval where people identify if they are a military veteran and/or are on active military duty. I am a cold war veteran, and all of the veterans I know are extremely upset with the election results. There is a common feeling that the American Citizens have betrayed all of the Veterans who have served to maintain our freedom and the Constitution. That is exactly how I feel. For those who have done nothing but take but have not given, well, someday you will regret what you have done. It might just be sooner than later. As far as I am concerned, the oath we took is for life even though I have an honorable discharge from the U.S. Army. It is really sad to watch the Country self-destruct. It is amazing how many people who do not live in America feel the same way.
All the veterans I know are very happy that the candidate who hid in Paris instead of serving in Vietnam lost.
Isn’t anecdotal evidence fun? 🙂
@michael reynolds: Only an Obama voter would make such a stupid comment. Unfortunately, you have lots of company. That could be looked upon as proof this Country is in really bad shape.
Our active duty servicemen have a strange way of showing their hatred.
“President Barack Obama’s campaign has received almost double the amount of military donations that Mitt Romney’s campaign has, according to data collected by a research group that tracks money and lobbying in U.S. politics.”
Only a person with no sense of humor would fail to recognize that I was joking and parodying extremist conspiracy theories.
Actually this country is in excellent shape. We have lower unemployment than Europe, our currency is strong, our stock market is healthy, our workers are highly productive, and we are without serious foes.
What we have is a big bulge of baby boomers who are nearing or already in retirement. That will definitely be a challenge for the budget, for employment and for things like housing. But we’ll adapt. Chin up. Fortunately we passed on the incompetent presidential candidate and stuck with the good one.
Now, don’t you feel better?
Michael, it’s Armistice Day. Put down your weapons. We can all agree the best way to honor veterans is to refrain from needlessly making more of them.
I served 6 years in the Army, and I am happy with the result. Back on topic, maybe Rass will fade after their bad performance, or, better yet, re-evaluate their method and get better results next time (not holding my breath). I have better hope for Gallup to either just scrap their LV method or seriously revamp the model. Better polls should result in better representation, if those either elected or running for office listen to them as part of their decision/policy process. Note I sad listen, not necessarily follow them to the letter. We hire these people to make smart decisions for the country, and good polling should help with that.
Let’s see your proof of voter fraud on the Democratic side. If you can’t post a link, it didn’t happen.
Republicans are the only ones who actually committed voter fraud in 2012:
As a veteran, I find the attitude that some how a veteran’s voice is more important than that of the non-veteran offensive. We served a country that has always believed civilian control of the military is vital. I honor your service, but that service does not give you any more importance or better insight into what is best for the country.
So your concept of freedom is that I have to vote the way you want me to because you were once in the military? My dad dropped out of college to enlist during a shooting war. My uncle was in the very worst combat in Korea. They did not present a bill for services rendered after they left the military.
You poor thing…you obviously suffer from Pauline Kael Syndrome…I humbly suggest you seek help immediately…
What an interesting statement, especially in light of the fact that so many of the Republican leaders that I assume the writer approves of did everything they could to avoid military service when it was their time to serve…
shrug don’t know what to think
The exceptionally stupid claim made in that Front Page article is addressed here.
It’s easy to find out how many people actually voted in that county, and it’s not even close to what’s claimed in that article.
Learn some critical thinking skills. It’s obviously a bullsh!t story and can be disproven in ten seconds simply by looking up the number of votes actually cast. Seriously, you don’t think the Republican governor and secretary of state in Ohio might have noticed and said something? I mean, come on, don’t “shrug” think.
As someone said recently:
This is yet another example in a very long series.
Actual accurate results? There is no money in that. If I see accurate results that tell me that the candidate I want to win is actually projected to lose…and lose badly? I am not going to throw a billion dollars of good money after bad for a result I don’t want. I would much rather be lied to.
I would not want to be Karl Rove right now though…he should be on CNBC’s American Greed for fleecing the Republican voters.
Harry, do you like follow any news at all?
I vote GOP because because Democrats are not only stupid but dangerous.You bring up abortion like its some game and then you say Amnesty for immigrants leaving illegal out of the sentence.
Democrats are not only stupid but dangerous I says.
I”ll say it again Democrats are not only stupid but dangerous.
I can show crap loads of stories about way over 100 percent voter turn out in states that don’t have voter I.D. states that went to Obama. but you won’t believe them.lol check what happened in Philly… you won’t believe it…
Voter fraud aside I lost because of this and yet, voter fraud is because of this http://www.youtube.com/watch?v=UQV_0nx5CuU&feature=share
lol good golly…
Reagan’s sunny optimism is dead in Republican/conservative circles…now it’s all about badmouthing the country and talking about what horrible shape it is in so Republicans can retake power…so pathetic…
Watch out what you say here, lots of commies in disguise that are up to no good. I just put in my petition for texas. Those that wants to live in the blue state, have fun with your leader. May God help you and Bless you all. I just don’t want to live in a country that will keep spending and spending, and printing, and printing like no tomorrow without stopping. It has to stop somewhere. Unless you want China to help you out. I just don’t know where or how our country is going to pay for all that debt, seriously. Btw, there probably isn’t going to be a big military if obama is going to try to shrink it some way or another just so we can have a weaker country.
I bet this thought never, ever crossed your mind prior to 1/20/09, even though Reagan tripled the debt and GWB almost doubled it. I bet you also don’t know that 75% of debt Obama inherited was created by three presidents: Reagan, Bush and Bush. I bet you also don’t know that the biggest deficit ever was in a budget created by GWB, not Obama.
Blue states subsidize red states. One of the many reasons Texas won’t be missed.
how in hell did I get caught by the spam?