Gallup Tries To Explain Why Its Polling Was So Bad In 2012
Gallup is one of the oldest names in polling, but it’s 2012 polling turned out to be among the most inaccurate of all the professional pollsters. On more than one occasion, its Daily Tracking Poll claimed to show the race tightening and movement in Mitt Romney’s direction that was not being reflected in any other poll with the exception of Rasmussen, which was also among the least accurate polls of the 2012 cycle. Ever since then, Gallup has apparently been conducting an internal review to attempt to figure out what went wrong, and they recently shared some of their findings with the public:
The review, led by Mr. Newport and Michael Traugott of the University of Michigan, was conducted by a team of statisticians, methodologists and analysts from Gallup, assisted by outside consultants. After identifying more than 20 possible elements that could have been related to the Republican skew in Gallup’s polls, the team isolated four factors as the likely causes.
Gallup’s model for identifying those most likely to vote — a series of seven questions — seemed to have failed in 2012, and the organization is re-evaluating its formula for ranking voters who will turn out.
Just as technology has changed the way campaigns work, it has altered the way survey researchers gather data. A change in the way that Gallup called respondents on land lines may have been a contributing factor that led to its sample to be older and more Republican. Half of their respondents, however, were reached on cellphones — a proportion that is at or above industry standards.
In addition, there were too many respondents from certain time zones. In the South and the Midwest, there were more respondents from the Central Time Zone, which tends to be more Republican, than the Eastern Time Zone, which skews Democratic.
Finally, the way Gallup asked callers about their race overrepresented some groups.
Individually, none of the four problems is considered to have been a significant factor in the results, but taken together, they helped lead to an overestimation of Mr. Romney’s share of the vote. Gallup plans to use the governor’s races this fall in New Jersey and Virginia to test new election polling methods.
Given that polling is what Gallup does to earn money, it’s in their best interests to make sure that the models that they are using are as accurate as possible. Gallup doesn’t really make money off the political polling that gets released to the public, but they do make money off of the private polling that they do, mostly for corporations, trade associations, and similar organizations. If the accuracy of their public polling is called into questions, that’s likely to raise concerns among private clients that could cut into revenues. Therefore, this kind of review is in their interests. Interestingly, though, I have yet to hear of a similar review being done by Rasmussen.
I look forward to a visit from Smooth Jazz to tell us that Gallup was really correct and Mitt Romney is in fact president.
Was the concept that Gallup’s senior leadership really really really wanted Mittens to win anywhere in that list of elements?
Was being a full-fledged member of the Far-Right-Entertainment-Complex one of the reasons listed???
I’m guessing Rasmussen’s private clients are paying for the push polls it’s doing….
Accuracy isn’t needed for what Rasmussen wants to accomplish, what’s needed is quite the opposite…
There is nothing like wishful thinking! It appears Gallup were wishing like crazy that Mitt Romney was actually ahead and going to win the presidency. Maybe they should take up another line of work. Obviously numbers and percentages are not in their wheelhouse.
What’s bizarre is that I don’t recall Gallup being an especial outlier during the cycle—just at the very end. But, aside from a brief spike after the 1st debate, there was never a point when it was rational to think Romney had any chance at all to win.
I, for one, am extremely grateful that Gallup caused Republicans do ignore all reality based polling, and caused people (like my father, and most of my 8 brothers and sisters) to believe that Mitt Romney was coasting to an easy victory in November. I’ve tried to minimize the gloating, I really have.
@James Joyner: James, see http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/
That chart includes 11 polls by Gallup over the last three weeks. Their error was was +7.2% in favor of the GOP; the next was at R +4.5; then Rasmussen at R +3.7.
When you are 3.5% more GOP than Rasmussen, something’s wrong.
BTW, ISTR Gallup being off for a long time, perhaps a couple of months, but I don’t have time right now to find the link.