The Beginning Of The End Of Polling?
The response rates for opinion polling of all types has become incredibly low.
The Pew Research Center reports that pollsters are finding it much more difficult to get people to respond to their surveys:
For decades survey research has provided trusted data about political attitudes and voting behavior, the economy, health, education, demography and many other topics. But political and media surveys are facing significant challenges as a consequence of societal and technological changes.
It has become increasingly difficult to contact potential respondents and to persuade them to participate. The percentage of households in a sample that are successfully interviewed – the response rate – has fallen dramatically. At Pew Research, the response rate of a typical telephone survey was 36% in 1997 and is just 9% today.
The general decline in response rates is evident across nearly all types of surveys, in the United States and abroad. At the same time, greater effort and expense are required to achieve even the diminished response rates of today. These challenges have led many to question whether surveys are still providing accurate and unbiased information. Although response rates have decreased in landline surveys, the inclusion of cell phones – necessitated by the rapid rise of households with cell phones but no landline – has further contributed to the overall decline in response rates for telephone surveys.
A new study by the Pew Research Center for the People & the Press finds that, despite declining response rates, telephone surveys that include landlines and cell phones and are weighted to match the demographic composition of the population continue to provide accurate data on most political, social and economic measures. This comports with the consistent record of accuracy achieved by major polls when it comes to estimating election outcomes, among other things.
(…)
The study is based on two new national telephone surveys conducted by the Pew Research Center for the People & the Press. One survey was conducted January 4-8, 2012 among 1,507 adults using Pew Research’s standard methodology and achieved an overall response rate of 9%. The other survey, conducted January 5-March 15 among 2,226 adults, used a much longer field period as well as other efforts intended to increase participation; it achieved a 22% response rate.
Most political pollsters will tell you that a poll conducted over a longer time period tends to be less reliable than one conducted over a two or three day period. This is because the longer the poll is in the field, the less representative it becomes of public opinion at a given point in time. A poll conducted over at two month period may end up with a 22% response rate, but it’s not going to be very valuable to analysts or other interested in figuring out what might be going on in the minds of the electorate. That’s why the typical political poll is usually only in the field for two or three days at the most. The only exception you’ll see to that generally will be tracking polls, which are a different animal from regular political polls in many ways anyway. So, if you’re going to conduct a political poll, you have to accept the fact that you’re going to get about 9% response rate, and plan accordingly.
As Pew notes, there have been several reports in recent years of pollsters having difficulty getting responses to polls in the way that they did in the past. Thanks to Caller ID and cell phones, many people screen calls and decide not to answer those from numbers they don’t recognize. Additionally, it seems fair to say that Americans have become more concerned about their privacy and less willing to answer questions from some stranger who could be half a country away for all they know. What this usually means for pollsters, of course, is that they have to find a way to maintain an acceptable sample size with the correct demographics, and the usual solution to that is to start out with a larger list of phone numbers to call. Of course, this increases costs to the pollster and, perhaps, brings the integrity of the poll itself into question.
Pew notes in the survey that, despite the low response rates, pollsters are still able to come up with representative samples if they properly balance the demographic factors, especially those such as age, race, and gender that tend to have a very strong influence on political opinions. The question that this raises, though, is whether we’re reaching a sea change in the polling industry where polling by telephone, whether cell phone or land line, is going to become less and less reliable. One solution that some pollsters, like Rasmussen and PPP have come up with is to use automated response polls. However, as I noted last month, those polls completely leave cell phones out of the mix because they are not allowed to call them. This potentially results in non-representative samples as more and more people go to a cell phone only lifestyle. The obvious next step for polling, of course, would be to go online, and there are some polling companies that use this method. However, as many have noted, they tend to be less reliable than the telephone polls. It also requires some degree of trust that the response one is getting are reliable and truthful, and assumes that people willing to go online and answer Internet surveys are representative of the population as a whole. As we’ve seen from the “flash polls” that many media companies put on their websites, that most assuredly isn’t the case. If it were Ron Paul would be the Republican nominee for President. The pollsters using online polling do things somewhat more scientifically than that, but it strikes me that they still have the same problem.
All of this leaves us with a question, if response rates continue to fall precipitously are we witnessing the end of polling?
Would that be a bad thing? For the media yes but for the election process I don’t think so. Note, I never answer polls.
Well, it would make the job of campaign managers much more difficult for one thing. Also, it’s worth noting that polling is used for a whole host of non-political matters and Pew reports that those pollsters have also experienced declining response rates. For example, one portion of the Unemployment Report issued each month by the Bureau of Labor Statistics each month is drawn from a telephone survey of a (hopefully) representative sample of American households.
I haven’t refused any cell phone polls, but I’ve had some odd “empty” phone calls.
I think the cheesy dialers only try to connect you to a person after you say “hello” and they are understaffed for that.
I expect they could get a higher participation rate at a higher investment … but that they are making a business decision here.
I can’t say I am surprised, the amount of political polling in this country has reached insanity.
I do believe the democratic process would benefit from a ban on political polling in the weeks preceding an election, if only to get the horse race chattering class to STFU.
My time and my personal space are valuable to me. I know that getting access to my opinions is valuable for the pollster. So obviously the solution to this problem is for me to receive some kind of compensation for participating in the poll.
@Doug Mataconis: I agree but I think the problem is the number of unsolicited calls we get now. The no call list was a failure. I don’t answer a call unless it’s someone I know for the most part.
I’ve kept a land line phone almost solely for the purposes of 9-1-1 calls but may well give up on it because I’m bombarded by robo-calls from pollsters, campaigns, and charities. The percentage of incoming calls that are from people I want to talk to are vanishingly small, indeed. And I now reflexively hang up on pollsters and such simply because I’m so irate at the constant intrusion.
I’ve only been polled (by a real pollster as opposed to a push-poll) once. I was dying to respond to this poll, which was about health care at the point when the ACA was being crafted. Although I did eventually get to register my opinion, there were several huge problems with the poll that would have daunted a less determined respondent:
1. It was incredibly long. Answering it took me well over half an hour. I entirely missed having dinner with my family, which was a big deal for me. If I hadn’t cared so much about the subject, I would have hung up after the first 5-10 minutes and become a non-respondent statistic.
2. The questions were actually very complicated, involved understanding some difficult concepts, and required having a pretty narrow range of opinions on them. Primarily, I would have preferred a single-payer option but there was NO way for me to register that, only to rank various other options on a scale of 1-5.
I have no idea whether being polled is normally such a time-consuming and frustrating process, but if it is, I can well understand that 9% statistic. Which, incidentally, is rather shocking. Who are these 9% and why do we think they represent all of us?
It’s interesting to layer this with the other meme around here of poll-skepticism as “poll denialism.”
I am actually coming around to the notion that anything that makes people trust polls less is probably a good thing. The increase in efficiency, and Taylor’s writing on the subject, has me concerned about polls becoming a determinative part of the process rather than a measuring of where we are.
We could well run into a situation where an election that would otherwise go one way narrowly goes instead another way because people are influenced by polling. If this makes that less likely: good.
Again, at some point they’re going to have to create some method of paying people to take polls.
The other problem with phone polling is its “interrupt” nature. You can’t stick the poll to the side for a few hours and then answer it when you have time, but rather you have to drop whatever you’re doing and do it RIGHT NOW.
@elizajane:
Which of course, is probably introducing a self selection bias to the poll: only people with a specific interest in the results of the poll are bothering to spend half an hour to finish the poll.
Or they could be down a the local union hall developing a list of people to go after for having the wrong opinions. I think it is short-sighted to not consider the impact of what the gay activists did in California when they didn’t get their way. They sued for donor lists, etc. of the opposition then mounted personal harassment campaigns. In such an environment, it is best to keep you opinions to yourself, or at least within your trusted circle. This goes double for individuals who live in deep liberal areas who might be straying from the party line. They have to live and go to school functions with people who are vindictive against non-believers.
This election seems becoming about more than the politics. If the polls prove unreliable, then the polling companies will be hard hit just as their big election season customers start realizing they’ve got to cut back since the public no longer considers the media anything of value.
I’m not saying the current polls are unreliable. I’m simply saying they could be given several factors that are starting to intersect. A black swan could happen.
Another point to add to this debate: there’s a lot of research going into ways of measuring public opinion by methods like monitoring Twitter. It may be the future is less direct polling and instead using indirect measures more accurately.
@Stormy Dragon: That sort of thing strikes me as too easily manipulated. (Which is one of my problems with “Though Shalt Not Doubt The Polls” even if they are likely to be correct…)
If ‘Unknown Number,’ private callers, or calls from numbers I don’t recognize don’t leave a message, its not important.
Nah. Polling is big business. Polling companies have ties to politicos and to media companies, which in turn also are tied to politicos. Like any big business with direct and indirect ties to politicos, especially of the Democrat variety, they’ll figure out a way to stay in business, by hook or by crook if necessary.
Plus there’s the P.T. Barnum principle. There are a lot of poll junkies out there. Like any set of junkies they need their fixes. Wouldn’t matter if the polls are not even ship-to-shore close to being accurate. Poll junkies still will tune in. They’ll still sign on. They’ll still obsess over polls. That brings in not only subcription fees it brings in ad dollars too. Consulting fees. Etc.
Ergo polls still will be with us, even if and when the only people responding to pollsers are the voices inside the pollsters own heads.
Oops, “pollsters,” that is. More coffee, please.
@Trumwill:
Indeed. And the reputation of the firms will likely end up being largely based at how good their algorithms get at detecting and eliminating manipulations.
Hmm, I talked about this here a bit back in May (and it’s a trend I’ve been noting for many years).
First, as long as pollsters can come up with accurate results most of the time, there will be no pressing need to change. As soon as they start hitting off the mark consistently and no amount of jiggering the demographics fixes it, they will need to start getting creative.
My prediction is that will happen sooner than later, because I think they are narrowing the response field in ways that will impact the validity of the results. If you omit all the people without landlines and those who do have them but refuse to take polls, I think at some point you are eliminating a type of person not captured in demographics, or possibly several types. Sure, there are people from all walks of life, ethnicities, and regions of the country who will be reachable and willing to participate. But do they represent the full breadth of attitudes among the populace? If not, it will start to impact results. The impact will start to be felt on a local level, where the population is much smaller to begin with and accurate polling is more difficult to achieve even under ideal circumstances.
@Tsar Nicholas:
Plus there’s the P.T. Barnum principle. There are a lot of poll junkies out there. Like any set of junkies they need their fixes. Wouldn’t matter if the polls are not even ship-to-shore close to being accurate.
Nonsense. People will stop paying for polls if they are not accurate, and likewise people will stop paying attention to polls if they are wrong all the time.
@JKB:
Or they could be down a the local union hall developing a list of people to go after for having the wrong opinions. I think it is short-sighted to not consider the impact of what the gay activists did in California when they didn’t get their way.
So you are attributing a decades-long trend of dropping response rates to polls to the fact that California activists found out who funded the Prop 8 campaign in 2008, which means that unions will start “going after” people who answer polls?
How do you type with a straightjacket on?
We have been cell phone only for about 3 years now and I don’t regret the move away from landlines at all (btw at least where I live the phone company has to provide 911 service even if the homeowner doesn’t pay for phone service-so if I keep a landline plugged into the jack, I can get 911 service if necessary).
We have gotten some polls on our cell phones, but I admit I am one of those who doesn’t answer the phone if I don’t recognize the number.
My daughter (age 17) got a call from a polling company the other day. She had to tell them multiple times that she was not old enough to vote.
@Ron Beasley: “The no call list was a failure”
Think so? Try getting a new phone number. We moved a few months back and, failing at our goal of living a landline-free life thanks to a mysterious Sprint dead zone, had a line installed. Within minutes of the connection we were getting two and three calls an hour from “businesses” wanting to sell us alarm systems and various other kinds of crap. We signed up for do not call (which we’d had on our previous phone) and the calls disappeared.
If that’s a failure, we need more like it!
@john personna:
Pretty much exactly. The predictive dialer’s prediction algorithm isn’t 100% accurate, so occasionally the called party will answer and there will be no call center agent available.
They’re not “understaffed,” they’re staffed to maximize agent utilization. So if a couple agents end up on longer-than-average calls, they won’t be available for the next connection, and the called party just hears silence. There’s supposed to be a message that plays after a few seconds, though.
@mantis:
Yes, that’s exactly it. I’m attributing decades long decline to one recent event. Just as Doug was when he mentioned people have become more security conscious.
This isn’t third grade. There aren’t simple one concept answers. The cause comes from many intersecting trends. In the future, try critical thinking before you do critical commenting.
Cell phone only is great, until the day when you leave your cellphone at work, or lose it in a cab, and then realize you’re stuck over a long holiday weekend without any kind of phone.
Though the saddest reason I still have a landline is that I have to use it to call my cell so I can find where I misplaced it in my apartment…..
@Rafer Janders: Get GoogleVoice! It sucks as a phone, but it lets you call your cell phone to find it pretty nicely.
And it doesn’t require a phone to use — just your computer. Don’t lose the computer.
@mantis: He types with his nose, we hope.
@JKB:
In the future, try critical thinking before you do critical commenting.
Who is the one thinking critically here, the person who thinks that eventually low response rates will filter out crucial parts of the population, or the the person who thinks people refuse to answer polls because union thugs will come get them?
I recently read something from one of the major pollsters (sorry, simply cannot recall the source but think it was in the last 10 days or so) along more or less the same lines. With fewer people answering calls (caller id, etc.) and response rates dropping, the liklihood of a major poll failure increases. I think he suggested that this might be the last election cycle in which we could depend on the polls (note: that was my takeaway, probably not exactly what he said)
One other thing: with response rates down to 9%, isn’t there the possibility that the people who are responding to pollsters are in some way different from those who don’t – a difference not picked up by such standard factors as race, age, gender, political party?
Without accurate polls, candidates, especially, for national office, have no way to efficiently allocate efforts and spending – which will probably increase the already excessive cost of campaigns.
I don’t know if the death of political polls would be bad or good. It might force the chattering classes to stop talking about the horse race and focus on the issues – nah. They’d just spend more time discussing the gaffe de jour.
Would be interesting to know if ‘product polling’ is similarly challenged. If so, there’s surely some work-around waiting in the wings. If not, we’ll know because there will be fewer and fewer polling companies. Until Procter and Gambel stops regarding polling as a way of budgeting millions of dollars, or the networks cast doubt on Nielson ratings, a critical eye on comparative results is as close as we are likely to get to knowing how elections are proceeding.
I don´t know. Here in Brazil the polls are less reliable(they are wrong most of the time) and it´s much more difficult to get reliable information. I miss all the information that I can get about the american public and American politics. The exit polls of American Presidential Elections are wonderfull to anyone that likes politics.
But no pollster tries to call me.(Most pollsters here get their information from people in the street).
Cell phone only is great, until the day when you leave your cellphone at work, or lose it in a cab, and then realize you’re stuck over a long holiday weekend without any kind of phone.
When we went cell phone only we got a line that is essentially the “home phone” and is always kept at our house.
So if for some reason I leave my phone somewhere or misplace it, there is always a phone at home.
@Andre Kenji:
Do you get the 24×7 “talking heads” during elections with everybody second-guessing the candidates?
(I get Globo Intl. but not Record, Band or SBT and Globo Intl. news – Jornal Nacional – is limited.)
@LC:
No. These 24×7 talking heads are mostly an American Phenomenon. Cable news is far from being popular here and the Electoral Legislation is so restrictive that´s very difficult to produce any kind of political content on TV. A show like The Daily Show would be illegal under campaign law(Seriously);
A week before the election the most astonishing thing about the polls is they aren´t saying anything. In most polls all candidates are registering low numbers. If you sum up all the numbers that all the candidates are getting in the polls for the mayorship of São Paulo you get a number far lower than 100%.
@Trumwill:
To be clear: I am not of the opinion one shalt not doubt the polls, but rather believe that one shalt not doubt them simply because they do not comport with one’s preferences. Or, more specifically: thou shalt not recalculate the results to make them fit one’s preferred version of reality.
Also: I am dubious of the notion that polling affects voting. Indeed, isn’t the current spate of poll denialism evidence that partisan ID is more powerful than polling? (Indeed, since the political science seems to indicate that a host of issues, such as advertising, conventions, and debates are weaker than voter preferences then why would polling be any different?).
Polls obviously have an impact on at least one citizen–me. I am very happy when my man or party is polling high, and very unhappy when they are polling low. If they poll too low, I simply stop reading them, thus avoiding the mental anguish for a little while.
When they poll about even, which has been the case recently, I sit and stare at the print and wonder why there are so very many idiots in the electorate: half of them, the other 50%, are obviously dead wrong, don’t you agree?
@Steven L. Taylor:
My belief is that, in the case of a systemic polling failure, it will almost always be revealed by boosters of the political party being under-represented. And so it will come across as ideological – and be ideological! – but not necessarily wrong. I think the dangers of a false positive are outstripped by the dangers of a false negative. I thought the Dems were wrong in 2004 and I think the Republicans are wrong now, but we really won’t know until election day*.
As for the basis for my concern about polling influencing turnout, that’s at least partially contingent on increased reliability. When/if we can reach the day we *know* who is going to win, I can’t believe that wouldn’t have an effect. I have difficulty believing the “might as well stay home” factor would not asymmetrically affect the “losing” side. There is a reason, I think, that the press is reluctant to release exit polling numbers while the polls are still open. Why it is not uncommonly held that east coast results affect west coast elections. Why both parties put on an optimistic front. Concern that a bad situation will be made worse if everybody admits that it’s bad.
* – Actually, we won’t even know then. The article I read in the Daily Caller suggested that the polls will collapse into something resembling the final result in the final week. This is commonly said of Democrats of Rasmussen (they exaggerate Republican performance until the last week). I don’t know if this is true or not – and again, I think the polls are likely right this time around – but if they are it’s not really provable.