Online Polling and Margin of Error
Mark Blumenthal has an interesting discussion of the accuracy of opt-in, online surveys as compared to those of traditional, telephone-based polls. While he takes issue with those who use the term “margin of error” with respect to the former, he nonetheless argues that they are becoming much more reliable.
I share Blumenthal’s concerns about opt-in surveys, and not just because my wife’s livelihood is tied to a polling firm that relies on traditional methodologies. Still, he’s right that survey research increasingly relies on weighting to control for demographic skews inherent in land-line telephone surveys and growing non-response rates. At some point, telephone surveys become essentially opt-in, too.
It wouldn’t surprise me if online polls caught up to and even supplanted telephone surveys in the not-too-distant future. My expectation is that firms, such as the one that employs my wife, will simply adapt. The art of interpreting polling data is only marginally tied to the technical means by which it is collected.
I suspect that you are right-the firms will try to adapt to using online polls, and there was a time when telephone surveys were a bust, because telephone use wasn’t saturated enough.
I think the one thing online pollsters have to be wary of, and work to combat is poll stacking, where a people of certain opinions decide to vote, revote, and revote some more. With telephone polls, the pollster calls the person, and they can’t keep calling back to support their opinion with more votes.
How can you get a random sample with an online survey?
just_me: Online research surveys are not like the CNN front-page poll. They usually provide a passcode to targeted respondents, who can then take the survey once and only once.
Triumph: Tying into the above, telephone and internet polling usually get a list of potential respondents from a sampling agency. These lists can be tailored to a specific demographics breakout to match the demographics of the target market. This isn’t usually an opt-in scenario, since most of the data probably comes from purchased user lists from other companies they did business with in the past.
I have worked in this industry before, and what online polling lacks in accuracy per response, they make up in volume of responses. A 1000 respondent online poll can be as accurate as a 500 respondent telephone poll. Also, having worked in the industry, I wouldn’t trust the accuracy of any telephone poll where I wasn’t able to monitor the interviewers and get the raw results. Frankly I’m surprised that James’ wife’s firm hasn’t already moved into online polling, how can they compete on cost? Or do they sell based on that nebulous “quality” attribute?