Are Media Polls Criminally Bad?

Charles Franklin takes issue with Stan Greenberg’s contention that media polls are “criminally bad” because of a variety of reasons, especially that “They are not carefully weighted and, as a result, show wide swings in voter preference that the media interpret wrongly as voter fickleness.”

I’d be curious what constitutes careless weighting. Almost all pollsters weight the data to demographic distributions derived from the Current Population Survey (a huge monthly government survey with over 90% response rate and therefore considered particularly reliable.) Pollsters might differ on some technical issues here but it is hard to believe that media polls are that different from Greenberg’s own methods. None of the weighting techniques are in any way secret– just buy a textbook on survey sampling or read the journals or attend panels at AAPOR (the pollsters conference) and the variety of options are all right there in the public domain. So there is little reason to think that variation in weighting practice is due to either secret knowledge that Greenberg and colleagues have that is unavailable to others, or that “media pollsters” systematically choose to be reckless by using poor weighting schemes.

[…]

My bottom line is that polling techniques and methodology are “open source.” Take classes in grad school and you can learn all the theory. Work in a polling firm and you’ll also learn a lot of practical wisdom. Survey professionals all have access to this. Greenberg may well think that his polls are superior to those conducted by others, but I’d disagree that this has anything to do with secret knowledge or methods.

I’m dealing with an excerpt of Greenberg’s argument, which is made in a book I don’t have access to, so it’s hard to assess his claim.  Certainly, media polls conducted early in cycles tend to be rather sloppy, in that they seldom apply “likely voter” screening.  They also tend to vary wildly from one another, which would seem to indicate that either 1) the people running them didn’t attend graduate school or AAPOR conferences or 2) polling is really hard and simply applying the basics isn’t sufficient.

Even late in cycles, when most of the national polling outlets switch to likely voter models, there’s wide variation in the numbers.  That’s not a big deal in races, such as the most recent presidential contest, where the margin is wide.  In close races, though, they’re virtually worthless.

Full disclosure: My wife, as I’ve mentioned more than once, is the COO of Public Opinion Strategies, the major Republican polling firm, which partners from time to time with Greenberg on, oddly enough, some media polls (for NPR and MSNBC).

I’d be interested to see the comparisons between the polling of Greenberg, POS, Gallup, and the various major media polls and electoral outcomes.   My strong suspicion is that the first two are closer in alignment with the thing they’re trying to project than the others, especially on a sustained basis.

Link via Political Wire.

FILED UNDER: Public Opinion Polls, , ,
James Joyner
About James Joyner
James Joyner is a Security Studies professor at Marine Corps University's Command and Staff College and a nonresident senior fellow at the Scowcroft Center for Strategy and Security at the Atlantic Council. He's a former Army officer and Desert Storm vet. Views expressed here are his own. Follow James on Twitter @DrJJoyner.

Comments

  1. So, is Mr. Greenberg trying to criminalize political differences? Or has the hyperbole becoming a bit too hyperbolic?

  2. tom p says:

    I am shocked, SHOCKED!!! that a poll can be slanted.

  3. Steve Plunk says:

    One of the worst is the old “is the country headed in the right or wrong direction?” polls. The conservative can say it’s headed in the wrong direction for one reason while the liberal can say it’s headed in wrong direction for some totally different reason. The result is we’re going to hell in a hand basket but the right direction would be two very different things. Worthless.

    Polls have become a cheap headline with underlying value. Journalism at it’s worst.

  4. Brett says:

    I think we’d all do better if everyone remembered exactly what polls are – they’re a snapshot at a particular instance. They don’t necessarily imply anything permanent about American political opinion unless a trend holds on the same type of question asked for a long time.

  5. But, ultimately, didn’t the polls predict the outcome of the last election quite well? Remember all the talk from conservatives about how the polls were biased and that McCain was really gonna win? I think even James — on the basis of insights gleened from his wife’s firm — signed on to a version of that argument at one point. And yet, the final result was pretty much right in line with what all the polls of polls sites — RCP, Pollster, 538 — said it would be.

    Sure, individual polls are subject to random error as is predictable by anyone who understands basic statistics, but ultimately I am on Franklin’s side here and see Greenberg’s arguments as a effort to privilege his own “expert” and “insider” knowledge over polling results. This is simply an effort to erect barrier to entry into what is a lucrative business. But ultimately, it ain’t rocket science. The basic techniques are well known, and the interpretive methods are not all that complex. Political operatives hate to think that “amateurs” can get equally good results, but my guess is that they can.

  6. James Joyner says:

    Bernard: McInturff had McCain losing. He merely put out a memo which I discussed here arguing that if a number of things went the right way, the race would be too close to call. My analysis was more than skeptical that things would in fact line up for McCain and predicted an Obama win.

    But, yes, I think the “poll of polls” approach is a pretty safe bet. I wouldn’t rely on any of the individual media polls, though, in a close contest.

  7. Bithead says:

    But, ultimately, didn’t the polls predict the outcome of the last election quite well?

    As a logical point, one supposes the answer to the question rests on our ability to trust the voting process.