Outside the Beltway

Polls Are Imperfect Criteria For Determining Debate Eligibility, But They’re The Best We Have

Candidates who have been excluded from tomorrow's Fox Business Network are complaining, but their complaints ignore the fact that polling is the best objective criteria we have to determine debate eligibility.

Doug Mataconis · Monday, November 9, 2015 · 15 comments

On Thursday, Fox Business Network announced a lineup for the next Republican debate that pushed Chris Christie and Mike Huckabee down to the undercard debate and kept Lindsey Graham, George Pataki, and Jim Gilmore out of both debates completely because they failed to meet the network’s previously announced criteria. Not surprisingly, that decision is being criticized both by those excluded and by others:

 Fox Business Network is under fire for its selection of polls used to determine the lineup of next week’s GOP debate, after bumping two candidates off the main stage and excluding a sitting U.S. senator and a former three-term governor of New York altogether.

The networks and the polls they choose to use have had outsize impact on the presidential race, with critics saying they are artificially splitting the unwieldy Republican field into top-tier and lower-tier candidates, and making a big statement with whom they completely freeze out of the events.

Fox Business is no exception. New Jersey Gov. Chris Christie and former Arkansas Gov. Mike Huckabee for the first time were downgraded to the undercard debate next Tuesday after failing to get an average of 2.5 percent in four polls selected by the network. And South Carolina Sen. Lindsey Graham and former New York Gov. George Pataki were for the first time excluded because they weren’t able to register 1 percent support in any of the four polls.

At the crux of this most recent controversy is which polls Fox Business used. Of the four surveys, two have the campaigns and other observers crying foul: NBC News/Wall Street Journal and Investor’s Business Daily/TIPP. They say that had Fox Business used alternative and — in some people’s view, more credible — polls, Huckabee would still be on the main stage, and Graham would still be suiting up for the event.

Graham’s campaign on Thursday said it was “sincerely disappointed” — and on Friday added that it had complained about the polls used to decide the participants on each stage.

“I don’t think ‘contest’ is the right word, but we have raised our concerns about the NBC/WSJ poll’s inclusion and I hope they will see this as something similar to reviewing a bad call that is easily reversible,” Graham campaign manager Christian Ferry told POLITICO on Friday.

Fox Business pushed back on any criticism, stressing that it accepted the four most recent live-interview polls, without discrimination. But the network’s criteria for inclusion — like those if its affiliated cable-news station, Fox News Channel — were less than clear when the network rolled them out last month. In order to make the main stage debate, the channel said, candidates must achieve “2.5 percent or higher in an average of the four most recent national polls conducted through November 4th, as recognized by FOX Business to participate in the 9PM/ET debate.”

Which polls are “recognized” by Fox Business? “Such polling must be conducted by major, nationally-recognized organizations that use standard methodological techniques (i.e., live interviewers, random digit-dial sampling techniques and include both landlines and cell phones),” the network said.

But an analysis of the polls that were — and were not — included shows some of the candidates may have legitimate cause to quibble with Fox Business’ decisions. If either or both the NBC or IBD polls were excluded, Huckabee would have made the main stage, and Graham would be in the undercard event.

There’s an extended discussion at the linked article about some of the differences between the polls that were used to determine who got into which debate, or if they got into a debate at all that’s worth reading if you’re interested in the subject. The short version of the analysis, though, is one that we’ve discussed before, namely the fact that one should approach any set of poll averages with caution in no small part because different pollsters conduct their polls in different ways. Questions are worded differently, sample sizes may or may not be weighted differently, and pollsters may approach respondents who don’t seem committed to any candidate firmly differently. On that last point, for example, it’s becoming more frequent for pollsters to ask respondents for a second choice after asking who they support, or to ask if they are firmly committed to their candidate or see themselves possibly changing their minds before the time to vote arrives. Other pollsters may instruct their poll takers to push noncommittal respondents to indicate which candidate they might be leaning toward. Because of all that, taking the top line numbers from a group of polls, adding them together and then averaging them out can sometimes be like mixing apples and oranges.  This isn’t necessarily a big deal when you’re just looking at the poll averages to get an idea of where the trend lines are going over a period of time, but it becomes somewhat of a bigger deal when the averages are used to decide who does and doesn’t get invited to a debate.

This problem with the way debate eligibility is being determined leads Dan Balz to wonder why polls should be used to determine debate eligibility at all:

If the stage cannot accommodate everyone, what is the right number for a debate and how is that determined? Are eight candidates really better than 10? Wouldn’t four be better than eight? In that case, why not Trump, Carson, Rubio and Sen. Ted Cruz of Texas — the two leaders and the two candidates who are drawing attention as their poll numbers increase?

Is it better to use national polls or those from states like Iowa and New Hampshire, where the candidates are spending the most time? The stratification of the GOP field looks somewhat different in the state polls than in the national polls, and it varies between the two states. Another question: Should the big field be split in half, randomly, for each debate night, providing a different mix and therefore less predictable exchanges among the participants?

For those sponsoring debates, there is a natural tendency to seek ways to limit participation — and not just for logistical reasons. Network sponsors are looking for the kind of on-stage chemistry that will produce political drama and fireworks, high ratings and big revenues. That argues for including fewer candidates but comes at a cost of not giving the public a look at as many as possible.

It’s easy to use polls as a proxy for the public to rationalize cutting out a few candidates with dismal numbers. But on this, polls are imperfect and the cut points arbitrary. The first contests are still almost three months away, in a cycle in which conventional wisdom has taken a beating. Who should be picking winners and losers at this point?

Balz does raise some fair points here. For one thing, relying solely on national polling in this context doesn’t necessarily make sense given the fact that primaries are a state by state affair. A candidate who is doing well among Republicans in a state like Iowa or New Hampshire may not see that success reflected nationally, and thus would be penalized notwithstanding the fact that they could be a factor in early races. One way that this could be handled would be for debate criteria to include both national and state polls in the relevant primary states, which at this stage of the race would be early primary and caucus states like Iowa and New Hampshire. If you look right now at the national poll average compared to those from Iowa and New Hampshire, there isn’t much of a difference, but there are a few significant ones. Chris Christie has been moving up in recent national polling in the Granite State to the point where he is doing better than Rand Paul and, in the most recent poll, outperformed Carly Fiorina, Ted Cruz and Jeb Bush. In Iowa, Louisiana Governor Bobby Jindal is doing substantially better than he is anywhere else in the country and currently stands in seventh place ahead of Huckabee, Christie, and Rand Paul. Mike Huckabee is also doing better in Iowa than he is nationally. If Fox Business applied its 2.5% cutoff to the state polling as well as the national polls, then Christie, Jindal and Huckabee would have a strong case for inclusion in the main debate. Doing this, of course, would mean more people on the main stage which is not exactly the best way to improve debate quality, but it’s arguably a fairer way to use polling to determine debate eligibility at this point in the race.

There have been some suggestions that polling shouldn’t be used to debate eligibility at all, but then the question becomes what criteria you do use. If the networks were willing to use some kind of multi-debate format that featured every candidate in a field as large as the one the GOP has this year, perhaps that would solve some of the problems, but even you have to decide who gets invited. If you look at the FEC records for the people who have actually filed a declaration of candidacy for the Republican nomination, the list goes beyond just the fifteen people we’re familiar with, seventeen if you count Rick Perry and Scott Walker who have already dropped out. If you aren’t going to use polling to determine who gets invited, then how do you tell any of those candidates that they’re not invited? It’s worth keeping in mind in this regard that Federal Election Commission regulations require that debate sponsors establish “pre-established objective criteria to determine which candidates may participate in a debate.” Polls may be imperfect, but the reputable ones are generally objective, and I’m not sure what other objective criteria a debate sponsor could use to determine who should and should not be invited to a debate.