• Facebook
  • Twitter
  • Subscribe
  • RSS

Ranking The Pundits

Gary Wyckoff and his students at Hamilton College analyzed the predictions of a number of op-ed columnists and TV pundits and found that most of them were bad and that conservatives tended to be wore than liberals at making predictions.

Jim Romenesko (“Claim: Krugman is top prognosticator; Cal Thomas is the worst”):

A Hamilton College class and their public policy professor analyzed the predicts of 26 pundits — including Sunday morning TV talkers — and used a scale of 1 to 5 to rate their accuracy. After Paul Krugman, the most accurate pundits were Maureen Dowd, former Pennsylvania Governor Ed Rendell, U.S. Senator Chuck Schumer (D-NY), and former House Speaker Nancy Pelosi. “The Bad” list includes Thomas FriedmanClarence Page, and Bob Herbert.

 

Press Release (“Pundits as Accurate as Coin Toss According to Study“):

Op-ed columnists and TV’s talking heads build followings by making bold, confident predictions about politics and the economy. But rarely are their predictions analyzed for accuracy.

Now, a class at Hamilton College led by public policy professor P. Gary Wyckoff has analyzed the predictions of 26 prognosticators between September 2007 and December 2008. Their findings? Anyone can make as accurate a prediction as most of them if just by flipping a coin.

Their research paper, “Are Talking Heads Blowing Hot Air? An Analysis of the Accuracy of Forecasts in the Political Media” will be presented via webcast on Monday, May 2, at 4:15 p.m., at www.hamilton.edu/pundit. The paper will also be available at that address at that time. Questions during the presentation can be posed via Twitter using #hcpundit.

The Hamilton students sampled the predictions of 26 individuals who wrote columns in major print media and who appeared on the three major Sunday news shows – Face the Nation, Meet the Press, and This Week – and evaluated the accuracy of 472 predictions made during the 16-month period. They used a scale of 1 to 5 (1 being “will not happen, 5 being “will absolutely happen”) to rate the accuracy of each, and then divided them into three categories: The Good, The Bad, and The Ugly.

The students found that only nine of the prognosticators they studied could predict more accurately than a coin flip. Two were significantly less accurate, and the remaining 14 were not statistically any better or worse than a coin flip.

The top prognosticators – led by New York Times columnist Paul Krugman – scored above five points and were labeled “Good,” while those scoring between zero and five were “Bad.” Anyone scoring less than zero (which was possible because prognosticators lost points for inaccurate predictions) were put into “The Ugly” category. Syndicated columnist Cal Thomas came up short and scored the lowest of the 26.

Even when the students eliminated political predictions and looked only at predictions for the economy and social issues, they found that liberals still do better than conservatives at prediction. After Krugman, the most accurate pundits were Maureen Dowd of The New York Times, former Pennsylvania Governor Ed Rendell, U.S. Senator Chuck Schumer (D-NY), and former House Speaker Nancy Pelosi – all Democrats and/or liberals. Also landing in the “Good” category, however, were conservative columnists Kathleen Parker and David Brooks, along with Bush Administration Treasury Secretary Hank Paulson. Left-leaning columnist Eugene Robinson of The Washington Post rounded out the “good” list.

Those scoring lowest – “The Ugly” – with negative tallies were conservative columnist Cal Thomas; U.S. Senator Lindsey Graham (R-SC); U.S. Senator Carl Levin (D-MI); U.S. Senator Joe Lieberman, a McCain supporter and Democrat-turned-Independent from Connecticut; Sam Donaldson of ABC; and conservative columnist George Will.

Landing between the two extremes – “The Bad” – were Howard Wolfson, communications director for Hillary Clinton’s 2008 campaign; former Arkansas Governor Mike Huckabee, a hopeful in the 2008 Republican primary; former House Speaker Newt Gingrich, a Republican; Sen. John Kerry of Massachusetts, the Democratic nominee for president in 2004; liberal columnist Bob Herbert of The New York Times; Andrea Mitchell of NBC; New York Times columnist Tom Friedman; the late David Broder, former columnist for The Washington Post; Chicago Tribune columnist Clarence Page; New York Times columnist Nicholas Kristof; and Hillary Clinton.

The group also found a link between conditional predictions and accuracy, that is, a prediction that was conditional (“If A, then B”) was less likely to be accurate. Finally, those prognosticators with a law degree were more likely to be wrong.

The full report, “Are Talking Heads Blowing Hot Air? An Analysis of the Accuracy of Forecasts in the Political Media,” is available in PDF form. The methodology strikes me as reasonable enough.

But, aside from the “small n” problem–that is, none of the pundits have enough predictions scored to make for a statistically interesting sampling–there are a host of issues.

Punditry is not about predictions.

Prediction is really an odd way to judge political pundits since guessing the future is a mug’s game. Television talking heads shows, especially, love to do it because it makes for entertaining dialog. The correct answer–Who the hell knows?–is not particularly interesting, so pundits who wish to be invited back hazard a guess.

The thing that separates a good pundit from a hack is the ability to analyze facts in an illuminating framework and a willingness to adjust his view as new information comes in.

The time frame is too small and biases the outcomes.

The period September 2007 and December 2008 coincides with the final stage of the Bush Administration, the 2008 presidential election, and the global financial crisis.

The last year of the Bush Administration was off-the-charts bad for the Republican Party. We would expect Democratic pundits are going to make more accurate predictions about bad outcomes for Republicans and good outcomes for Democrats. Most of these people have been blathering on the public record for years. How did they do in, say, 1994?

During the primary phase of the 2008 campaign, Democrats had two plausible candidates and the Republicans had a half dozen. The sheer odds make it more likely that the Democrats are more accurate. Additionally, the early Republican frontrunner, Rudy Giuliani, performed horrendously. Mike Huckabee, a virtual unknown when the period began, won the Iowa Caucuses and made a surprisingly strong showing.

Once the race settled down to an Obama-McCain contest, it became a binary choice prediction-wise. Obama won. Presumably, all the Democrats correctly predicted that. Until late in the game, most Republican pundits were naturally going to spin ways that McCain could pull it off.

The financial meltdown took place while a Republican was in office and was more catastrophic than most expected. We would expect Democratic pundits to predict worse outcomes under a Republican president.

Pundits make different types of predictions.

Unfortunately, the report does not show the raw data; it just presents raw numbers: How many predictions, how many right, and how many wrong. That means we can’t look at the actual predictions. Some things are obviously harder to predict than others. Some pundits are more cautious about predicting things that are hard to predict, sticking instead to relative certainties and near events rather than far. Some pundits are only asked on to talk about things within the scope of their expertise, while others are expected to offer an opinion on just about everything.

Pundits have different goals.

As noted earlier, there are thoughtful analysts who are biased but fair. That is, they have an ideological point of view, likes and dislikes, and a rooting interest in the outcome of elections, policy debates, and other things they’re talking about and yet acknowledge the strong points of the opposite argument. Others are hacks, either because they simply lack the analytical tools to think outside their narrow worldview or, more often, they’ve calculated that extreme partisanship and churlishness is the key to longevity.

But many of the “pundits” rated in the study aren’t pundits at all; they’re politicians or political operatives. These people all meet my definition of “hack,” but that’s only because an unfair framework is being applied to them. That is, they’re not in the business of presenting accurate analysis but rather of advancing a specific agenda, whether it’s their own re-election or the benefit of their party or cause. They’re propagandists by design, not analysts.

Indeed, one of the main reasons I’ve long since stopped watching the talking heads shows is because this group tends to dominate the guest list. It’s simply not interesting to hear what the designated spokesman for Position X thinks about Position X. But most of the shows in the post-Crossfire era have operated in that framework, even having hosts that play characters. The classic example of the latter was late Robert Novak, a terrific reporter and columnist who was an extreme hack as a television talking head, playing the character Bob Novak, Prince of Darkness.

An exception to this format is ABC’s This Week, which started during the George Stephanopoulos era to assemble a rotating panel of usually interesting roundtable discussants. George Will remains a regular and Paul Krugman is on most weeks as well. They’re paired with two or three guests, most of whom are quite good. Even the partisan operatives, notably Donna Brazille and Matthew Dowd, come close enough to taking off their activist hats to make enjoyable commentators. But I don’t have any idea how good they are as prognosticators; I’m just interesting in their ability to frame the discussion in interesting ways and incorporate known facts into their argument in a fair manner.

Related Posts:

About James Joyner
James Joyner is the publisher of Outside the Beltway, an associate professor of security studies at the Marine Corps Command and Staff College, and a nonresident senior fellow at the Atlantic Council. He's a former Army officer and Desert Storm vet. He has a PhD in political science from The University of Alabama. Views expressed here are his own. Follow James on Twitter.

Comments

  1. Chad S says:

    “You can’t judge pundits based on what they say-signed, another pundit” lol

    Like or Dislike: Thumb up 0 Thumb down 0

  2. John Peabody says:

    This is all well and good. But the average take-away would be: “Look at how many paragraphs of blather are needed to try to deny the fact that conservatives are always wrong!” Your best point? These shows are entertainment. A serious, thoughtful pundit will hedge his bets, stress the conditionals, and will not be as nearly interesting as the one who yells “The next two months are CRITICAL!” Pshaw. Watch some of these shows from 10-15 years ago. Those guys are awfully worked up about trvial matters.

    Like or Dislike: Thumb up 0 Thumb down 0

  3. Dave Schuler says:

    It’s a silly study. The timeframe is quite narrow: the fifteen month period from September 1, 2007 and December 31, 2008. I seem to recall that presidential primaries and a presidential election took place during that period and that the Democratic candidate won. My suspicion is that would tend to skew things a bit if all the “pundits” did was announce their preferences.

    I’d like to see a study over a much longer period, say, from January 1, 2000 through December 31, 2009 or, if you’d prefer, from January 1, 2001 through December 31, 2010.

    As you may recall, James, I make an annual practice of each year at the beginning of the year attempting to predict what will happen in the coming year. My success rate is pretty fair. However, what I’ve learned is that it’s not what you do predict and get wrong or right that’s interesting but what you don’t predict that actually occurs. I suspect that’s the case with the professional pundits as with amateur ones like me and the methodology of this study does not take that into account at all.

    One last thought: the pundits whose predictions have been checked constitute quite a narrow field: 10 liberals, 10 conservatives, 2 moderates (whatever that means) as I read the study. For all any of us know there may be an editorial writer for the Des Moines Register whose predictions have been 100% accurate and has predicted every major development for the last 20 years. The World’s Champion Pundit. He or she wouldn’t show up in this study at all.

    Like or Dislike: Thumb up 0 Thumb down 0

  4. Hey Norm says:

    I just drove through Hamilton College last Thursday. It’s a beautiful hilltop campus with some very interesting Architecture. Beyond that, this thread seems rather pointless.

    Like or Dislike: Thumb up 0 Thumb down 0

  5. Davebo says:

    I’d like to see a study over a much longer period, say, from January 1, 2000 through December 31, 2009 or, if you’d prefer, from January 1, 2001 through December 31, 2010.

    While I agree with you Dave, don’t you think such a study would make the pundits look even worse than this one?

    I guess what I’m saying is that blathering about aluminum tubes and yellow cake would seem even more obscene than it did at the time. And in the same vein, Rumsfeld and Condi have been all over the news of late.

    Some people are truly incapable of shame it seems.

    Like or Dislike: Thumb up 0 Thumb down 0

  6. “[B]y the year 2005 or so it will become clear that the Internet’s impact on the economy has been no greater than the fax machine.” — Paul Krugman, 1998

    Like or Dislike: Thumb up 1 Thumb down 0

  7. Billy says:

    “Finally, those prognosticators with a law degree were more likely to be wrong.”

    As a lawyer, this strikes me as entirely accurate.

    Like or Dislike: Thumb up 0 Thumb down 0

  8. Dave Schuler says:

    don’t you think such a study would make the pundits look even worse than this one?

    Yes, I do. Indeed, I would hope that it would. I think that strong ideological preference attracts readers to your column but isn’t particularly conducive to forthright, clear-eyed analyses of events.

    Like or Dislike: Thumb up 0 Thumb down 0

  9. john personna says:

    It’s nothing new. Tetlock wrote “Expert Political Judgment: How Good Is It? How Can We Know?” …

    oh gosh, it wasn’t that long ago. 2005.

    I guess if anyone is surprised by these results, or questions the methodology, they should read the Tetlock. This is just the way the world works.

    Like or Dislike: Thumb up 0 Thumb down 0

  10. john personna says:

    “[B]y the year 2005 or so it will become clear that the Internet’s impact on the economy has been no greater than the fax machine.” — Paul Krugman, 1998

    So how’s that job situation looking Charles?

    Seriously, economists and pundits are looking for the widespread jobs gains, that should have come with a major technology cycle.

    Sure, geeks like me did OK, but as broader demographics stagnated.

    Like or Dislike: Thumb up 1 Thumb down 0

  11. ponce says:

    I don’t think anyone really takes pundits seriously.

    Punditry is a form of entertainment, like astrology.

    Like or Dislike: Thumb up 0 Thumb down 0

  12. Janis Gore says:

    Good Lord.

    Hamilton’s tuitiomn is now $42, 000 a year. Harvard is only $39,000.

    Like or Dislike: Thumb up 0 Thumb down 0

  13. [...] From James Joyner: “Gary Wyckoff and his students at Hamilton College analyzed the predictions of a number of [...]

    Like or Dislike: Thumb up 0 Thumb down 0

  14. john personna — um, what? If anything you seem to be arguing that the Internet has greatly affected the economy, just not necessarily in a positive manner.

    Like or Dislike: Thumb up 0 Thumb down 0

  15. john personna says:

    I just want to give Krugman partial credit, because most people would have had a more-wrong guess about the Internet in 1998. Traditional thinking was that tech boom brought a sustained economic boom. Not a dot-com crash and jobless recoveries.

    In fact the economists I’m reading are only now registering that each internet boom cycle (now social media) brings fewer jobs.

    And they’re wondering if net-tech won’t do it, what will.

    Like or Dislike: Thumb up 0 Thumb down 0

  16. john personna says:

    On this, a particularly harsh paragraph by Felix Salmon:

    On top of that is the fact that US workers are massively overpaid compared to their equally-productive and well-educated counterparts in countries all over the world. There are a number of ways that the discrepancy can be narrowed: wages in countries from Slovenia to South Africa could go up; US wages can go down; or the dollar can simply depreciate. Which is a lot easier than nominal or even real wage cuts.

    That is in The real reasons for the dollar’s decline

    Like or Dislike: Thumb up 0 Thumb down 0

  17. [...] Ranking The Pundits (outsidethebeltway.com) [...]

    Like or Dislike: Thumb up 0 Thumb down 0

  18. [...] About the Framework  Outside the Beltway's James Joyner says. "The thing that separates a good pundit from a hack is the ability to analyze facts in [...]

    Like or Dislike: Thumb up 0 Thumb down 0

  19. Nitpicker says:

    If the “thing that separates a good pundit from a hack is the ability to analyze facts in an illuminating framework and a willingness to adjust his view as new information comes in” then conservatives would do even worse than this study suggests. After all, the top conservative pundits have the same initial answer for everything. Budget looking good? Tax cuts. Huge deficits looming? Tax cuts? Years of relative peace? Tax cuts? Two wars? Tax cuts. Hack hack hackity hack.

    Like or Dislike: Thumb up 0 Thumb down 0