Outside the Beltway

How Much Should We Trust Those Jobs Reports?

Doug Mataconis
27

Comments

Buzzfeed’s Willie Herrmann points out that the Bureau of Labor Statistics has been doing a lot of revising of its employment reports lately:

A full 50% of the time, the initial perception of the jobs numbers would have been incorrect: Reported jobs growth exceeded economists’ expectations, but then the revised numbers actually fell short of them, or vice versa. In other words, if you want to know what today’s jobs report means for the economy, reading the headlines is no more useful than flipping a coin.

Just take a look at the chart:

enhanced-buzz-23694-1362761219-8Months in red are the months in which the initially reported number either fell short of or exceeded economists’ expectations but then the revised either exceeded or fell short of those expectations.  Leaving the expectations issue aside, though, it seems pretty clear that we probably should be more skeptical about those initial Jobs Reports.

Categories

Tags

, ,

27 responses to “How Much Should We Trust Those Jobs Reports?”

  1. michael reynolds Avatar
    michael reynolds

    Well, we should be skeptical unless it’s bad news. Then we should rush to embrace the bad news as proof that we should immediately return to the gold standard and begin digging bunkers and hoarding guns.

  2. Scott Avatar
    Scott

    Month to month horserace reporting is entertaining but not really useful. Use of trend analysis, such as a moving average, is more accurate.

  3. anjin-san Avatar
    anjin-san

    Why don’t you show comparable data for the Bush & Clinton years? Are the revision rates higher? Lower? Roughly equal? If you are not willing to do that, this is a joke, like pretty much everything you say about the economy.

  4. Scott Avatar
    Scott

    Oh, and BTW, all the revised figures, except for Jul 11, are better than the initial. So much for Obama and the Commerce Dept cooking the books.

  5. Gromitt Gunn Avatar
    Gromitt Gunn

    The BLS hasn’t been revising a lot of its reports lately – it revises all of its reports for 12 months after initial release as a part of its methodology. That’s why the Buzzfeed analysis stops at March 2012 – it is the most recent month to have its revision period closed out.

    Its also important to note – which the Buzzfeed article does not – that of the twelve months that it chose to highight – the ending number was higher than the beginning number in 11 out of 12 months. Why is that important? It is important because it reveals that almost all of the time, the initial job growth report is ultimately revised *upward.* This demonstrates a) independence on the part of the BLS, since – as you note – only the initial numbers get full press coverage and were the BLS politicized, there would be pressure to overinflate initial growth numbers, and b) the BLS’s model is built with an emphasis on conservatism, with a propensity to underreport its initial findings.

  6. john personna Avatar
    john personna

    @michael reynolds:

    Well, we should be skeptical unless it’s bad news. Then we should rush to embrace the bad news …

    I’ve got to admit that when I saw the jobs data this morning I did spend a moment or two wondering if Doug would even cover it. It did not fit his drum-beat.

    Well, I did not see his endorsement of the long term averages coming! I’ve been pointing to those same long term charts for a year now. They never seemed to matter when the news of the week was bad.

    Imagine, a come to Jesus moment .. driven by a good jobs report.

  7. Surreal American Avatar
    Surreal American

    If it’s a jobs report under a Republican Administration, it doesn’t matter because the White House is considered the God-given property of the GOP.

    If it’s a jobs report under a Democratic Administration, especially under a Kenyan Muslim Usurper Administration, it’s always a Harbinger of Doom and you should place all your investments in gold, can goods and shotguns.

  8. Tom Strong Avatar
    Tom Strong

    The difference between the initial reports and the revised reports is +900,000 jobs.

    I’m no Pollyanna with regard to our current economic situation, but that seems like something worth noting.

  9. Tom Strong Avatar
    Tom Strong

    Also:

    Months in red are months where initial headlines said the jobs numbers fell short of economists’ expectations, but then the revised numbers actually exceeded expectations (or vice versa).

    That’s from the Buzzfeed article. It’s not great chart design, but that explanation makes more sense.

  10. matt bernius Avatar
    matt bernius

    @Tom Strong:

    It’s not great chart design, but that explanation makes more sense.

    Not great design?! It’s absolutely aweful design — especially the choice of red to mark the off months. Regardless of author intention, for most westerners, that immediately conjurers up debt (i.e. in the red). That’s only magnified by the fact that this is about financial numbers.

    And the analysis isn’t all that much better. The issue isn’t that it’s wrong 50% of the time, but that the formula that it uses is clearly being conservative. Now admittedly, being too conservative is problematic as well (and the administration would have profited from having the numbers right when they came out).

    But suggesting that it’s a coin flip as to whether the number will be accurate misses the larger point.

  11. Argon Avatar
    Argon

    “How Much Should We Trust Those Jobs Reports?”

    More than Dick Morris, Karl Rove, Steve Doucy and Charles Krauthammer on their best days ever. Certainly more than most Cardinals attending the Conclave. Far less than evolution and general relativity.

  12. matt bernius Avatar
    matt bernius

    From the buzzfeed article:

    A full 50% of the time, the initial perception of the jobs numbers would have been incorrect: Reported jobs growth fell short of economists’ expectations, but then the revised numbers actually exceeded them, or vice versa. In other words, if you want to know what today’s jobs report means for the economy, reading the headlines is no more useful than flipping a coin.

    This is really the height of bad journalism.

    I quickly cobbled together a different chart featuring the same data (with and added delta and % off column). Check out how this chart tells a different story:
    http://www.mattbernius.com/images/jobsPerceptions.PNG

    > 100% of the time from April 11 to March 12 the job numbers were wrong.

    > 50% of the time a negative jobs report was actually positive.

    > 8% of the time a positive report was actually negative.

    > 92% of the time the job predictions were revised UP (by an average of ~200% not counting the major Nov 11 outlier)!

    So the idea that this is a coin toss — at least for this period — is completely false.

    Now, as I said above you can debate if the initial calculations are too conservative — leaving off one major outlier, the final job numbers were, on average, ~163% higher than the prediction. But given the choice between constantly revising down, this doesn’t seem a bad thing.

    More importantly, based on this period, no one should ever trust the initial job numbers.

  13. commenter1 Avatar
    commenter1

    http://www.washingtonpost.com/blogs/ezra-klein/files/2012/10/revisions.jpg

    This is a chart showing revisions going back to the 80s. They fluctuate widely in both magnitude and direction. The point isn’t that the numbers are likely to be revised up – they’re not, and the sample is too small to show that. The point is that month-to-month, horserace analysis is overblown and not very useful.

  14. Tsar Nicholas Avatar
    Tsar Nicholas

    If we’re prepared to treat as gospel public opinion polls which are based upon phone calls with around 1,000 people and sampling extrapolations, I think we should be well prepared to trust surveys based upon 60,000 phone interviews (BLS household report) and 400,000 payroll reports (BLS establishment survey), along with standard sampling techniques and long-standing seasonality adjustment factors created by experts.

    Would you stake your life on these BLS reports? Obviously not. But they’re pretty f’n good approximations. Always have been. Certainly they’re more than well within the ballpark of accurate.

  15. Console Avatar
    Console

    The jobs report is nothing but a survey. It has a margin of error of something crazy like plus/minus 100,000 jobs in the first place. Even without revision, slight month to month changes aren’t the best place to focus. You need a good deal more context than month X vs. month Y

  16. rjs Avatar
    rjs

    anyone who reads the BLS technical notes knows this…the margin of error on the establishment survey is 100,000, and it’s over 400,000 on the household survey…blame the reporters, not the BLS..

  17. 11B40 Avatar
    11B40

    Greetings:

    Another aspect that never seems to draw media attention is what statisticians refer to as “statistical significance”. Any measure includes some degree of inaccuracy or insignificance, so there are any number of mathematical tests which show whether a change in a measure is of any real statistical significance. When a measure of 7% or 8% varies by a tenth of a percent or so, that may very well not be of any mathematical significance or of extremely little.

    Since the Obama administration has figured out how to “game” the unemployment and other economic numbers, as is unburdened by any real sense of personal honor, these reports should be viewed as as good for them as they felt they could get away with. Which, I suppose, is better than nothing, and certainly better than those “rates” of inflation that no longer seem to come up in economic discussions at all.

    Forward !!!

  18. Surreal American Avatar
    Surreal American

    @11B40:

    Since the Obama administration has figured out how to “game” the unemployment and other economic numbers,

    Those darn Chicago guys with their cookin’ the books and all…

    So let me guess: The reality is that no-one has a job?

  19. anjin-san Avatar
    anjin-san

    Since the Obama administration has figured out how to “game” the unemployment and other economic numbers

    Perhaps you could show proof (from a reputable news source of couse) that the Obama administration is doing anything with the numbers that has not been done before.

    The memory of conservatives screeching about how the polls that showed Obama heading for a big win last November were rigged and wildly wrong is fresh in everyones minds.

    Conservatives have yet to show that they can successfully add 1 and 1 and get the correct answer…

  20. Surreal American Avatar
    Surreal American

    @anjin-san:

    Perhaps you could show proof (from a reputable news source of couse) that the Obama administration is doing anything with the numbers that has not been done before.

    On the other hand, we should be somewhat honored that Jack Welch deigns to post comments here.

    Conservatives have yet to show that they can successfully add 1 and 1 and get the correct answer…

    Eleven?

  21. al-Ameda Avatar
    al-Ameda

    I think the lesson here is that we should be skeptical unless the jobs reports are issued by a Republican Administration.

  22. An Interested Party Avatar
    An Interested Party

    Since the Obama administration has figured out how to “game” the unemployment and other economic numbers, as is unburdened by any real sense of personal honor…

    And bravo to you for holding this administration accountable! However, it is amusing how people like you gave previous administrations a pass on things like the deceitful campaign to get us into Iraq or the Iran-Contra Affair…I guess personal honor is only important sometimes…

  23. Jeremy R Avatar
    Jeremy R

    @al-Ameda:

    I think the lesson here is that we should be skeptical unless the jobs reports are issued by a Republican Administration.

    Yup. If Romney had won the election, the stock market was at all-time highs & we had this jobs report, the GOP would be bleating from every rooftop about the Romney Recovery, the media would be nodding along and they’d be poo-pooing any on the left who didn’t buy in to their frame as, “talking down the economy.”

    Oh, and that BuzzFeed article is amazing. What they did with that chart (taking data that was almost entirely positive, subtracting from it expectations and then using the red to turn it into a 50/50 muddle) and the headline (“The Jobs Report Is Wrong”) has got to be one of the most blatant examples of Drudge-baiting I’ve ever seen. I’d be shocked if he didn’t bite. Doug certainly couldn’t resist.

  24. al-Ameda Avatar
    al-Ameda

    @Jeremy R:

    Yup. If Romney had won the election, the stock market was at all-time highs & we had this jobs report, the GOP would be bleating from every rooftop about the Romney Recovery, the media would be nodding along and they’d be poo-pooing any on the left who didn’t buy in to their frame as, “talking down the economy.”

    Romney would probably have announced the jobs report from the deck of the USS Lincoln, the backdrop being a huge banner proclaiming “Socialism is Dead!”

  25. Jeremy R Avatar
    Jeremy R

    @al-Ameda:

    Romney would probably have announced the jobs report from the deck of the USS Lincoln, the backdrop being a huge banner proclaiming “Socialism is Dead!”

    Or maybe something recycled from the campaign:
    “We Built It“

  26. Tyrell Avatar
    Tyrell

    If I want an accurate picture of the jobs situation all I have to do is check the walking track right down the road at 10:00am during the week. Right now it is holding steady after increases since 2008.
    And are you tired of these daily stock market record reports? Like that has something to do with the economic factors that affect the middle class: food prices, wages, gas prices, and the hot dog prices at the local convenience stores.

  27. Drew Avatar
    Drew

    @michael reynolds:

    Nice snark, but dumb. Too much inaccuracy in the craft over the short term.

    The only think we know is that the over the long term the numbers have been miserable, with the unemployment rate fueled only by people dropping out of participation. Now THAT’s a fact.

Latest Posts