How Reliable Are Government Economic Projections?
Dylan Matthews went back and looked at the economic projections from the Federal Reserve Board and the Congressional Budget Office to see just how accurate they turned out to be. The results were not encouraging.
Here’s the result for the Fed:
And here’s the one for the CBO:
Given that much of what the Fed does, and the projections the CBO makes about the economic impact of legislation, are based on these projections, one wonders if we’d all just be better off flying blind.
Anyone serious about this question should read Fooled by Randomness as well as The Signal and The Noise.
Generally, “growth” is a hard problem, and not one Nate Silver is going to call “easy to predict” any time soon.
(The follow-up question might be “does knowing everyone’s guesses tell us something, even if those guesses are generally wrong?”)
Economics is astrology with an even worse track record.
It’s quackery.
Dear Emperor: Dude, you’re naked.
We are flying blind. We’re just flying blind with the illusion that we have a clue.
Except the CBO most likely isn’t taking the GOP actions like the debt ceiling and the sequester when it makes the forecasts. They probably would be closer if we didn’t have one insane political party.
It does make you wonder if they’re working with two sets of books.
@David M:
The key words in any economic projection, whether they are stated explicitly or merely implied, are:
The problem with any system containing people (or animals, really) is that things do not have to remain equal. People can decide differently at any moment. Today’s markets are a good example of that. Look at gold. It was in a fairly tight trading band around $1400 per oz, until suddenly people decided differently, and it fell towards $1300.
We in the physical sciences don’t have to deal with that. If you want to take some ice out of the freezer and watch it melt, it will do exactly the same thing every time (at STP). It will never “decide” to do something different.
The interesting thing in the modeling of crowds might be identifying where and when choices become stable. Nate Silver apparently has an easy time with elections because people don’t flip very much, once decided. The “market may drop out” for a candidate, but it’s pretty rare, or at least much more rare than market gyrations.
@michael reynolds:
I’d suggest that if you come out that strong, you have to treat Nate Silver as badly.
There are a range of problems. He and I try to distinguish between hard and easy ones.
@john personna:
If you cannot predict, you cannot prescribe.
Number of economists who predicted that we’d be seeing sharply lower deficits this year? Number of economists who predicted health care costs would actually ease? Number of economists who predicted that GM would be riding high and profitable five years after the bail-out?
Probably not zero in each case, but I’ll bet it’s close.
I’ll restate a question I asked before: what is not part of the economy? What is outside of the economy? I think the answer is that nothing is not the economy. If that’s the case then “the economy” is everything done by, for, and to humans on planet earth by each other, by themselves and by nature. You’re talking about predicting the system of everything at a point in the development of economics where we’re not even sure what the effect of a tax increase is, let alone the effect of a tax increase in a global economy where 20 other governments are raising taxes and 20 more are lowering taxes and all of it together doesn’t add up to 5% of the total complexity.
There are too many factors — many of them incapable of being reduced to numbers — that effect something as big as “the economy.” Which is why no one ever gets a prediction right unless it’s a case of a stopped clock being right twice a day.
I would like to see a couple more years before making that conclusion. How did its predictions turn out 5, 10, 15, 20, or 25 years ago?
@michael reynolds:
The economists looking at the future economy are just giving you the “all other things being equal” answer.
If they understand that, they don’t pretend it is anything else. If you understand that, you don’t ask for it to mean something else.
That’s actually a good way to distinguish between predictors. If they say (in Taleb’s words) “do over” and never learn or acknowledge that they missed, they aren’t really helping. On the other hand, if they can say “you know, something really interesting happened there …” then they might be contributing.
You actually have to apply that second kind of rigor, or else you treat every tick as a trend in itself. Why did revenues or health expenses surprise? I’ve read some good answers, some of which are worrying. For instance if short-term health costs fell because in a recession people stayed away from doctors, that may be a net win (they got better) or ultimately a loss (they got worse).
In ending though, the big lesson is that not all problems are the same, and picking a hard one doesn’t make them all hard.
@john personna:
A prediction that rests on nonsensical assumptions — all other things being equal — is of no use. The prediction is only of use if it is sufficiently accurate as to be a basis for action.
As a purely academic thing of course I have no problem with economics. No more than I object to studying French Literature. But we act as though economists have some ability to point a way forward. I don’t think they do.
In fact, I don’t think we’d be any worse off if we chose some entirely different basis for making decisions. We could, for example, decide to base our decisions on a moral system that valued the continuation of civilization. In other words, we could do the right thing — make sure that poor people didn’t live in misery while at the same time allowing the successful to enjoy a better lifestyle. Would that work? Eh, probably about as well as letting various ax-grinding academics and political hacks pretend that their graphs and charts allow them to see the future.
@michael reynolds:
You keep saying “prediction,” but we’ve established that projections and predictions are different.
(I don’t think anyone wants to set the social security savings rate and the withdrawal rate without any kind of projection. I mean, absent that what would you even do? Use a random number generator?)
Come to think of it, the thing that messed up social security was something out of left field. People started living much longer. The answer to that can’t just be to say “they were wrong, wrong I tell you!” Because without those models there would be no social security in the first place.
What you have to do is say “ok, things weren’t equal, we face more longer retirements, let’s put that in the projection and try again ….”
@john personna:
Let me ask you this: Social Security started in 1935. What do you think the value of those long-ago projections (or predictions) are today? My guess is they were crap by 1945. Why? Because all other things were not equal. And they never will be.
Would we be better or worse off today if we’d simply decided back in 1935 on a fair contribution — say 5% of gross income (plus matching from employer) with no cap, and a pay-out limited to a poverty level plus 20%, and only to those who had assets less than five times the average? My guess is the program would have been solvent very quickly, run up a huge surplus and be rolling along nicely today.
Those numbers are pulled out of the air, obviously, and based on nothing but a vague notion of fairness. As opposed to what we have, which is a terribly scientific approach that has to be tinkered with constantly and is forever threatening to go broke.
Imagine the political power behind making a “fair” decision, easily comprehended by one and all, and then following through on that. As it is we have a system of competing economists all bullsh!tting away to earn a pat on the head from this party or that special interest. We end up with what looks like a rigged game to the citizenry, who, perceiving correctly that what we have is a race to gather up spoils, throws all caution to the wind and just keeps yelling, “More! More!”
My point is that we have opted to put our faith in a completely unproven science. In the west we love us some science, even when it’s a pseudo-science, or perhaps an undeveloped and immature science, like medicine in an earlier era.
This is a famous and humorous case of medical treatment, circa 1685:
That, to me, is economics as practiced today. We pretend to know what we’re doing, but we don’t. Raise taxes? Lower taxes? More imports? Fewer imports? Stimulus? Budget cutting? Bleeding? Purging? Powdered bezoar stone?
All the best economic minds keep predicting the death of the Euro, which is at $1.30. Exactly where it was four years ago. So, it’s absolutely, positively doomed. . . and it’s worth just what it was before all the doomsaying. Riiiight.
All the best economic minds have accumulated quite a record of not knowing what they’re talking about. My conclusion is that they’re full of baloney and poor old Charles the Second might just live if we stopped trying to treat him.
@David M:
If there is a Republican anywhere near an issue, some progressive will blame him for everything that is wrong. For the first two years of the Obama Administration, the Democrats controlled Congress and the White House. Of course, the Democrats put all of their effort into creating new entitlements instead of worrying about the economy.
@superdestroyer:
Bull. As usual.
First of all, so long as we have a filibuster the White House cannot be said to control the Senate.
Second: the stimulus, which was not an entitlement, and was about the economy. Ditto the auto and bank bailouts. Ditto money for clean energy, ditto extending tax cuts, extending unemployment and a bunch of other stuff I’m forgetting.
Third: health care (the entitlement I assume you were addressing) is not about the economy? This is news. I had the impression that health care was part of the economy.
@michael reynolds:
MR,
Progressives cannot claim credit for TARP when they constantly blame Republicans for bailing out “big banks”. As usual, progressives try to have it both ways. Claim a program is theirs is there is any level of success but blame others if there is any failure. Besides, TARP was passed during the Bush Administration.
The stimulus spending was nothing but a pay off to state and local government since most of the money was used to make payroll. No long term economic benefit. The auto bailout was just a pay off to the United Auto Workers since the bail out helped them while screwing over the stock and bond holders.
The real blame of the Obama Administration was to create entitlement (Obamacare), pay off core groups of the Democratic Party (money to cities, states, industrial unions), and plan on the economy going through a normal recovery. What the Obama Administration did not plan on is that economic reality has changed forever and the old rules just do not apply.
The new reality is that unemployment will remain high (and could go higher if comprehensive immigration reform passes). That there are fewer paths to being middle class and escaping being around the poor is much harder. Image how much smaller the middle class will be when healthcare reform kicks in and hospitals and care providers start laying people off.
@michael reynolds:
Obviously for those 10 years the projections had great value, and then came a course adjustment.
It’s a bit of a strawaman to say that projections should last 100 years or be no good,.
Complex mathematical models are only as good as the underlying assumptions. For example certain economists produced papers showing fiscal multipliers were in the .5 range, doing so by selectively including and excluding data. The results were a vast underestimation of the damage fiscal austerity would do in Greece, Portugal, Ireland etc. People who hold out a paper and say, “See! It’s all mathy so it must be right!” miss the reality that mathematical rigor often creates an illusory sense of certainty regarding a system of highly complex variables, many of which cannot be quantified.
Ever notice how economic forecasts never include a range? These prognosticators think they can zero in to a tenth of a point how spending will change; no scientist does that. Scientists speak in terms of probabilities.
@David M: The other party seems to be mentally challenged. :^( Still, that’s better than being insane. :^)