Obama Approval Up! Or Not!
Don't confuse modest bumps in a polling trend with actual changes in the thing being polled.
This, from Gallup‘s Jeffrey Jones, is rather amusing:
President Barack Obama’s job approval rating has shown modest improvement in the past week. His latest rating, based on Oct. 24-26 Gallup Daily tracking, is 43%, and his approval has been at or above 42% in each of the last seven days. In the prior two weeks, his averages were generally at or below 40%.
More broadly, Obama’s approval rating averaged 40% from Oct. 1-19, but 43% since Oct. 20.
The increase in Obama’s approval rating could be tied to two recent major foreign policy events — the death of Libyan President Moammar Gadhafi and Obama’s announcement that virtually all U.S. troops would be withdrawn from Iraq by Dec. 31. Additionally, the U.S. stock market has shown gains this month, particularly in the past week.
Indeed, it could be tied to those events in the news. Then again, it could be normal fluctuation within the poll’s sampling error of ±3 percentage points. Given that every single result during the time frame sampled falls within that margin, I’m going to go with that explanation.
UPDATE (bumped from the comments): Given a few sustained days, the probability increases that we’re seeing a trend rather than random noise. I’m not arguing that Obama’s approval isn’t rising, just that we don’t know.
Look at the movement around October 16: He dropped from 42 to 38-a whopping 5 points-in a single day. A journalist will look to see what happened that day. A political scientist will start observing for a trend but note that that’s actually within the margin of error. 42 isn’t really 42 but the median possibility in the range 39 to 45. Likely, 38 is the representation of a range 35 to 41. Given that these two intersect, it’s not exciting until it’s sustained.
Does three days in a row from there mean that Obama’s numbers had fallen and something else occurred on the 20th to boost him back to where he was? Well, either we’ve got five days of confirmation of that or we can look at the drop-off around the 20th as an aberration and say that, essentially, Obama’s approval has been somewhere around 40 for quite some time now.
That’s not me reading my preferences into the numbers, just my skepticism that tiny fluctuations in poll numbers tell us anything meaningful given the nature of sampling.
via Taegan Goddard
???
When Obama’s approval rating is 44%, the range within the margin of error is 41-47%.
@ponce: Right. And the lowest number on this chart is 38–which means 35-41.
I don’t believe “sampling error” would produce the fluctuation within the polling group. Notice how day to day variation is like 1%.
Sampling error would be the expected divergence from the sampled group to the larger population. That’s something we can’t see. It could be systematically higher or lower, or it could be random, but we can’t know without a bigger/different survey.
The last week seems to be a significant bump, but hindsight is best. In a week or two it’ll show if this is a start of a shift upward or just another short lived euphoria.
@john personna: Sure, but we should expect fluctuation within 3 percentage points in these samples. I’m not saying the trend isn’t probably slightly up–the center point is the best guestimate of where in the margin we are, after all–but the numbers don’t tell us that with any confidence.
What we’re seeing is pretty much steady performance over this period. And it’s actually remarkably high given how crappy the economy is–presumably a function of his likability.
@john personna:
It’s not actually the same sampling group every day is it? I would think this was a different random samplling every day, which would bring the fluctuation back to the fluctuation range for the whole population that the sample group is coming from. I’m not necessarily trying to challenge you if you have specific knowledge of how the sampling is done, I just always thought it was a random sampling each time, though it probably varies some depending on the poll..
James, when Gallup reports Obama’s approval rating is 44% plus or minus 3 points, the possible range is 41-47%.
38% is not within that range.
Think about it.
44 plus or minus three 3, could be 41
38 plus or minus three 3, could be 41
They could be the effectively the same number, but that is just a single intersection at the extremes. However given three days at 38/39 then a week at 42-44, seems to indicate a slightly upward trend, as James also indicated. But when has a sampling over 10 days ever meant much. In another 14 days, we’ll know if its sustained or not.
@john personna:
It’s something we can estimate – that’s the whole point of the margin of error. So for any one of those data points, there’s some (unstated, but probably .95) probability that the population’s approval rate is within ±3 percentage points of that of the sample.
Once again, James, you prove yourself to be a reactionary shill for the Republican Party.
You bastard.
@john personna and@ponce: I think : @rodney dill formulates it better: Given a few sustained days, the probability increases that we’re seeing a trend rather than random noise. I’m not arguing that Obama’s approval isn’t rising, just that we don’t know.
Look at the movement around October 16: He dropped from 42 to 38–a whopping 5 points–in a single day. A journalist will look to see what happened that day. A political scientist will start observing for a trend but note that that’s actually within the margin of error. 42 isn’t really 42 but the median possibility in the range 39 to 45. Likely, 38 is the representation of a range 35 to 41. Given that these two intersect, it’s not exciting until it’s sustained.
Does three days in a row from there mean that Obama’s numbers had fallen and something else occurred on the 20th to boost him back to where he was? Well, either we’ve got five days of confirmation of that or we can look at the drop-off around the 20th as an aberration and say that, essentially, Obama’s approval has been somewhere around 40 for quite some time now.
That’s not me reading my preferences into the numbers, just my skepticism that tiny fluctuations in poll numbers tell us anything meaningful given the nature of sampling.
But 42 is the answer to everything.
@James Joyner:
You forgot a rather important part about margins of errors.
A statistician will point out that there is a 95% confidence that the actual result is within the margin of error, the +/- 3 percentage points, and that the 38% result on the 17th may be one where the actual result isn’t within +/- 3 percentage points.
Look at the surrounding results, this since Gallup is using a three day average.
40, 40, 42, 38, 39, 39, 42, 42
The numbers on the 17th were so bad that they caused a 4 point drop from the previous average, and the two following days that the numbers from that day were used to calculate the average, the average was down. Then when it was removed from the calculations, the average went up 3 points and the average has been around 42-44% since.
The highest drop and gain in that image are both connected to the result on the 17th.
That’s the explanation I would use.
I didn’t link the wikipedia page, because I’ve been given a hard time about that, but “sampling error” has a specific meaning.
And for what it’s worth, I think that the seven day average in early October is a few percent lower than the seven day average in late October is significant. The day to day consistency tells us that the polling method has little internal noise.
Of course a few percent is just a few percent. It is not a mountain.
Funny how it is that when the numbers for Obama seem to go up, we get this great caution about reading too much into “tiny fluctuations”.
When there seems to be evidence for Obama numbers being down however, no matter how small, it is always evidence of some grand trend – with rarely even a nod to the larger context of what other polls are saying or the longer term trends.
Granted, that usually comes from Doug – but it stands as the OTB take on things….
@Tano: Doug’s posting with the frequency that I used to and that can lead to a lot of discussion of individual polls and such. There is plenty of opportunity for “Obama reaches another all-time low” posts in that mode.
I’m more into looking at patterns than snapshots at the moment.
@James Joyner: Your missing the fact that about 1 in 20 polls will end up being outside of the margin of error. You can raise that to a lower number, but then the margin of error goes up.
And when one poll does that, because Gallup is using a three day average, the average for the next two days will be also be affected by the polling that day. Which is what we are seeing here.
The explanation that something happened on the 16th that caused a 4 point drop to the average and then three days later (didn’t happen two or four days later, remember Gallup’s three day average) something else happened that caused a 3 point gain to the average… Occam’s razor and all that…
To add:
If there was something that happened on the 16/17th that caused lower numbers for Obama and this then would give Obama a lower approval on the next three days polling (not talking about the average), then removing the polling on the 17th wouldn’t have caused the polls to go up as much three days later, because the polling on the 18th and the polling on the 19th would have still kept the average low, even if the polling on the 20th would have been great for Obama.
And to further drive in the nail, Gallup has done about 971 of these up and until the 25th October.
289 times the change from the previous average has been 0.
215 times the change from the previous average has been 1.
205 times the change from the previous average has been -1.
99 times the change from the previous average has been -2.
99 times the change from the previous average has been -2.
24 times the change from the previous average has been 3.
33 times the change from the previous average has been -3.
2 times the change from the previous average has been 4.
4 times the change from the previous average has been -4.
The change has never higher than 4 or lower than -4.
Nothing happened on the 16th or 17th to create this drop, nor did anything happen on the 19 or 20th to create the gain.
The polling on the 17th was one of the times when the result is outside of the margin or error.
Nothing else.
But it is one major reason why journalists, political scientists, and everyone else who talks about polls really need to get at least a crash course in statistics…