Oklahoma High Schoolers Are Very Familiar With George Washington, Thanks

george-washington-1782-paintingYou may recall that a couple of months ago, there was a “shocking survey” circulating around which showed that only 23% of Oklahoma students knew that George Washington was the first President of the United States. At the time, my colleague James expressed his extreme skepticism of this result, a skepticism shared by most of the commenters. Well, as it turns out, this skepticism was quite justified, as Nate Silver points out:

[Oklahoma State Representative Ed] Cannaday therefore had little difficulty setting up an experiment: he arranged to have all the seniors in the 10 secondary schools in his district take the Strategic Vision/OCPA survey. Cannaday tried to replicate the Strategic Vision survey to the greatest extent possible. The same exact questions were used, and as in the case of the original survey, the answers were open-ended rather than multiple choice. The survey was administered to a total of 325 seniors, including special education students.

Cannaday’s survey however, found his students doing just fine: They answered an average of 7.8 out of the 10 questions correctly. By comparison, the high school students that were purportedly surveyed by Strategic Vision had gotten just 2.8 out of the items correct. 98 percent of the students on Cannaday’s survey — not 23 percent — knew that George Washington was the first President. 81 percent — not 14 percent — knew that Thomas Jefferson had written the Declaration of Independence. 95 percent — not 43 percent — knew that the Democrats and Republicans are the major political parties. There was just no comparison between the two.

[…]

There is no reason to think, in other words, that the students in House District 15 should have gotten such profoundly superior results to the “students” in Strategic Vision’s survey. Nor could Strategic Vision’s results have been the result of any sort of mathematical or methodological oddity. Consider their claim that literally none of the 1,000 students they surveyed were able to answer more than 7 of the 10 questions correctly — lower than the average score achieved in Cannaday’s test.

There are, rather, only two possibilities. Either the Strategic Vision survey was entirely fabricated — or Cannaday’s was.

I would put every dollar to my name on Cannaday, who has kept the surveys and is happy to show them to them to anyone who comes asking.

So would I. Cannaday’s results appear to have a sounder methodology, and also make more intuitive sense.

FILED UNDER: Education, , , , , ,
Alex Knapp
About Alex Knapp
Alex Knapp is Associate Editor at Forbes for science and games. He was a longtime blogger elsewhere before joining the OTB team in June 2005 and contributed some 700 posts through January 2013. Follow him on Twitter @TheAlexKnapp.

Comments

  1. JVB says:

    Alex Knapp should, in the future, curb his enthusiasm for getting an article submitted until he has at least read it completely. He should use all the modern tools available now, such as spell check, or at least have someone else proof read it.

    I am happy, however, that someone showed an interest in highlighting the flaws in the original test.

  2. Rick Almeida says:

    Strategic Vision seems to be an increasingly problematic outfit, see http://www.fivethirtyeight.com/search/label/strategic%20vision

  3. G.A.Phillips says:

    What the hell I’ll give it a shot:

    What is the supreme law of the land?

    Separation of Church and state.

    What do we call the first ten amendments to the Constitution?

    Not going far enough.

    What are the two parts of the U.S Congress?

    Taxing money and Spending money.

    How many justices are there on the Supreme Court?

    Way to many unenlightened conservative Christian teabaggers.

    Who wrote the Declaration of Independence?

    An almost enlightened humanist.

    What ocean is on the east coast of the United States?

    One of the other polluted one.

    What are the two major political parities in the United States?

    Liberals and Progressives.

    We elect a U.S. senator for how many years?

    30 or until they die.

    Who was the first President of the United States?

    An evil Christian slave lord terrorist.

    Who is in charge of the executive branch?

    Our Dear leader Barrack Obama.

  4. Wayne says:

    “There are, rather, only two possibilities”

    There is a third possibility and that is one of them is a fluke. It happens in surveys and in science experiments. That is why in the true science community, results of an experiment are not accepted until it can be replicated and tested for flaws. Unfortunately the MSM like a sensational headline and pushes and agenda, therefore they will splatter a finding they like all over the place regardless of its merits. After all perception is reality in their world.

    In this case IMO the Strategic Vision survey was probably fabricated on some level.

  5. Franklin says:

    Maybe after the first result, all the teachers and parents in Oklahoma taught their kids the correct answers to the questions.

  6. Tlaloc says:

    There is a third possibility and that is one of them is a fluke. It happens in surveys and in science experiments.

    There are mathematical tests for the kind of variance you can from a legitimate poll of a subset of a population. In fact the margin of error is exactly one of those tests. This was so far outside of the margin of error that a genuine fluke (i.e. a random selection just happened not to get a really random selection) really isn’t a possibility. Either somebody horribly mangled the data, which certainly should have been caught, or it was a fraud.

  7. Rick Almeida says:

    There are mathematical tests for the kind of variance you can from a legitimate poll of a subset of a population. In fact the margin of error is exactly one of those tests.

    Just FYI, margin of error is not a mathematical test of variance, or anything else. Margin of error is an estimate of sampling error – the discrepancy between a statistic and its underlying parameter due to the precision lost through sampling.

    What you’re talking about, Tlaloc, is an hypothesis test; that the observed results were so far from reality that chance (sampling) alone is exceedingly unlikely to account for the discrepancy.

    In either case, margin of error is only part of the story. Polls also have a confidence level; a probability that the parameter values are within the poll’s margin of error. Generally, that’s .95. Even in the best case, there’s always a 5% chance a poll’s population is outside the poll’s margin of error, which is why Silver would like to see the original data.

  8. Wayne says:

    I don’t believe Tlaloc knows what a genuine fluke is.

    Rick’s post was pretty good. The end story is a legit survey or experiment can fall way out of the expected range even without a screw up although it is more often due to screw ups. Even in a pure mathematical statistic even though I can say that I have better than 95% confidence that a person flipping a coin 10 (or 5) times will not land up all heads, it does happen.

    Once again I think the Strategic Vision was probably fix but am aware of other possibilities.

  9. Franklin says:

    I don’t believe Wayne knows what a genuine fluke is.

    What was the margin of error on Strategic Vision’s test? Pretending for a moment that it is a somewhat typical 5% (for two standard deviations), and that Cannaday’s results are the “correct” data, this means the SV test was off by about 30 standard deviations on the Washington question.

    The chance of this is many many orders of magnitude different than getting 10 heads in a row (which is roughly 1 in 10,000). Even if the SV test was “only” off by 6 standard deviations, the chance would be roughly 1 in 500,000,000. While it’s still technically “possible”, it is not a possibility for any practical purpose.

    Agreed that it is most likely fixed.

  10. Wayne says:

    Getting caught for rules violation so here is my shorten post

    95 % statistical confidence means that 1 in 20 times it doesn’t happen, sometimes to the extreme.

    http://en.wikipedia.org/wiki/Confidence_in_statistical_conclusions

  11. Wayne says:

    Try another Part

    Standard deviation using the case in my reference, 95% confidence of the height of a man would be between 64-76 inches. Does that mean there are not men shorter or taller? Of course not and I think most of us know men that fall out of that range.

    http://en.wikipedia.org/wiki/Standard_Deviation

  12. Rick Almeida says:

    What was the margin of error on Strategic Vision’s test? Pretending for a moment that it is a somewhat typical 5% (for two standard deviations)

    Again, respectfully, this statement conflates margin of error, confidence level, and now, the empirical rule.

    Silver’s post shows that the Strategic Vision study claimed a 3% margin of error. I assume, because it is standard and not stated elsewise, a 95% confidence level.

    It is possible, repeat possible, that SV’s sample was astronomically far outside the likely population value because of sampling error. People win the Powerball, after all. Low probability events happen.

    Again, this is why it’s important to see the original data.

  13. Wayne says:

    Rick
    I agree with you. However raw data is hard to come by at times. I am a firm believer in this internet age that raw data should be release on most polls and studies.

    If anyone has ever taken mathematic statistic and business statistic in college they would know that they are not the same. Business statistic for ease of purpose and practicality will make all sorts of assumptions and creative formulations that would drive mathematicians nuts. Polling firms takes on a business attitude toward statistics. Unfortunately layman consider the business approach as same as mathematicians approach. Many similarities but they are not the same.