By Joe Peters
The single most useful thing a news organization can do to enhance news coverage these days is to take a vow to never conduct or report another poll again.
A couple of decades ago, occasionally you might see reference to a poll done as a sidebar to some more significant news story. Today, more and more we see news organizations leading with "according to a latest poll ..." This is inventing the news, not reporting it and certainly not analyzing it. Rather than polling as to how people feel about a president's economic policy, why not report on those benefitting and those being hurt by that policy.
As a tag line to such a poll you might hear or see the line, the poll has a "margin of error of plus or minus" some percent. The problem is margin of error is a mathematically defined term that few understand. Many infer that the total possibility for error is somehow contained in this factor. The truth is margin of error only covers a portion of all the possible error that might be contained in the study.
Margin of error can be computed by dividing 1 by the square root of the total sample size. So when you see a margin of error of 5 percent, that means the sample size was 400 (i.e., the square root of 400 is 20, 1/20 equals 0.05 or 5 percent). Any news organization would be doing a better service to its viewers and readers if instead of reporting this cryptic and misleading factor of margin of error, they simply stated their sample size.
It paints a much more accurate picture to hear the poll surveyed only 400 of the possible million people who might vote in an upcoming election.
What margin of error represents is not validity of the results of the survey, but the likelihood of the survey being accurately random. It speaks nothing to the quality of the questions, whether the sample group was biased, or other types of "sampling errors," the most likely source of overall error in quick polls or studies.
You rarely hear "sampling error" in regard to polls and studies, in part because there is no clean formula to quantify it. Sampling error refers to all the biases and other errors that can be introduced to a questions and process.
Given the hastily assembled polls that arise in news rooms, think tanks, and lobbying groups, it is easy to see how sampling errors can be introduced into polls that claim "plus or minus 2.5 percent error" (800 people sampled).
The classic error is a question like: "Have you stopped stealing?" This question also represents the logical fallacy of "begging the question." While an extreme example, it represents how a poorly worded or thought question can offer no right answer (a honest person might answer yes because he or she never stole to begin with).
Sampling errors apply to errors in the questions and also errors in other logistics such as whom did you ask. If one called all Democrats to ask whom they might vote for in the next election, there is good reason to believe the results will lean toward the Democratic candidate.
However, this is just the practical argument against polls. A much more central question is what value does a poll present? Quite simply, just because people believe in something, it does not make it right. As late as the 1950s and '60s, some doctors would suggest pregnant women ought to drink beer to appropriately gain weight. A survey of these very qualified doctors would have indicated beer drinking to be a reasonable activity while pregnant. By a similar token, cigarette smoking was not considered a health risk to either mother or fetus in the same time period. Consider how much research and expert opinions have changed in such a short period.
Thankfully, health officials and the media didn't rely on polls to report on drinking or smoking. They turned to good solid research. In short, they did their job. Today's political journalists must do the same. Leave the polls behind, bring the issues and facts to the fore.