|
|
THE TEN COMMANDMENTS OF POLLING
by Ken Blake
UNC-CHAPEL HILL
1996
1. Thou shalt know how the people
interviewed for the poll were
chosen. The best polls choose people at random. There are many
different schemes for doing so. For example, a pollster can
program a computer to generate telephone numbers at random.
Another strategy involves putting people's names on a list in
some non-systematic fashion and then choosing every sixth, tenth
or "whateverth" one starting from some random point on the list.
The basic rule is that everyone whom the poll purports to
represent should have an exactly equal chance of being
interviewed. The results of polls based on non-random samples --
the "dial-in" polls so popular with television newscasts, for
example -- are generalizable to no one except the people who were
interviewed.
2. Thou shalt know the poll's
response rate. Here's how to
calculate a poll's response rate using any calculator. 1. Enter
the number of people interviewed for the poll. 2. Press the
division key. 3. Enter the number of people the poll attempted to
interview (for example, the number of questionnaires mailed out,
or the total number of working, residential telephone numbers
dialed in a poll claiming to represent residents of some
population). 4. Press the "equal" key. 5. The resulting number
should be .66 or higher. In other words, 66 percent or more of
the people asked to take part in the poll should agree to do so.
The lower the response rate, the great the risk that those who
declined to be interviewed belong to some significant subgroup
whose attitudes and opinions the poll will fail to represent.
3. Thou shalt know the poll's
error margin. Here's how to
calculate a poll's error margin using any calculator.
-
Enter
the number .25.
-
Press
the division key.
-
Enter
the number of people interviewed for the poll.
-
Press
the "equal" key.
-
Press
the "square root" key.
-
Press
the multiplication key.
-
Enter
the number 1.96.
-
Press
the "equal" key.
-
Press
the multiplication key.
-
Enter
the number 100.
-
Press
the "equal" key. The resulting number is the poll's accuracy at
the
95 percent level of confidence. For example, a result of 4.2
means that if 60 percent of the people interviewed for the survey
said they preferred chocolate ice cream over vanilla, then one
can be 95 percent confident that the actual percentage of people
favoring chocolate over vanilla in the population the poll
purports to represent lies somewhere between 55.8 (that's 60
minus 4.2) and 64.2 (which is 60 plus 4.2). What's a good error
margin? It depends. An error margin of 4.2 will let you predict
the winner of an election if the poll shows one of two candidates
has 80 percent of the vote. Subtracting the error margin from 80
percent reveals that, at the very worst, the candidate has 75.8
percent of the vote and is clearly still ahead. If, however, the
poll shows one of two candidates with 53 percent of the vote,
then you can't predict a winner (without using some really hairy
statistics that you'll need a computer and some fancy software to
calculate) because the candidate could have anywhere from 57.2
percent of the vote to only 48.8 percent of the vote.
4. Thou shalt know the
poll's confidence level. The 95 percent
confidence level is the most common. Confidence levels exceeding
95 percent are even better. Confidence levels less than 95
percent should be increasingly mistrusted as they get further
from 95 percent.
5. Thou shalt recalculate
the error margin when drawing
conclusions about a subgroup in a poll's sample. If 600 randomly
chosen North Carolinians are interviewed for a poll, the poll's
error margin at the 95 percent level of confidence is plus-or-
minus four percent. But conclusions about the, say, 312 people in
the sample who are female have an error margin of plus-or-minus
5.5 percent at the 95 percent confidence level. A pollster who
wants an error margin of plus-or-minus four percent at the 95
percent level of confidence when drawing conclusions about North
Carolina females needs to talk to 600 randomly selected North
Carolina females, not 312.
6. Thou shalt make a big
deal out of a crosstabulation only if
the difference is significant. A crosstabulation compares how
people in different subgroups answered a question. For example, a
poll might find that 60 percent of the women interviewed plan to
vote for Candidate A compared to only 55 percent of the men
interviewed. So, Candidate A enjoys stronger support among women
in the general population than among men in the general
population, right? Not necessarily. As a rule of thumb, it takes
a difference of 10 percentage points or more between subgroups in
a random sample to accurately suggest that a similar difference
exists in the general population from which the sample was drawn.
Ideally, you should get a statistician to calculate something for
you called a Chi Square. Or, you can pick up just about any basic
statistics book and figure out how to calculate a Chi Square on
your own. It's too involved to detail here, but you can do it
with any calculator and the table of probabilities in the back of
the statistics book.
7. Thou shalt generalize
a poll's results only to the
population sampled. A poll based on a sample of UNC-Chapel Hill
students age 18 to 21, for example, does not necessarily
represent the views of all people age 18 to 21. Nor does a poll
based on a sample of North Carolina state residents necessarily
represent the views of all Americans.
8. Thou shalt know who
paid for the poll, and who conducted it.
Exercise a little healthy skepticism if whoever paid for the poll
has a vested interest in its outcome, and if whoever conducted it
doesn't have a solid reputation for telling the truth.
9. Thou shalt know the
wording of the question. Leading
questions are easy to write. For example, "You certainly don't
plan to vote for putting that crook, John Smith, back into
office, do you?" might tend to yield results unfavorable to John
Smith, whatever his actual character might be. Balanced questions
present all possible answers as equally attractive. For example,
"Do you plan to vote for John Smith, or for Jane Doe." Also try
to find out the wording of previous questions. The answer to one
question can be influenced by questions asked earlier in the
poll. For example, a respondent asked about support for the death
penalty after being asked several questions about the threat of
crime might answer differently than a respondent asked about
support for the death penalty during a poll that made no mention
of crime.
10. Thou shalt consider the
ability of the people interviewed to
give an informed, well-considered answer. There's not point in
doing a poll on whether Newt Gingrich wears boxers or briefs.
Only he knows, and he's not telling. What everyone thinks he's
wearing isn't going to alter the reality of what he is wearing.
If there are any appropriate ways to investigate the question,
polling is certainly not among them. Similarly, ordinary people
should not be asked whether they approve or disapprove of U.S.
policy on Argentinian beef imports. Most have no idea what the
policy is, or if there even is one. Even if the pollster
patiently explains the policy, the resulting answer is likely to
be a snap judgment prone to reverse itself the next day. The best
polls probe not only a respondent's opinion but also the level of
deliberation and conviction associated with the opinion.
|