Polls are most interesting in close races. However, this is also when they are the most unreliable. The news media frequently ignore, misinterpret and misrepresent public opinion polls for the sake of simplicity.
The most common method of polling opinion is via probability sampling. Probability sampling is any method of sampling that utilizes some form of random selection. This means that everyone in the population has an equal chance of being selected. Random Digit Dialing (RDD) uses a program that randomly generates phone numbers and then once contacted, there are random selection protocols. For example, once somebody answers the phone the pollster could ask for the second oldest female in the house.
The confidence level is the confidence that a particular sample is representative of the overall population. The confidence interval is the range of uncertainty in this estimate. Most surveys have a confidence interval of 6%, meaning that the margin of error is no more than 3% higher and 3% lower than the sample mean. When a study cites 95% confidence, that means that confidence interval is (100-95)% = 5%. Similarly, sampling error results from not representing the actual population.
- For example, interpret a confidence interval of 6% if 30% of the population approves of the president and 70% disapprove. Between 27% and 33% of the population approves and between 67% and 73% of the population disapproves.
- As a second example, interpret a confidence interval of 10% if 45% of the population would vote for Candidate A and 55% for Candidate B. Between 40% and 50% would vote for Candidate A and between 50% and 60% would vote for Candidate B.
A primary source of uncertainty is due to question wording effects. Question wording effects are seemingly incidental differences in question wording that profoundly affect responses. Incidental wording differences exclude those that push public opinion in a certain direction.
An example of question wording effects arise from the question, "Are we spending too much, too little or about the right amount on halting the rising crime rate vs law enforcement?"
"Are we spending too much, too little or about the right amount on dealing with drug addiction vs drug rehabilitation?"
"Are we spending too much, too little or about the right amount on assistance to the poor vs welfare?"
The solution to question wording effects is to ask the question repeatedly but with different wording. Also, sometimes it is possible to ask the same question over time since the most consistent answers are the most reliable.
Another primary source of uncertainty arises from question order effects. This describes how the order of questions asked can profoundly affect responses. Polling political knowledge and media consumption habits gives a marvelous example of question order effects. If questions testing knowledge are asked first, then people report watching less news.
Another example of question order effects arose amidst Cold War surveys. Two questions were asked and the following results were observed:
- "Do you think the U.S. should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it?" 36% agreed.
- However, if this question is asked first, "Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?" Now, 73% agreed with the question above (as opposed to the original 36%).
The solution to question order effects is to randomize the order of questions in the same survey. Also, to create a buffer between two questions that impact each other. A pollster could ask if Communist Russia ought allow in U.S. reports, follow this by ten unrelated questions, then ask whether America ought allow in Communist reporters.