A sample is supposed to stand in for the population. This page is about the ways it can fail to: samples that choose themselves, samples chosen for convenience, lists that leave people out, people who do not answer, and answers that are not true. Each one is a kind of bias.
What bias means
A sampling method is biased if it systematically overestimates or underestimates the value it is trying to estimate. "Systematically" is the key word. Every sample gives a slightly different answer by chance; that is expected. Bias is different: it pushes the answer the same way every time, so repeating the survey, or making it bigger, does not help. A poll of 50,000 self-selected viewers is just a very precise measurement of the wrong group.
Samples that are not chosen by chance
- A voluntary response sample is made of people who choose themselves: online polls, call-in shows, comment cards. People with strong opinions, especially negative ones, are the most likely to take part, so these samples usually overstate strong views.
- A convenience sample is made of individuals who are easy to reach: shoppers at one store, students in one class. They may differ from the population in ways that matter to the question.
Both are nonrandom. That is the explanation to give: "the sample was not chosen at random, so it may not represent the population", followed by how it likely differs. "The sample is too small" is not the problem and earns no credit.
Problems that can affect any sample
- Undercoverage: part of the population is missing from the sampling frame, so those individuals can never be chosen. A landline phone list misses people who have only mobile phones, who tend to be younger.
- Nonresponse: individuals chosen for the sample cannot be reached or refuse to answer. If 290 of 500 people do not mail a survey back, the 210 who did may care more about the topic.
- Response bias: people answer, but not truthfully or accurately. Causes include leading question wording ("Do you support a small fee to keep streets clean and safe?"), sensitive questions asked in person ("Have you ever cheated on a test?"), and pressure from the interviewer.
Random sampling from a complete frame removes the selection problems of voluntary response, convenience and undercoverage. It does not stop nonresponse or response bias, which happen after people are chosen.
Explaining a bias for full credit
Name it, explain it, give the direction. A city council posts a link on its website inviting residents to say whether they oppose a new downtown parking fee.
Name: this is a voluntary response sample.
Explain how the sample differs: residents who feel strongly about the fee, most likely those who object to paying it, are the most likely to visit the site and answer.
Direction: the proportion who oppose the fee in the sample will likely be higher than the proportion among all adult residents.
In context: "Because residents choose whether to respond, the sample will likely overrepresent residents who strongly oppose the fee, so the survey will probably overestimate the proportion of all adult residents in the city who oppose it."
Comparing the problems
| Problem | Where it happens | Example |
|---|---|---|
| Voluntary response | people choose themselves | an online poll on a news site |
| Convenience | the researcher picks easy people | shoppers on one street at one time |
| Undercoverage | the frame leaves people out | a landline list, a list of registered voters |
| Nonresponse | chosen people do not answer | 290 of 500 mailed surveys never returned |
| Response bias | answers are not true | leading wording, sensitive questions |
Fixes follow from the cause: choose people by chance from a complete frame, follow up with people who do not respond, and write short, neutral questions that describe the issue plainly and offer both sides.