No study can include everyone. Researchers study a smaller group and hope it tells them about a bigger one. Whether that hope is justified depends on how the smaller group was chosen, and on who was in it.
Population and sample
The population is everyone the researchers want to draw conclusions about: all U.S. high-school seniors, all adults in a town, all patients with a certain condition. The sample is the people actually studied. A study of 300 seniors drawn by lottery from a national list has a sample of 300 and a population of all U.S. seniors.
Random sampling and representative samples
In random sampling, every member of the population has the same chance of being chosen, as when names are drawn from a complete list. Chance does not favor any kind of person, so the sample is likely to be a representative sample: its mix of ages, backgrounds and opinions matches the population's.
Most real samples fall short of this. A convenience sample takes whoever is easy to reach: students in one class, users of one website. A self-selected sample takes volunteers, who often care more about the topic than others do. Either way the result is sampling bias: the sample differs from the population in ways that can change the results.
Why does a bigger sample not fix this? Because a larger biased sample repeats the same bias more confidently. In 1936 a magazine collected more than two million mailed ballots and predicted the wrong winner of the presidential election, because the people it mailed and the people who replied were not like voters in general. A few thousand randomly sampled voters would have done better.
Two random steps, two different jobs
| Random sampling | Random assignment | |
|---|---|---|
| Decides | who from the population is in the study | which condition each participant gets |
| Used in | any method: surveys, correlational studies, experiments | experiments only |
| Allows | generalizing results to the population | a cause-and-effect conclusion |
| Without it | the sample may be biased | confounding variables may explain the results |
A study can use one, both or neither. An experiment with 40 volunteers randomly assigned to two conditions can show cause and effect for people like those volunteers, but it says less about everyone else. A national survey with a random sample can describe the population, but it cannot show cause.
Generalizability: whom do the results describe?
Generalizability is how far results can be applied beyond the participants. Judge it by comparing the participants with the group you care about: age, culture, country, setting, education, health. Exam readers want you to argue from these participant characteristics. They do not give the point for "the sample was small" or "the study had a flaw".
Worked example: a generalizability answer.
"An experiment on a memory strategy used 40 psychology majors at one private university, ages 19 to 22. Do the results generalize to ninth graders?"
Misses the point: "No, because 40 people is a small sample." Sample size is not a participant characteristic.
Misses the point: "Yes, because it was an experiment." Random assignment supports a causal claim for these participants, not for other ages.
Earns the point: "Only partly. The strategy worked for students who were learning in school, which ninth graders also are, but the participants were 19 to 22 and studied psychology at a private university, while ninth graders are about 14 and attend many kinds of schools." A "partly" answer needs both sides.