Skills

Sampling and Generalizability

A sample stands in for a population.

Practice 2: Research Methods and Design

Part 1 · Hook

Why this matters

In 1936 a popular magazine mailed about ten million ballots, mostly to names from phone books and car registrations, and predicted the challenger would win the presidential election. He lost in a landslide. More than two million people mailed ballots back, but they were not like voters in general: during the Great Depression, people with phones and cars were better off than most, and the people who chose to reply were more opposed to the president than those who did not. A huge sample, drawn and returned the wrong way, gave the wrong answer.

Part 2 · Prerequisite check

Quick check before you start

1. A study compares students who chose to join a study group with students who did not. What kind of study is this?

  1. A correlational study, since no one was randomly assigned
  2. An experiment, since two groups are compared
  3. A case study, since students are studied
  4. Naturalistic observation, since nothing was asked
Show the answer

The students chose their own groups, so nothing was manipulated or randomly assigned: the study is correlational.

  • Correct: A correlational study, since no one was randomly assigned:
  • An experiment, since two groups are compared:
  • A case study, since students are studied:
  • Naturalistic observation, since nothing was asked:

2. What is the job of random assignment in an experiment?

  1. It makes the groups alike, on average, before the treatment
  2. It picks who from the population is studied
  3. It hides the conditions from the researchers
  4. It says how the outcome will be measured
Show the answer

Random assignment uses chance to decide which condition each participant gets, so the groups start out alike.

  • Correct: It makes the groups alike, on average, before the treatment:
  • It picks who from the population is studied:
  • It hides the conditions from the researchers:
  • It says how the outcome will be measured:

Part 3 · See it

See it first

Left: a population of 48 people drawn as dots in three colors (three kinds of people, such as age groups). Random sampling gives every member the same chance of being chosen; 12 are chosen, 4 of each color, so the sample mirrors the population. It decides who is studied and lets results generalize to the population. Middle: the sample of 12. Right: random assignment uses chance to split the 12 into a treatment group and a control group of 6 each. It decides which condition each person gets and lets an experiment claim cause and effect. A study can use either step, both, or neither.
Random sampling decides who is studied; random assignment decides which group each person joins. LevlPrep original diagram.

Part 4 · Step by step

How it works, step by step

  1. Every member of the population has the same chance of being chosen (random sampling)the sample is likely to be representative: it mirrors the population
  2. People are chosen because they are easy to reach or volunteer (a convenience or self-selected sample)the sample is biased, and a larger biased sample is still biased
  3. The participants differ from a group in age, culture or settingresults may not generalize to that group; argue from those characteristics

Part 5 · Key ideas

Key ideas

  • The population is everyone a study wants to describe; the sample is the people actually studied.
  • Random sampling gives everyone the same chance of being chosen, so a representative sample is likely. A convenience sample or volunteers create sampling bias.
  • Random sampling decides who is studied and supports generalizability. Random assignment decides which condition each person gets and supports cause and effect.
  • Judge generalizability from participant characteristics (age, culture, setting), not from the sample size alone.

Part 6 · Misconception

A common mistake

The wrong idea: A bigger sample always fixes sampling problems, and random sampling and random assignment are the same thing.

What actually happens: Size cannot fix a biased sample: two million returned ballots still did not represent voters. Random sampling (who is studied) and random assignment (which condition) are separate steps with separate jobs.

Part 7 · Check yourself

Check yourself

Exam-style questions. Anything you miss goes into your review queue.

1. To study teen screen habits, a researcher posts a link on a gaming forum and analyzes the 2,000 replies. Which problem most limits who the results describe?

  1. There was no random assignment to screen habits
  2. Sampling bias: forum users may differ from typical teens
  3. The sample of 2,000 replies is too small to describe teens in general
  4. A wording effect in the questions that the forum survey used
Show the answer

A convenience sample takes whoever is easy to reach. Gaming-forum users differ from teens in general, so the sample is biased no matter how many reply.

  • There was no random assignment to screen habits: Random assignment matters for causal claims in experiments; the question is who the results describe.
  • Correct: Sampling bias: forum users may differ from typical teens: Right: a convenience sample of forum users is unlikely to be representative.
  • The sample of 2,000 replies is too small to describe teens in general: 2,000 is a large sample; size cannot fix a sample drawn from the wrong place.
  • A wording effect in the questions that the forum survey used: The stem says nothing about the wording; the problem it describes is where the sample came from.

2. A study of a new reading program randomly selects 60 schools from across the state, then randomly assigns 30 to use the program and 30 to continue as usual. What does each random step allow?

  1. Random selection supports a causal claim; random assignment supports generalizing to the state’s schools
  2. Both steps do the same job, so the second one adds nothing to the study design
  3. Neither step matters much, because 60 schools is a large enough sample on its own
  4. Random selection supports generalizing to the state’s schools; random assignment supports a causal claim
Show the answer

Random sampling (selection) lets results generalize to the population; random assignment to conditions lets an experiment show cause and effect.

  • Random selection supports a causal claim; random assignment supports generalizing to the state’s schools: This swaps the two: assignment is what allows the cause-and-effect conclusion.
  • Both steps do the same job, so the second one adds nothing to the study design: They do different jobs: one is about who is studied, the other about which condition each gets.
  • Neither step matters much, because 60 schools is a large enough sample on its own: Size does not replace either step: a large biased sample is still biased.
  • Correct: Random selection supports generalizing to the state’s schools; random assignment supports a causal claim: Right: sampling decides who is studied, assignment decides which condition.

3. An experiment on a memory strategy used 40 psychology majors at one private university, ages 19 to 22. A teacher asks whether the results apply to her ninth graders. Which answer is best supported?

  1. Only partly: it helped young adults, but ninth graders are younger
  2. No, because 40 participants is too few for a study to give useful results
  3. Yes, because the study was an experiment that used random assignment
  4. Yes, because memory works in much the same way in people of any age
Show the answer

Generalizability depends on how similar the participants are to the group you want to describe: here, age and type of school differ.

  • Correct: Only partly: it helped young adults, but ninth graders are younger: Right: it argues from participant characteristics, both sides.
  • No, because 40 participants is too few for a study to give useful results: Generalizing depends on who the participants were, not on the count alone.
  • Yes, because the study was an experiment that used random assignment: Random assignment supports a causal claim for these participants; it says nothing about other ages.
  • Yes, because memory works in much the same way in people of any age: Ages, schooling and backgrounds differ; a claim about everyone needs evidence from different groups.

Data

Who answered an online sleep survey?

Ages of survey respondents compared with the town’s adult population (invented data for practice)
Age groupShare of respondentsShare of the town’s adults
18 to 2955%20%
30 to 4930%35%
50 to 6912%30%
70 and older3%15%

4. What does the table show about the survey’s sample?

  1. Young adults are overrepresented; older adults, underrepresented
  2. The sample matches the town’s age makeup closely in each group
  3. Older adults are overrepresented in the sample
  4. Adults aged 30 to 49 are missing from the sample
Show the answer

Compare each row: a share among respondents far above the town’s share means overrepresented; far below means underrepresented.

  • Correct: Young adults are overrepresented; older adults, underrepresented: Right: 55% versus 20% for ages 18 to 29, and 3% versus 15% for 70 and older.
  • The sample matches the town’s age makeup closely in each group: Every age group differs, some by more than 30 points.
  • Older adults are overrepresented in the sample: The two oldest groups make up a smaller share of respondents than of the town.
  • Adults aged 30 to 49 are missing from the sample: They make up 30% of respondents, close to their 35% of the town.

5. The survey found that most respondents sleep under seven hours. Why should the town not conclude that most of its adults do?

  1. A survey is a poor way to measure how long people sleep
  2. The sample is biased toward younger adults
  3. The survey was not an experiment, so it has no results
  4. Seven hours is not an operational definition of sleep
Show the answer

An online survey that mostly reached young adults is a biased sample, so its results describe young respondents more than the town.

  • A survey is a poor way to measure how long people sleep: Surveys can ask about sleep; the problem here is who answered.
  • Correct: The sample is biased toward younger adults: Right: a sample that misrepresents the population cannot describe it.
  • The survey was not an experiment, so it has no results: Surveys produce descriptive results; the issue is the sample.
  • Seven hours is not an operational definition of sleep: The measure is clear enough; the problem is who answered.

6. Which change would most improve how well the survey represents the town’s adults?

  1. Leave the online survey open longer to get more replies
  2. Ask the respondents to share the link with their friends
  3. Contact a random sample from the town’s address list
  4. Randomly assign respondents to two versions of the survey
Show the answer

Random sampling from a list of the whole population gives every adult the same chance of being chosen, so all age groups appear in about their true shares.

  • Leave the online survey open longer to get more replies: More replies from the same kind of respondents does not fix the bias.
  • Ask the respondents to share the link with their friends: Friends of young respondents are likely to be young too.
  • Correct: Contact a random sample from the town’s address list: Right: random sampling from the whole population.
  • Randomly assign respondents to two versions of the survey: Random assignment compares conditions; it does not change who is in the sample.

Part 8 · Summary

Summary

A sample stands in for a population. Random sampling makes a representative sample likely, while convenience and self-selected samples are biased however large they are. Random sampling supports generalizing to the population; random assignment supports a causal claim. Whether results generalize to a group depends on how much the participants resemble that group in age, culture and setting.

Part 9 · Up next

What comes next

Part 10 · Connections

Connections

Share to Google Classroom (opens in a new tab)

Question set for this topic