An observational study can show that two variables are related, but not why. To learn whether a treatment causes a change, you have to impose it, and you have to do it so that nothing else can explain the result. This page covers the vocabulary of experiments, the four principles of a good design, the three standard designs, and what random assignment lets you conclude.
The parts of an experiment
- Experimental units are the individuals treatments are given to. When they are people, they are called subjects.
- A treatment is a condition imposed on the units. The explanatory variable in an experiment is often called a factor, and its values are levels: "glasses" is a factor with levels "blue-light blocking" and "clear".
- The response variable is measured after the treatments: sleep quality, push-up gain, tomato weight.
- A control group gives a baseline: no treatment, the standard treatment, or a placebo, a fake treatment that looks real. People often improve just because they expect to, the placebo effect; a placebo group experiences it too, so the comparison removes it.
Four principles
- Comparison. Use two or more treatments (one may be a control).
- Random assignment. Let a chance process decide which units get which treatment. Then other variables, measured or not, are balanced between the groups on average, so they cannot explain a difference.
- Replication. Give each treatment to enough units that chance differences between individuals average out.
- Control. Keep other conditions the same for every group (same instructions, same timing, same measurement).
Blinding adds protection against expectations. In a single-blind experiment the subjects do not know which treatment they got; in a double-blind one, neither do the people who interact with them or measure the response.
Three designs
| Design | How units are assigned | Use it when |
|---|---|---|
| Completely randomized | every unit assigned to a treatment by chance, all at once | units are fairly similar |
| Randomized block | group similar units into blocks, then randomize within each block | a known variable (experience, location) affects the response |
| Matched pairs | pairs of similar units get one treatment each, or each unit gets both treatments in random order | units can be paired closely, or can receive both treatments |
Blocking works like stratifying in a sample, but for assigning treatments. If beginners and experienced members gain push-ups at very different rates, a completely randomized design lets that difference blur the comparison. Comparing programs within each block removes it, so the effect of the program is easier to see. A matched pairs design is a block design with blocks of size two.
Describe a randomized block design, with the randomization. A gym compares training programs P and Q with 16 beginners and 24 experienced volunteers.
Blocks: beginners and experienced members, because experience likely affects push-up gains.
Randomization in the beginner block: write the 16 names on identical slips, mix them in a container, draw 8 slips; those 8 do program P and the other 8 do program Q. Repeat in the experienced block: 24 slips, draw 12 for P, the other 12 do Q.
Response: each member’s gain in push-ups after six weeks, compared between programs within each block.
A procedure that says only "randomly assign the members" loses credit. Say how: slips or labels, mixing or generating, and which group the chosen units join.
Statistically significant
Even if a treatment does nothing, random assignment will put slightly different units in each group, so the group results will differ a little by chance. A difference is statistically significant when it is too large to be explained by that chance variation alone. Later units show how to measure this; for now, "statistically significant" means "bigger than chance would usually produce".
What can you conclude?
Random assignment and random sampling do different jobs. Random assignment allows a cause-and-effect conclusion. Random sampling allows generalizing to the population the sample came from.
| Random assignment | No random assignment | |
|---|---|---|
| Random sample | cause and effect, for the population | a relationship, for the population |
| No random sample | cause and effect, for units like those studied | a relationship, for the units studied |
State the scope of inference. Sixty student volunteers were randomly assigned to blue-light or clear glasses, double-blind, and the blue-light group slept much better, a statistically significant difference.
Cause: yes, because the glasses were randomly assigned.
Generalizing: only to students like these volunteers, because they were not a random sample of all college students.
In context: "Because the students were randomly assigned to the glasses, the study gives evidence that blue-light glasses cause better sleep quality for students like those in the study. Because they were volunteers, the result should not be generalized to all college students."