Unit 5 · Topic 5.5 Beta

Least-Squares Regression

The least-squares regression line makes the sum of squared residuals as small as possible; its slope is b = r(s_y)/(s_x) and it passes through (x-bar, y-bar) .

Practice 3: Analyze DataPractice 4: Interpret Results

Part 1 · Hook

Why this matters

Draw a line through a scatterplot by eye and someone else will draw a slightly different one. Which is best? Statistics has a precise answer: the line that makes the squared misses as small as possible. That one line, its r2r^2 and its typical error ss are what every regression printout reports, and one unusual point can change them all.

Part 2 · Before you start

What this builds on

Part 3 · Prerequisite check

Quick check before you start

1. A point is at (6,50)(6, 50) and the line predicts 46 there. What is the residual?

  1. 44
  2. −4-4
  3. 4646
  4. 5050
Show the answer

50−46=450 - 46 = 4.

  • Correct: 44:
  • −4-4:
  • 4646:
  • 5050:

2. A correlation of r=−0.9r = -0.9 describes

  1. a strong negative linear association
  2. a weak negative association
  3. a strong positive association
  4. no association
Show the answer

Size near 1, negative sign.

  • Correct: a strong negative linear association:
  • a weak negative association:
  • a strong positive association:
  • no association:

Part 4 · See it

See it first

Practice hours against free throw percentage for ten players rising steeply, with slope 2.92 for their least-squares line; an eleventh player at 16 hours and 58% pulls the line down to a slope of 0.49.
One player at 16 hours pulls the least-squares slope from 2.92 down to 0.49: an influential point. LevlPrep original diagram.

Part 5 · Step by step

How it works, step by step

  1. Every line leaves residuals; square them so misses above and below both countthe least-squares line is the one line that makes ∑(y−y^)2\sum (y - \hat{y})^2 as small as possible
  2. Solving for that linegives b=rsysxb = r\frac{s_y}{s_x} and a line through (x¯,y¯)(\bar{x}, \bar{y}), so a=y¯−bx¯a = \bar{y} - b\bar{x}
  3. Compare the scatter around the line with the scatter around y¯\bar{y}r2r^2 is the share of the variation in yy the line accounts for, and ss is the typical size of a residual
  4. Squaring makes far-off points count heavilya point with an extreme xx value (high leverage) can pull the line toward itself and be influential

Part 6 · Key ideas

Key ideas

  • The least-squares regression line minimizes the sum of squared residuals; b=rsysxb = r\frac{s_y}{s_x} and it passes through (x¯,y¯)(\bar{x}, \bar{y}).
  • r2r^2: "About [ ]% of the variation in [y] is accounted for by the linear model using [x]."
  • ss: "The actual [y] typically differs from the predicted [y] by about ss [units of y]."
  • Outlier: a large residual. High-leverage point: an extreme xx. Influential point: removing it changes the slope or intercept a lot.

Part 7 · Misconception

A common mistake

The wrong idea: r2=0.764r^2 = 0.764 means the line predicts 76.4% of the cases correctly.

What actually happens: r2r^2 is about variation, not cases: 76.4% of the variation in yy is accounted for by the linear model using xx; the rest is scatter the line does not explain.

Part 8 · Check yourself

Check yourself

Exam-style questions. Anything you miss goes into your review queue.

Table

Computer output: car weight and fuel economy

For a random sample of 30 car models, a magazine regresses highway fuel economy (miles per gallon) on weight (thousands of pounds). Part of the output is below. (Invented data.)
Regression of fuel economy (mpg) on weight (1,000 lb)
PredictorCoef
Constant48.7
Weight−6.25-6.25
S = 2.81R-Sq = 76.4%

1. Which is the correct interpretation of the slope?

  1. For each extra 1,000 lb, predicted fuel economy drops by about 6.25 mpg
  2. For each additional 1,000 pounds, each car loses exactly 6.25 mpg
  3. A car that weighs 0 pounds is predicted to get 6.25 mpg
  4. For each additional pound of weight, the predicted fuel economy decreases by about 6.25 mpg
Show the answer

Slope in context with "predicted" and the units of xx.

  • Correct: For each extra 1,000 lb, predicted fuel economy drops by about 6.25 mpg: Right: the slope is the predicted change in yy per one-unit increase in xx, and the unit of xx here is 1,000 pounds.
  • For each additional 1,000 pounds, each car loses exactly 6.25 mpg: The slope describes predicted values, not an exact change for each car.
  • A car that weighs 0 pounds is predicted to get 6.25 mpg: That mixes up the slope and the intercept, and a weight of 0 is not sensible.
  • For each additional pound of weight, the predicted fuel economy decreases by about 6.25 mpg: Weight is in thousands of pounds, so one unit of xx is 1,000 pounds.

2. Which is the correct interpretation of R-Sq = 76.4%?

  1. About 76.4% of the variation in fuel economy is accounted for by the linear model using weight
  2. Weight causes about 76.4% of the variation in a car's fuel economy
  3. About 76.4% of the cars in the sample have fuel economy the line predicts correctly
  4. The correlation between weight and fuel economy for these 30 car models is 0.764
Show the answer

r2r^2: "About [r squared]% of the variation in [y] is accounted for by the linear model using [x]."

  • Correct: About 76.4% of the variation in fuel economy is accounted for by the linear model using weight: Right: r2r^2 is the fraction of the variation in yy explained by the line.
  • Weight causes about 76.4% of the variation in a car's fuel economy: r2r^2 measures how much variation the line accounts for, not cause.
  • About 76.4% of the cars in the sample have fuel economy the line predicts correctly: r2r^2 is not a share of cars; predictions are rarely exactly right.
  • The correlation between weight and fuel economy for these 30 car models is 0.764: The correlation is r=−0.764=−0.874r = -\sqrt{0.764} = -0.874, negative because the slope is negative.

3. Which is the correct interpretation of S = 2.81?

  1. A car's actual mpg is typically about 2.81 mpg away from its predicted mpg
  2. The predicted fuel economy changes by about 2.81 mpg for each additional 1,000 pounds
  3. About 2.81% of the variation in fuel economy is left unexplained
  4. The cars' weights typically differ from their mean weight by about 2.81 thousand pounds
Show the answer

ss: "The actual [y] typically differs from the predicted [y] by about ss [units]."

  • Correct: A car's actual mpg is typically about 2.81 mpg away from its predicted mpg: Right: ss, the standard deviation of the residuals, is the typical size of a prediction error, in the units of yy.
  • The predicted fuel economy changes by about 2.81 mpg for each additional 1,000 pounds: That would be a slope; the slope here is −6.25-6.25.
  • About 2.81% of the variation in fuel economy is left unexplained: The unexplained share is 100%−76.4%=23.6%100\% - 76.4\% = 23.6\%; ss is in mpg, not a percent.
  • The cars' weights typically differ from their mean weight by about 2.81 thousand pounds: ss measures scatter around the line in yy units, not the spread of xx.

Table

Summary statistics for ten players

For the ten players who practice 2 to 8 hours a week, the coach computes these summaries of practice hours (xx) and free throw percentage (yy). The scatterplot is linear with no unusual points. (Invented data.)
Summary statistics
MeanStandard deviation
Practice hours, xx5.002.00
Free throws (%), yy61.406.04
Correlationr=0.966r = 0.966

4. Write the expression for the slope of the least-squares line, b=rsysxb = r\frac{s_y}{s_x}, with the values substituted, then evaluate it.

Type the expression as on the exam, with the values substituted.

Show the answer

bb=0.966(6.042.00)= 0.966\left(\frac{6.04}{2.00}\right)=2.917= 2.917 percentage points per hour.

  • Answer: 0.966⋅6.042\frac{0.966 \cdot 6.04}{2}

5. A player practices exactly 5 hours a week, the mean. What free throw percentage does the least-squares line predict?

  1. About 64.3%, one SD of xx above the mean of yy
  2. 46.8%, the intercept
  3. 5%, the mean of xx
  4. 61.4%, the mean of yy
Show the answer

y^\hat{y} at x¯\bar{x} is y¯\bar{y}.

  • About 64.3%, one SD of xx above the mean of yy: At x=x¯x = \bar{x} the prediction is exactly y¯\bar{y}.
  • 46.8%, the intercept: The intercept is the prediction at 0 hours, not at 5.
  • 5%, the mean of xx: The prediction is in percent made, the yy variable.
  • Correct: 61.4%, the mean of yy: Right: the least-squares line always passes through (x¯,y¯)(\bar{x}, \bar{y}).

6. What makes the least-squares regression line "least squares"?

  1. It minimizes the sum of the squared residuals
  2. It passes through as many data points as possible
  3. It makes each residual as small as possible
  4. It has the smallest possible slope
Show the answer

Least squares: minimize ∑(y−y^)2\sum (y - \hat{y})^2.

  • Correct: It minimizes the sum of the squared residuals: Right: among all lines, it gives the smallest ∑(y−y^)2\sum (y - \hat{y})^2.
  • It passes through as many data points as possible: It may pass through none of the points.
  • It makes each residual as small as possible: No line can make every residual small; it minimizes the sum of their squares.
  • It has the smallest possible slope: The slope is whatever minimizes the squared residuals.

Part 9 · Summary

Summary

The least-squares regression line makes the sum of squared residuals as small as possible; its slope is b=rsysxb = r\frac{s_y}{s_x} and it passes through (x¯,y¯)(\bar{x}, \bar{y}). Report r2r^2, the share of the variation in yy the line accounts for, and ss, the typical prediction error in the units of yy. Check for high-leverage and influential points, which can change the line a lot.

Part 10 · Up next

What comes next

This is the last topic in the course.

Part 11 · Connections

Connections

Share to Google Classroom (opens in a new tab)

Question set for this topic