Unit 5 Beta

Regression Analysis: the one-page sheet

5.1 Graphical Representations Between Two Quantitative Variables

A scatterplot plots two quantitative variables, explanatory on the horizontal axis and response on the vertical. Describe what you see with four features in context: direction, form, strength and unusual features such as outliers. A pattern in observational data shows association, not cause.

A scatterplot shows the relationship between two quantitative variables measured on the same individuals.
Describe direction, form, strength and unusual features, in context: "a strong, negative, linear association between hours of sleep and reaction time, with no outliers".
Strength is how closely points follow the pattern, not how steep it is; a curved pattern can be strong.
Association is not causation: observational data show a pattern, not its cause.

plot one point per individual: explanatory variable on the horizontal axis, response on the vertical axis direction: positive (up together), negative (one up, one down) or none form (linear or curved) and strength (how closely points follow the form) unusual features: outliers and clusters, which deserve a reason, and then describe it all in context

scatterplot: A graph of two quantitative variables measured on the same individuals: one point per individual, with the explanatory variable on the horizontal axis and the response variable on the vertical axis.
direction: The features used to describe a scatterplot. Direction: positive (both rise together), negative (one rises as the other falls) or none. Form: linear or curved. Strength: how closely the points follow the form.

5.2 Correlation

The correlation r = 1/(n − 1) ∑ z_xz_y gives the direction (sign) and strength (closeness to ±1 ) of a linear association. It has no units, ignores which variable is explanatory, is pulled by outliers, misses curved patterns, and says nothing about cause.

The correlation $r$ measures the direction and strength of a linear association; $-1 \le r \le 1$ .
Interpret: " $r = 0.95$ : a strong, positive, linear association between hours studied and quiz score."
$r$ has no units, does not change when units change, and is the same if $x$ and $y$ swap.
$r$ is not resistant, measures only linear patterns, and never shows cause.

units disappear, so r is the same in any units give positive products z_xz_y ; mixed points give negative products r is positive for a rising pattern, negative for a falling one, and near ±1 when points hug a line always plot first: a curve can have r near 0, and one outlier can change r a lot

correlation: The correlation coefficient <math class="stat-m"><semantics><mrow><mi>r</mi><mo>=</mo><mfrac><mn>1</mn><mrow><mi>n</mi><mo>−</mo><mn>1</mn></mrow></mfrac><mo largeop="true" movablelimits="false">∑</mo><msub><mi>z</mi><mi>x</mi></msub><msub><mi>z</mi><mi>y</mi></msub></mrow><annotation encoding="application/x-tex">r = \frac{1}{n - 1}\sum z_x z_y</annotation></semantics></math>: a number from <math class="stat-m"><semantics><mrow><mo form="prefix" lspace="0" rspace="0">−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">-1</annotation></semantics></math> to 1 that measures the direction and strength of the linear association between two quantitative variables. It has no units and is not resistant to outliers.

5.3 Linear Regression Models

A regression line y-hat = a + bx predicts the response from the explanatory variable. Interpret the slope as the predicted change in y for each one-unit increase in x , and the intercept as the predicted y at x = 0 when that makes sense, always in context and with "predicted". Do not extrapolate beyond the data.

A regression line $\hat{y} = a + bx$ gives predicted values $\hat{y}$ of the response for values of the explanatory variable.
Slope: "For each additional [unit of $x$ ], the predicted [ $y$ ] increases (decreases) by about $|b|$ [units of $y$ ]."
Intercept: "When [ $x$ ] is 0, the predicted [ $y$ ] is $a$ ." Interpret it only when $x = 0$ makes sense.
Extrapolation, predicting outside the range of the data, is unreliable.

a line y-hat = a + bx can summarize it and predict y from x the result y-hat is a predicted value: typical for that x , not exact for any one individual interpret it as "for each additional ..., the predicted ... increases (decreases) by about |b| ..." predicting far outside that range (extrapolation) can give nonsense, and the intercept means something only when x = 0 is reasonable

regression line: A line <math class="stat-m"><semantics><mrow><mover accent="true"><mi>y</mi><mo stretchy="false">^</mo></mover><mo>=</mo><mi>a</mi><mo>+</mo><mi>b</mi><mi>x</mi></mrow><annotation encoding="application/x-tex">\hat{y} = a + bx</annotation></semantics></math> that describes how a response variable <math class="stat-m"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math> changes, on average, as an explanatory variable <math class="stat-m"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math> changes. Its output <math class="stat-m"><semantics><mrow><mover accent="true"><mi>y</mi><mo stretchy="false">^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{y}</annotation></semantics></math> (y-hat) is a predicted value of <math class="stat-m"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math>.
extrapolation: Using a regression line to predict for values of <math class="stat-m"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math> outside the range of the data. Such predictions are unreliable because the pattern may not continue.

5.4 Residuals

A residual is actual minus predicted, y − y-hat : positive when the point is above the line, negative when below, in the units of y . A residual plot shows the residuals against x ; random scatter around 0 means a line is appropriate, while a curve or a fan shows what the line misses.

Residual $= y - \hat{y}$ , actual minus predicted, in the units of $y$ .
Positive residual: actual above the line (underpredicted). Negative: below the line (overpredicted).
A residual plot graphs residuals against $x$ (or against $\hat{y}$ ).
No pattern: a line is appropriate. A curved pattern: a line is not. A fan: predictions are less precise where the spread is larger.

each point misses by its residual, y − y-hat , the vertical gap from the line has a positive residual: the line underpredicts; below the line, negative: it overpredicts the residual plot removes the trend, so what is left is the pattern the line missed use the residual plot to decide whether a linear model is appropriate

residual: The difference between an observed value and the value a regression line predicts: residual <math class="stat-m"><semantics><mrow><mo>=</mo><mi>y</mi><mo>−</mo><mover accent="true"><mi>y</mi><mo stretchy="false">^</mo></mover></mrow><annotation encoding="application/x-tex">= y - \hat{y}</annotation></semantics></math>. Positive when the point lies above the line, negative when below.
residual plot: A scatterplot of the residuals against the explanatory variable (or the predicted values). Random scatter around 0 means a linear model is appropriate; a curved pattern means it is not.

5.5 Least-Squares Regression

The least-squares regression line makes the sum of squared residuals as small as possible; its slope is b = r(s_y)/(s_x) and it passes through (x-bar, y-bar) . Report r² , the share of the variation in y the line accounts for, and s , the typical prediction error in the units of y . Check for high-leverage and influential points, which can change the line a lot.

The least-squares regression line minimizes the sum of squared residuals; $b = r\frac{s_y}{s_x}$ and it passes through $(\bar{x}, \bar{y})$ .
$r^2$ : "About [ ]% of the variation in [y] is accounted for by the linear model using [x]."
$s$ : "The actual [y] typically differs from the predicted [y] by about $s$ [units of y]."
Outlier: a large residual. High-leverage point: an extreme $x$ . Influential point: removing it changes the slope or intercept a lot.

the least-squares line is the one line that makes ∑ (y − y-hat)² as small as possible gives b = r(s_y)/(s_x) and a line through (x-bar, y-bar) , so a = y-bar − bx-bar r² is the share of the variation in y the line accounts for, and s is the typical size of a residual a point with an extreme x value (high leverage) can pull the line toward itself and be influential

least-squares regression line: The least-squares regression line, <math class="stat-m"><semantics><mrow><mover accent="true"><mi>y</mi><mo stretchy="false">^</mo></mover><mo>=</mo><mi>a</mi><mo>+</mo><mi>b</mi><mi>x</mi></mrow><annotation encoding="application/x-tex">\hat{y} = a + bx</annotation></semantics></math>: the line that makes the sum of the squared residuals as small as possible. Its slope is <math class="stat-m"><semantics><mrow><mi>b</mi><mo>=</mo><mi>r</mi><mfrac><msub><mi>s</mi><mi>y</mi></msub><msub><mi>s</mi><mi>x</mi></msub></mfrac></mrow><annotation encoding="application/x-tex">b = r\frac{s_y}{s_x}</annotation></semantics></math>, and it passes through <math class="stat-m"><semantics><mrow><mo stretchy="false">(</mo><mover accent="true"><mi>x</mi><mo stretchy="true">¯</mo></mover><mo>,</mo><mover accent="true"><mi>y</mi><mo stretchy="true">¯</mo></mover><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(\bar{x}, \bar{y})</annotation></semantics></math>.
coefficient of determination: The coefficient of determination, <math class="stat-m"><semantics><mrow><msup><mi>r</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">r^2</annotation></semantics></math>: the fraction of the variation in the response variable that is accounted for by the least-squares line using the explanatory variable.
standard deviation of the residuals: The standard deviation of the residuals, <math class="stat-m"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math><wbr><math class="stat-m"><semantics><mrow><mo form="infix">=</mo><msqrt><mfrac><mrow><mo largeop="true" movablelimits="false">∑</mo><mo stretchy="false">(</mo><mi>y</mi><mo>−</mo><mover accent="true"><mi>y</mi><mo stretchy="false">^</mo></mover><msup><mo stretchy="false">)</mo><mn>2</mn></msup></mrow><mrow><mi>n</mi><mo>−</mo><mn>2</mn></mrow></mfrac></msqrt></mrow><annotation encoding="application/x-tex">= \sqrt{\frac{\sum (y - \hat{y})^2}{n - 2}}</annotation></semantics></math>: the typical size of a prediction error, in the units of the response variable.
influential point: A point whose removal would change the least-squares line a lot. Points with <math class="stat-m"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math> values far from <math class="stat-m"><semantics><mrow><mover accent="true"><mi>x</mi><mo stretchy="true">¯</mo></mover></mrow><annotation encoding="application/x-tex">\bar{x}</annotation></semantics></math> have high leverage and are often influential; a point with a large residual is an outlier.