Simple Linear Regression

Overview

Simple linear regression models the relationship between two variables by fitting a straight line to the data. It predicts the value of a dependent variable ($Y$) based on an independent variable ($X$).

The Model

$$\hat{y} = \beta_0 + \beta_1 x$$

Or equivalently:

$$Y = \beta_0 + \beta_1 X + \varepsilon$$

Where:

• $\hat{y}$ = predicted value of $Y$
• $\beta_0$ = y-intercept
• $\beta_1$ = slope
• $\varepsilon$ = error term

Least Squares Method

Minimizes the sum of squared residuals:

$$SSE = \sum (y_i - \hat{y}_i)^2$$

Formulas

Slope ($\beta_1$)

$$\beta_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$$

Or:

$$\beta_1 = r \times \frac{s_Y}{s_X}$$

Or:

$$\beta_1 = \frac{n\sum x_i y_i - (\sum x_i)(\sum y_i)}{n\sum x_i^2 - (\sum x_i)^2}$$

Intercept ($\beta_0$)

$$\beta_0 = \bar{y} - \beta_1 \bar{x}$$

Interpretation

Slope ($\beta_1$)

For each 1-unit increase in $X$, $Y$ is predicted to change by $\beta_1$ units.

Intercept ($\beta_0$)

The predicted value of $Y$ when $X = 0$. (May not be meaningful if $X = 0$ is outside the data range)

Sum of Squares

$$SST = \sum (y_i - \bar{y})^2 \quad \text{(Total variation)}$$ $$SSR = \sum (\hat{y}_i - \bar{y})^2 \quad \text{(Explained variation)}$$ $$SSE = \sum (y_i - \hat{y}_i)^2 \quad \text{(Unexplained variation)}$$ $$SST = SSR + SSE$$

Standard Error of the Estimate

$$s = \sqrt{\frac{SSE}{n - 2}}$$

Measures typical prediction error.

Assumptions

1. Linearity: Relationship is linear
2. Independence: Observations are independent
3. Normality: Errors are normally distributed
4. Homoscedasticity: Constant variance of errors

Example

Data: Hours studied ($X$) vs Test score ($Y$)

$x$	$y$
1	50
2	55
3	60
4	65
5	70

$$\bar{x} = 3, \quad \bar{y} = 60$$ $$\sum (x_i - \bar{x})(y_i - \bar{y}) = (-2)(-10) + (-1)(-5) + 0 + (1)(5) + (2)(10) = 50$$ $$\sum (x_i - \bar{x})^2 = 4 + 1 + 0 + 1 + 4 = 10$$ $$\beta_1 = \frac{50}{10} = 5$$ $$\beta_0 = 60 - 5(3) = 45$$ $$\text{Regression equation: } \hat{y} = 45 + 5x$$

Interpretation: Each additional hour of study increases the predicted score by 5 points.

Making Predictions

For $x = 4$ hours:

$$\hat{y} = 45 + 5(4) = 65$$

Extrapolation Warning

Don't predict outside the range of observed $X$ values.

Residuals

$$e_i = y_i - \hat{y}_i$$

Residuals should be:

• Random (no pattern)
• Centered around 0
• Constant spread (homoscedasticity)
• Normally distributed

Residual Plots

Plot residuals vs predicted values or $X$:

• Look for patterns
• Check for non-constant variance
• Identify outliers