Regression Analysis and Time Series Forecasting

Linear Regression

Predicts a continuous numerical outcome from input features.

Simple Linear Regression

$$\begin{aligned} Y &= \beta_{0} + \beta_{1}X + ε \\ &\beta_{0}: \text{intercept}, \beta_{1}: \text{slope} (\text{coefficient}), ε: \text{error} \end{aligned}$$

Ordinary Least Squares (OLS): Finds β₀ and β₁ that minimize the sum of squared residuals.

Multiple Linear Regression

$$Y = \beta_{0} + \beta_{1}X_{1} + \beta_{2}X_{2} + ... + \beta_{n}X_{n} + ε$$

Key assumptions: Linearity, normality of residuals, homoscedasticity (constant variance), independence, no multicollinearity.

---

Regularization

Penalizes large coefficients to prevent overfitting:

Ridge (L2 Regularization)

$$\text{Cost} = MSE + \lambda\sum\beta_{i}^2$$

• Shrinks coefficients toward zero but never exactly to zero
• Effective for multicollinearity

Lasso (L1 Regularization)

$$\text{Cost} = MSE + \lambda\sum|\beta_{i}|$$

• Drives some coefficients exactly to zero → automatic feature selection
• Produces sparse models

ElasticNet

Combination of L1 + L2. Useful when features are many and correlated.

---

Time Series Analysis

Time series data: Observations collected in temporal order (stock prices, temperature, sales).

Decomposition

$$\text{Time series} = \text{Trend} + \text{Seasonality} + \text{Residual}$$

Trend: Long-term direction (rising/falling). Seasonality: Recurring patterns (summer AC sales spike). Residual: Unexplained variation.

ARIMA Model

AR (AutoRegressive): Current value predicted from past values

$$Y_{t} = c + φ_{1}Y_{t-1} + ... + φₚY_{t-}ₚ + ε_{t}$$

I (Integrated): Differencing to achieve stationarity.

MA (Moving Average): Current value predicted from past errors.

ARIMA(p, d, q): p=AR order, d=differencing steps, q=MA order.

SARIMA: Extends ARIMA with seasonal terms.

---

Key Concept Cards

Ridge vs. Lasso ★★★★★ : Ridge=L2, shrinks coefficients (not to zero), handles multicollinearity. Lasso=L1, zeroes out coefficients (feature selection), sparse model.

Time Series Decomposition ★★★★★ : Trend + Seasonality + Residual. Remove seasonality before modeling if present.

ARIMA(p,d,q) ★★★★☆ : AR(autoregression)+I(differencing)+MA(moving average). Requires stationary data — difference first if non-stationary.

---

Practice Quiz

Q1. A house price model has 50 features. How can you automatically identify which features matter most?

Use Lasso regression. L1 regularization drives unimportant feature coefficients exactly to zero, effectively performing automatic feature selection from the 50 candidates. Ridge shrinks coefficients toward zero but keeps all of them — no selection effect.

Q2. Monthly sales data shows a sharp peak every December. Which model should you use instead of basic ARIMA?

SARIMA (Seasonal ARIMA). Standard ARIMA doesn’t handle seasonality. SARIMA adds seasonal AR, differencing, and MA terms to capture periodic patterns. Facebook Prophet is another strong option — it automatically handles seasonality and holiday effects.