Regression Modeling: Linear, Regularization & Demand Forecasting

When the target variable is continuous — predicting customer lifetime revenue, real-estate appraisal prices, demand supply quotas, or hospital length of stay — Regression Modeling is the foundational technique. In this class, we master Regularized Regression (Ridge, Lasso, ElasticNet) and residual diagnostics.

Regularization Mathematics: Taming Overfitting

• Ordinary Least Squares (OLS) Loss: \[ L_{\text{OLS}}(\mathbf{w}) = \frac{1}{n} \sum_{i=1}^{n} (y_i - \mathbf{w}^T \mathbf{x}_i)^2 \]
• Ridge Regression (\(L_2\) Penalty): Shrinks weights, stabilizes multicollinearity: \[ L_{\text{Ridge}}(\mathbf{w}) = L_{\text{OLS}}(\mathbf{w}) + \alpha \sum_{j=1}^{p} w_j^2 \]
• Lasso Regression (\(L_1\) Penalty): Produces sparse models via automatic feature selection: \[ L_{\text{Lasso}}(\mathbf{w}) = L_{\text{OLS}}(\mathbf{w}) + \alpha \sum_{j=1}^{p} |w_j| \]

Hands-On Python: Ridge vs. Lasso Benchmarking

import numpy as np
import pandas as pd
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Generate correlated dataset with 15 informative and 15 pure noise features
np.random.seed(42)
n_samples, n_features = 500, 30
X = np.random.randn(n_samples, n_features)
true_weights = np.zeros(n_features)
true_weights[:5] = [4.5, -3.2, 2.1, 1.8, -1.2]  # Only first 5 features matter!
y = X @ true_weights + np.random.randn(n_samples) * 0.5

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train Lasso with 5-Fold Cross-Validation for optimal alpha
lasso = LassoCV(cv=5).fit(X_train_scaled, y_train)
y_pred = lasso.predict(X_test_scaled)

print(f"Optimal Alpha:     {lasso.alpha_:.4f}")
print(f"Test RMSE:         {np.sqrt(mean_squared_error(y_test, y_pred)):.3f}")
print(f"Test MAE:          {mean_absolute_error(y_test, y_pred):.3f}")
print(f"Test R^2:          {r2_score(y_test, y_pred):.3f}")
print(f"Features Retained: {np.sum(lasso.coef_ != 0)} of {n_features} (Lasso zeroed out {np.sum(lasso.coef_ == 0)} noise features!)")