Eigenvalues, Eigenvectors & Singular Value Decomposition (SVD)

When working with high-dimensional datasets containing hundreds of features, multicollinearity and the "curse of dimensionality" can degrade machine learning performance. Matrix Decompositions — specifically Eigenvalues, Eigenvectors, and Singular Value Decomposition (SVD) — allow us to compress data while preserving its essential variance.

1. The Eigenvalue Equation

For a square matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\), a non-zero vector \(\mathbf{v}\) is an eigenvector with corresponding scalar eigenvalue \(\lambda\) if:

To find the eigenvalues, solve the characteristic polynomial: \(\det(\mathbf{A} - \lambda \mathbf{I}) = 0\).

2. Singular Value Decomposition (SVD) Architecture

Real-world data matrices \(\mathbf{X}\) with \(m\) rows and \(n\) columns are almost never square. SVD factors any matrix into three constituent matrices:

3. Dimensionality Reduction: PCA via SVD in Python

import numpy as np
from sklearn.decomposition import PCA

# Generate synthetic 4-dimensional correlated dataset
np.random.seed(42)
X = np.random.randn(100, 4) @ np.array([
    [2.0, 1.5, 0.2, 0.1],
    [0.0, 1.2, 0.8, 0.3],
    [0.0, 0.0, 0.5, 0.1],
    [0.0, 0.0, 0.0, 0.2]
])

# Fit PCA using Scikit-Learn (internally powered by fast randomized SVD)
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)

print(f"Original Shape: {X.shape}")
print(f"Reduced Shape:  {X_reduced.shape}")
print(f"Explained Variance Ratio: {pca.explained_variance_ratio_}")
print(f"Total Variance Retained:   {sum(pca.explained_variance_ratio_) * 100:.2f}%")