Production ML Monitoring: Data Drift & Concept Drift

Traditional software either works or crashes with an explicit stack trace. Machine learning models, however, fail silently: a model deployed to production can continue returning HTTP 200 OK responses with valid-looking numbers, while its actual real-world accuracy degrades catastrophically due to shifts in user behavior, market inflation, or sensor wear.

Data Drift vs. Concept Drift

Statistical Methods to Detect Drift

1. Kolmogorov-Smirnov (KS) Test

The two-sample KS-Test is a non-parametric test comparing the continuous cumulative empirical distributions of a feature in training data vs. live production inference data. If the resulting p-value < 0.05, statistically significant data drift is flagged!

2. Population Stability Index (PSI)

Widely used in risk scoring and fintech, PSI quantifies how much a variable has shifted over time:

PSI = ∑ (Actual% - Expected%) · ln( Actual% / Expected% )

• PSI < 0.10: No significant shift; model is stable.
• 0.10 ≤ PSI < 0.25: Moderate drift detected; schedule model review.
• PSI ≥ 0.25: Severe distribution drift! Trigger automated retraining immediately.

from scipy.stats import ks_2samp
import numpy as np

# Training distribution (baseline) vs Production inference stream (drifted)
baseline_income = np.random.normal(loc=55000, scale=12000, size=1000)
prod_income_drifted = np.random.normal(loc=68000, scale=14000, size=1000)

# Run two-sample KS test
stat, p_value = ks_2samp(baseline_income, prod_income_drifted)
print(f"KS Statistic: {stat:.4f}, P-value: {p_value:.4e}")

if p_value < 0.05:
    print("ALERT: Statistically significant Data Drift detected! Firing webhook to trigger CI/CD retraining...")