Hypothesis Testing & Modern A/B Testing Framework

In modern technology organizations, product decisions are not made by executive intuition; they are driven by rigorous empirical experimentation. Hypothesis Testing and A/B Testing provide the mathematical foundation to separate genuine causal impact from random noise.

Figure 6.1: The End-to-End A/B Testing Architecture — Null Hypothesis, p-values, Critical Regions & Error Matrix.

The 5-Step Hypothesis Testing Workflow

1. State the Hypotheses:
   • Null Hypothesis (\(H_0\)): There is no real difference or effect (e.g., \(\mu_{\text{variant}} - \mu_{\text{control}} = 0\)).
   • Alternative Hypothesis (\(H_1\)): There is a significant difference (e.g., \(\mu_{\text{variant}} \neq \mu_{\text{control}}\)).
2. Select Significance Level (\(\alpha\)): Typically set to \(\alpha = 0.05\) (5% false positive risk tolerance), providing 95% confidence.
3. Calculate Statistical Power & Sample Size: Ensure power \(1 - \beta \ge 0.80\) so the test has an 80%+ probability of detecting a real Minimum Detectable Effect (MDE).
4. Compute Test Statistic & \(p\)-value: Evaluate data using appropriate parametric or non-parametric test routines.
5. Make the Business Decision: If \(p < \alpha\), reject \(H_0\); otherwise, fail to reject \(H_0\).

The Statistical Test Selection Matrix

Hands-On Python: Full A/B Test Execution

import numpy as np
from scipy import stats

# Simulated checkout conversion revenue data
np.random.seed(101)
control_revenue = np.random.normal(loc=48.50, scale=12.0, size=2500)
treatment_revenue = np.random.normal(loc=50.20, scale=12.5, size=2500)

# Welch's t-test (Two-tailed)
t_stat, p_val = stats.ttest_ind(treatment_revenue, control_revenue, equal_var=False)

print(f"Control Mean:    $ {control_revenue.mean():.2f}")
print(f"Treatment Mean:  $ {treatment_revenue.mean():.2f}")
print(f"Lift:            $ {treatment_revenue.mean() - control_revenue.mean():.2f}")
print(f"t-statistic:     {t_stat:.4f}")
print(f"p-value:         {p_val:.5f}")

alpha = 0.05
if p_val < alpha:
    print("SUCCESS: Reject Null Hypothesis (Statistically Significant Lift!)")
else:
    print("INCONCLUSIVE: Fail to reject Null Hypothesis (Difference may be noise)")