Activation Functions (ReLU, GELU, Sigmoid) & Loss Functions

Q: What is the 'Dying ReLU' problem?

If a neuron's linear combination z becomes negative across all training samples, ReLU outputs 0 and its derivative is 0. The neuron becomes inactive ('dead') and will never update its weights again.

Q: Why do modern Transformers like GPT prefer GELU over standard ReLU?

GELU is smooth and non-monotonic everywhere. It scales inputs probabilistically according to how much greater they are than other inputs, allowing subtle negative gradient flows.

Q: Why is Softmax paired with Cross-Entropy loss?

When Softmax is paired with Cross-Entropy, their combined derivative simplifies into the elegant term (ŷ - y), creating linear error scaling that makes backpropagation smooth and numerically stable.

The success of modern neural networks relies directly on two mathematical design choices: Activation Functions (which introduce non-linear expressiveness into neurons) and Loss Functions (which define what the network is penalizing during training).

Modern Activation Functions Compared

Activation Function	Mathematical Formula	Output Range	Pros & Modern Use Cases
Sigmoid (σ)	`1 / (1 + e⁻ᵞ)`	(0, 1)	Probabilistic output layer for binary classification. Not zero-centered.
Tanh (Hyperbolic Tangent)	`(eᵞ - e⁻ᵞ) / (eᵞ + e⁻ᵞ)`	(-1, +1)	Zero-centered; common in recurrent networks (LSTMs).
ReLU (Rectified Linear)	`max(0, z)`	[0, +∞)	Default standard: Blazing fast to compute; derivative=1 solves vanishing gradients.
Leaky ReLU	`max(0.01z, z)`	(-∞, +∞)	Solves the "Dying ReLU" problem by providing a small gradient for z < 0.
GELU (Gaussian Error Linear)	`z · Φ(z)`	[-0.17, +∞)	Modern LLM Standard: Smooth probabilistic gating used in GPT-4, BERT, and Claude.
Softmax	`e^{z_i} / ∑ e^{z_j}`	(0, 1), sums to 1.0	Output layer for multi-class classification (converts logits into probabilities).

Standard Machine Learning Loss Functions

1. Regression Loss Functions

Mean Squared Error (MSE / L2 Loss): (1/N) ∑ (y - ŷ)². Heavily penalizes large outlier errors; yields conditional mean.
Mean Absolute Error (MAE / L1 Loss): (1/N) ∑ |y - ŷ|. Robust to outliers; yields conditional median.
Huber Loss: Quadratic for small errors (|e| ≤ δ), linear for large errors. Combines the best of MSE and MAE.

2. Classification Loss Functions

Binary Cross-Entropy (Log-Loss): Used for 2-class problems with Sigmoid outputs: - [ y log(ŷ) + (1-y) log(1-ŷ) ].
Categorical Cross-Entropy: Used for mutually exclusive multi-class problems with Softmax outputs: - ∑ y_c log(ŷ_c).

Frequently Asked Questions (FAQ) - Student & Interview Review

Q1. What is the 'Dying ReLU' problem?

If a neuron's linear combination z becomes negative across all training samples, ReLU outputs 0 and its derivative is 0. The neuron becomes inactive ('dead') and will never update its weights again.

Q2. Why do modern Transformers like GPT prefer GELU over standard ReLU?

GELU is smooth and non-monotonic everywhere. It scales inputs probabilistically according to how much greater they are than other inputs, allowing subtle negative gradient flows.

Q3. Why is Softmax paired with Cross-Entropy loss?

When Softmax is paired with Cross-Entropy, their combined derivative simplifies into the elegant term (ŷ - y), creating linear error scaling that makes backpropagation smooth and numerically stable.