Logistic Regression & Sigmoid Decision Boundary

Despite having "regression" in its name, Logistic Regression is one of the most widely used supervised classification algorithms in data science. It models the probability that a given input sample belongs to a particular discrete class (e.g., spam vs. not spam, loan approved vs. denied, malignant vs. benign).

Why Linear Regression Fails on Classification

If we attempt to fit a standard linear regression line ŷ = wᵖx + b to binary class labels {0, 1}:

• Linear regression predictions are unbounded; it outputs values like -2.4 or +4.8, which cannot be interpreted as valid probabilities.
• Extreme positive or negative outliers tilt the linear regression line, shifting the decision threshold and destroying accuracy.

The Sigmoid (Logistic) Function

To map unbounded real numbers z = wᵖx + b into valid probabilities strictly bounded between 0 and 1, Logistic Regression applies the Sigmoid Activation Function σ(z):

The Log-Loss Cost Function (Binary Cross-Entropy)

We cannot use Mean Squared Error (MSE) to train Logistic Regression because substituting the sigmoid function into MSE results in a non-convex loss surface with countless local minima. Instead, we use Binary Cross-Entropy (Log-Loss), which is guaranteed to be convex:

J(w) = - (1/m) ∑ [ yᵢ log(ŷᵢ) + (1 - yᵢ) log(1 - ŷᵢ) ]

• If actual y = 1 and predicted ŷ → 1, Loss = 0. If predicted ŷ → 0, Loss → ∞ (Massive penalty).
• If actual y = 0 and predicted ŷ → 0, Loss = 0. If predicted ŷ → 1, Loss → ∞.

Python Implementation: Breast Cancer Tumor Classification

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, roc_auc_score

# 1. Load clinical dataset: 569 patient tumors with 30 cell nuclei features
data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

# 2. Scaling is critical for gradient convergence in Logistic Regression
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc = scaler.transform(X_test)

# 3. Fit Model
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train_sc, y_train)

# 4. Predict discrete class vs raw probability
y_pred = clf.predict(X_test_sc)
y_probs = clf.predict_proba(X_test_sc)[:, 1]

print(f"Test Set ROC-AUC Score: {roc_auc_score(y_test, y_probs):.4f}")
print("
Classification Report:
", classification_report(y_test, y_pred, target_names=data.target_names))