Convolutional Neural Networks (CNNs) and Computer Vision Architectures

When fully-connected Multi-Layer Perceptrons (MLPs) are applied to image processing, they fail catastrophically due to two fundamental architectural flaws: Parameter Explosion and Lack of Spatial Invariance. A modest 1000x1000 RGB image fed into a hidden layer of 1,000 neurons requires ( (1000 imes 1000 imes 3) imes 1000 = 3,000,000,000 ) parameters for a single layer! Furthermore, an MLP treats pixels as independent coordinates; if an object shifts two pixels to the right, all learned weights are rendered useless.

Convolutional Neural Networks (CNNs) (pioneered by Yann LeCun with LeNet-5 in 1989) solve this through two core inductive biases: Local Receptive Fields and Weight Sharing.

The 3 Key Operations of a Convolutional Layer

1. 2D Convolution & Learnable Filters

Instead of connecting every neuron to every pixel, a small learnable matrix called a Kernel / Filter (typically ( 3 imes 3 ) or ( 5 imes 5 )) slides across the input image. At each spatial position, it computes the element-wise dot product between the kernel and the overlapping image patch, outputting a 2D Feature Map:

Weight Sharing: The exact same filter weights are used across the entire image. If a filter learns to detect a vertical edge in the top-left corner, it automatically detects vertical edges anywhere else in the image (Translation Equivariance).

2. Padding (Valid vs Same) & Strides

• Stride (( s )): The step size with which the kernel slides across the image. A stride of 2 halves the spatial resolution.
• Padding (( p )): Adding border pixels (typically zeros) around the input:
  • Valid Padding (( p=0 )): No padding; output feature map shrinks: ( W_{ ext{out}} = lfloor rac{W - K}{s} floor + 1 ).
  • Same Padding: Padding chosen such that output spatial dimensions equal input dimensions: ( p = rac{K-1}{2} ) (for odd ( K ) and ( s=1 )).

3. Pooling Layers (Max Pooling & Average Pooling)

Pooling downsamples spatial dimensions, reducing computational overhead and providing Translation Invariance (small shifts or distortions in input pixels do not alter the pooled summary). A ( 2 imes 2 ) Max Pooling layer with stride 2 takes the maximum value within each ( 2 imes 2 ) window, reducing spatial area by 75% without adding any learnable parameters.

Milestone CNN Architectures That Revolutionized AI

The ResNet Revolution: Skip Connections

Prior to ResNet, stacking more layers caused performance to degrade rapidly due to vanishing gradients during backpropagation. Kaiming He et al. realized it is easier for a layer to learn a residual mapping ( mathcal{F}(mathbf{x}) = mathcal{H}(mathbf{x}) - mathbf{x} ) than an unreferenced mapping ( mathcal{H}(mathbf{x}) ):

Because the gradient ( rac{partial mathbf{y}}{partial mathbf{x}} = rac{partial mathcal{F}}{partial mathbf{x}} + 1 ), the identity term ( +1 ) allows gradient signals to flow backward unhindered through hundreds of layers directly to the early layers!

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super(ResidualBlock, self).__init__()
        self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)

    def forward(self, x):
        identity = x  # Store residual shortcut
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        
        # Add residual skip connection BEFORE final ReLU
        out += identity
        out = self.relu(out)
        return out

# Instantiate block with 64 channels
res_block = ResidualBlock(channels=64)
sample_tensor = torch.randn(8, 64, 32, 32)  # Batch: 8, Channels: 64, H: 32, W: 32
output_tensor = res_block(sample_tensor)

print(f"Input Shape  : {sample_tensor.shape}")
print(f"Output Shape : {output_tensor.shape} (Dimensions perfectly preserved via skip connection!)")