Diffusion Models and Modern Generative Image Synthesis

While GANs dominated image generation for nearly a decade, modern state-of-the-art visual AI systems — including Stable Diffusion, Midjourney, and DALL-E 3 — are powered by Denoising Diffusion Probabilistic Models (DDPM) (Sohl-Dickstein et al. 2015, Ho et al. 2020). Diffusion models achieve superior sample fidelity, complete mode coverage, and rock-solid training stability by reframing generation as iterative thermodynamic denoising.

Figure 27.1: The Diffusion Lifecycle — Forward Process (Adding Gaussian Noise) and Reverse Denoising Process (Conditioned U-Net & Text Encoders).

The Two Phases of Diffusion Models

1. The Forward Process (( q )): Gradual Noise Addition

The forward process is a fixed, non-learnable Markov chain that gradually destroys structure in an image ( mathbf{x}_0 ) over ( T ) discrete time steps (typically ( T=1000 )) by adding Gaussian noise according to a variance schedule ( eta_1, dots, eta_T ):

The Closed-Form Shortcut: Using the reparameterization trick with ( alpha_t = 1 - eta_t ) and ( ar{alpha}_t = prod_{s=1}^t alpha_s ), we can sample ( mathbf{x}_t ) at any arbitrary time step ( t ) directly in one step without simulating intermediate steps:

2. The Reverse Process (( p_ heta )): Neural Denoising

At step ( T ), the image has been transformed into pure, isotropic Gaussian noise: ( mathbf{x}_T sim mathcal{N}(0, mathbf{I}) ). If an AI model can learn to reverse each tiny noise step, it can start from pure random static and generate a brand-new, photorealistic artwork!

The reverse distribution ( p_ heta(mathbf{x}_{t-1} | mathbf{x}_t) ) is parameterized by a U-Net Neural Network with time-step embeddings. Instead of predicting the clean image directly, Jonathan Ho et al. proved that training the network ( oldsymbol{epsilon}_ heta(mathbf{x}_t, t) ) to predict the noise vector that was added yields optimal results:

Latent Diffusion Models (LDM / Stable Diffusion)

Running 1000 diffusion steps directly on high-resolution pixel matrices (e.g., ( 1024 imes 1024 imes 3 )) is computationally crushing. Latent Diffusion Models (Rombach et al., 2022) solved this by dividing the process into two stages:

1. Perceptual Compression: A pre-trained Variational Autoencoder (VAE) compresses the image into a low-dimensional latent space (downsampling spatial resolution by ( 8 imes )).
2. Latent Denoising: The U-Net performs diffusion strictly within the lightweight latent space, speeding up training and inference by orders of magnitude. A VAE Decoder then projects the denoised latent back into high-resolution pixels!

Text Conditioning & Classifier-Free Guidance (CFG)

To generate images matching a text prompt (e.g., "A futuristic cyberpunk city in the rain"), the prompt is encoded into vector representations using a pre-trained language model (CLIP ViT-L or T5-XXL) and injected into the U-Net via cross-attention layers.

Classifier-Free Guidance (CFG): During generation, the model predicts noise both with the text prompt (( oldsymbol{epsilon}_ heta(mathbf{x}_t, c) )) and without the prompt (( oldsymbol{epsilon}_ heta(mathbf{x}_t, emptyset) )). The final noise estimate pushes generation away from the unconditioned direction:

Where ( s ) is the Guidance Scale (typically 7.0 to 9.0). Higher scale enforces strict adherence to the text prompt.

import torch
from diffusers import StableDiffusionPipeline

# 1. Load Pretrained Latent Diffusion Pipeline (FP16 for memory efficiency)
model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
)

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = pipe.to(device)

# 2. Define Prompts and Hyperparameters
prompt = "A high-tech artificial intelligence laboratory with glowing teal holograms, 8k resolution, cinematic lighting"
negative_prompt = "blurry, low quality, distorted, watermark, oversaturated"

# 3. Generate Image
image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=30,      # Number of denoising steps (e.g. DDIM/DPMSolver)
    guidance_scale=7.5           # Classifier-Free Guidance (CFG) scale
).images[0]

# Save output
image.save("ai_generated_lab.png")
print("Image successfully synthesized and saved to ai_generated_lab.png!")