Diffusion Models and DiT: Noise Schedules, DDIM, and Scalable Transformers

A diffusion system is five separable parts. A perceptual compressor maps pixels into the space the model works in. A probability path and its training objective say what the network regresses at each noise level. A backbone computes that regression. A sampler integrates the learned field back to data. A conditioning path carries the class label, the prompt, or the control image into the backbone. The literature in this chapter is almost entirely a record of papers that change one of these and hold the others fixed, and that is why its numbers can be compared at all. DDIM changes only the sampler and reuses the DDPM weights untouched. Flow matching changes only the loss and reuses the EDM U-Net unchanged. DiT changes only the backbone and inherits LDM's frozen VAE. ControlNet, Video LDM and AnimateDiff change only the conditioning path and freeze the image model underneath. Table tbl:dit-what-changed indexes the milestones by which part each one moved, and Table tab:diffusion-sota gives the reported quality, scale and sampling cost of the same lineage. Read together they answer a question a leaderboard alone cannot, which is not which model is best but which component the gain came from.

Diffusion models are now the most widely used family of generative models, surpassing GANs in image quality while offering stable training and mode coverage. The foundational work on Denoising Diffusion Probabilistic Models (DDPM) (Ho et al. 2020) showed that iteratively denoising samples from Gaussian noise produces high-quality images, reporting an FID of 3.17 on CIFAR-10, the lowest published at the time, and competitive results on LSUN.[fn:s0-ddpm-impl] This work established the connection between diffusion models and denoising score matching, providing both theoretical grounding and practical training recipes.

Subsequent work focused on accelerating sampling, which originally required 1000+ steps. Denoising Diffusion Implicit Models (DDIM) (Song et al. 2021) introduced a non-Markovian formulation enabling 10-50× faster sampling by treating the reverse process as solving an ODE rather than simulating an SDE. Improved DDPM (Nichol and Dhariwal 2021) showed that learning the variance schedule and using fewer steps maintains quality while reducing computation. DPM-Solver++ (Lu et al. 2022) further improved sampling efficiency through high-order ODE solvers specifically designed for guided diffusion, enabling 15-20 step generation.

The architecture evolution moved from U-Net to Transformer backbones. Diffusion Models Beat GANs (Dhariwal and Nichol 2021) showed that architectural improvements (attention at multiple resolutions, adaptive group normalization) and classifier guidance enable diffusion models to surpass BigGAN on ImageNet. The EDM framework (Karras et al. 2022) unified the design space, reaching an FID of 1.79 on class-conditional CIFAR-10 (1.97 unconditional) through careful analysis of noise schedules, preconditioning, and sampling.

The Diffusion Transformer (DiT) (Peebles and Xie 2023) replaced U-Net with a Vision Transformer operating on latent patches, showing clean scaling laws: higher compute (GFLOPs) consistently yields lower FID. DiT-XL/2 achieved 2.27 FID on ImageNet 256×256, establishing transformers as viable diffusion backbones. U-ViT (Bao et al. 2023) showed that treating time, condition, and image patches uniformly as tokens with long skip connections achieves comparable results with simpler architecture.

Text-to-image synthesis scaled these techniques to billions of parameters and web-scale data. Latent Diffusion Models (LDM) (Rombach et al. 2022) moved diffusion to the latent space of a pretrained autoencoder, reducing computation by 4-16× while maintaining quality. GLIDE (Nichol et al. 2022) showed classifier-free guidance outperforms CLIP guidance for photorealism. Imagen (Saharia et al. 2022) showed that scaling the text encoder (T5-XXL) matters more than scaling the image model, and reported markedly better text-image alignment on DrawBench. DALL-E 2 (Ramesh et al. 2022) used a two-stage approach, a CLIP prior generating image embeddings followed by a diffusion decoder.

The Stable Diffusion family made text-to-image generation widely available through open weights. SDXL (Podell et al. 2023) scaled to 3× larger U-Net with dual text encoders, achieving quality competitive with closed-source models. PixArt-$\alpha$ (Chen et al. 2023) showed efficient training by decomposing into stages (pixel dependency, text alignment, aesthetics), requiring only 10.8% of Stable Diffusion's training compute. Stable Diffusion 3 (Esser et al. 2024) introduced the MM-DiT architecture with separate weights for image and text modalities and rectified flow formulation.

Video generation extended these techniques temporally. The mechanism is the same in all four papers below. A frame-wise image denoiser already exists and works, so the video model keeps it and inserts an operator that mixes information across frames at each spatial position. What separates the papers is where that operator sits, what is trainable, and how the model is stretched past its training window. Video Diffusion Models (Ho, Salimans, et al. 2022) added 3D attention for temporal coherence, training the whole stack jointly on a mixture of images and clips. Imagen Video (Ho, Chan, et al. 2022) used cascaded diffusion with spatial and temporal super-resolution, so temporal mixing is spread over seven separately trained models. Video LDM (Blattmann, Rombach, et al. 2023) inserted temporal layers into pretrained image LDMs and trained only those layers, freezing the spatial weights. Stable Video Diffusion (Blattmann, Dockhorn, et al. 2023) kept that recipe and moved the variable to the data, showing the importance of curation for video quality. AnimateDiff (Guo et al. 2024) pushed the separation furthest and showed motion modules can animate any personalized text-to-image model without retraining, which is the claim that motion and appearance are learned in disjoint parameter sets.

Flow matching (Lipman et al. 2023) provided an alternative training objective for continuous normalizing flows. Instead of learning a score function, flow matching directly regresses a velocity field that transports noise to data; the score $\nabla_x \log p_t(x)$ is replaced by a learned velocity $v_\theta(x, t)$ generating a probability path $p_t$ interpolating between noise $p_0 = \mathcal{N}(0, I)$ and data $p_1 = p_{\text{data}}$. The conditional flow matching (CFM) objective is: $$\begin{equation} \mathcal{L}{\text{CFM}} = \mathbb{E}{t, x_1, x_t} \|v_\theta(x_t, t) - u_t(x_t | x_1)\|^2 \end{equation}$$ where $u_t(x_t | x_1)$ is the target velocity for the conditional path $x_t = (1-t)x_0 + t x_1$ (linear interpolation with OT paths). Compared to score matching, CFM yields straighter trajectories that require fewer integration steps at inference (typically 20-50 NFEs vs. 100-1000 for standard diffusion). The OT variant uses $x_t = (1-(1-\sigma_{\min})t)x_0 + tx_1$, producing nearly straight paths that can be sampled with Euler integration in as few as 5 steps. Consistency models (Song et al. 2023) learn to directly map any noisy $x_t$ to $x_0$ via a self-consistency constraint $f_\theta(x_t, t) = f_\theta(x_{t'}, t')$ for all $t, t'$ on the same trajectory, enabling one-step generation.

Controllable generation became practical with ControlNet (Zhang et al. 2023), which adds spatial conditioning (edges, depth, pose) to pretrained models via trainable copies of encoder blocks connected with zero convolutions. InstructPix2Pix (Brooks et al. 2023) enabled instruction-based editing by training on synthetic edit pairs. Prompt-to-Prompt (Hertz et al. 2022) showed that manipulating cross-attention maps enables localized editing without retraining.

Sora (SORA 2024) carries the same separation into video at scale, keeping the DiT backbone and replacing image patches with spacetime patches. It generates minute-long clips with 3D consistency and object permanence that nobody wrote a prior for, which is the observation behind the world simulator phrasing. The phrasing is a claim about emergent behaviour under scale, not a demonstration that physics was learned, and OpenAI's own report lists the counterexamples: glass that shatters wrongly, a bitten cookie with no bite mark, hands with the wrong finger count. Section Sora takes those failures up in detail.

References