# Discrete Diffusion Models for Language Modelling

- Channel: [Anastasia Borovykh](https://www.youtube.com/channel/UC2BjavqyTVQiJvH9udm2SSA)
- Published: 2026-04-22
- Source video: [Watch on YouTube](https://youtube.com/watch?v=rtHgD0CYW78)
- Read on ReadTube: https://www.read.tube/p/videos/rtHgD0CYW78
- Language: Original (source language)

## Summary

**离散扩散模型语言建模**

Discrete diffusion models provide an alternative to auto-regressive language modeling by iteratively denoising masked tokens rather than predicting them one by one. This approach potentially offers faster inference speeds and greater control over generation, although currently it faces challenges like high training variance and the need for complex, often non-causal, generation strategies to match the performance of traditional auto-regressive models.

Discrete diffusion models propose an alternative to auto-regressive (AR) language modeling by gradually denoising masked tokens rather than predicting them sequentially. This approach targets key limitations of AR models, specifically their rigid left-to-right generation, limited self-correction, and potential architectural bottlenecks.

- Training involves corrupting sequences via masking and teaching the model to reconstruct masked tokens in arbitrary orders.
- Inference gains speed by denoising multiple tokens in parallel, bypassing the token-by-token bottleneck.
- Performance currently approaches AR benchmarks but often requires complex modifications to maintain coherence.
- Recent research reveals that high-performing diffusion models frequently rely on pre-trained AR weights or reintroduce causal attention to stabilize generations.

Despite the promise of increased flexibility and speed, current literature suggests that pure discrete diffusion models still struggle to surpass the reliability of standard auto-regressive architectures, leading to hybrid approaches that merge concepts from both fields.