Autoregressive Image Generation: PixelRNN, PixelCNN, and VQ-VAE

Autoregressive visual generation evolved through four overlapping waves: (i) pixel-level models that factorise the image distribution as a product of conditionals over raw RGB values (PixelRNN (Oord, Kalchbrenner, and Kavukcuoglu 2016), PixelCNN (Oord, Kalchbrenner, Vinyals, et al. 2016), PixelCNN++ (Salimans et al. 2017)); (ii) discrete tokenisation that compresses pixels into learned codebooks before autoregression (VQ-VAE (Oord et al. 2017), VQ-VAE-2 (Razavi et al. 2019), RQ-VAE (Lee et al. 2022)); (iii) transformer-based generation that ports GPT-style decoders to image and text-to-image (iGPT (Chen et al. 2020), DALL-E (Ramesh et al. 2021), Parti (Yu et al. 2022), LlamaGen (Sun et al. 2024)), inheriting the encoder-decoder sequence-modelling architecture from neural machine translation (Sutskever's sequence-to-sequence formulation (Sutskever et al. 2014) and Bahdanau's content-based attention (Bahdanau et al. 2015)) where the conditional-token-by-token factorisation was first stress-tested at scale; and (iv) alternative orderings and parallel decoding that break the strict raster-scan dependency (MaskGIT (Chang et al. 2022), MAGE (Li et al. 2023), Muse (Chang et al. 2023), ARDM (Hoogeboom et al. 2022), VAR (Tian et al. 2024), Infinity (Han et al. 2025), InfinityStar (Liu et al. 2025), VARGPT (Zhuang et al. 2025)). Video extends these designs along the temporal axis (VideoGPT (Yan et al. 2021), TATS (Ge et al. 2022), Phenaki (Villegas et al. 2023), MCVD (Voleti et al. 2022)) and scene-conditioned generation injects layout priors (Make-A-Scene (Gafni et al. 2022), Semantic-Diffusion (Wang et al. 2023)). Theoretical scaffolding around scale and substrate is provided by autoregressive scaling laws (Henighan et al. 2020), ViT scaling (Zhai et al. 2022), and hardware-lottery analysis (Hooker 2020).

Current results on standard image-generation benchmarks show AR models reaching parity with and in places exceeding diffusion. The leaderboard below collates the headline numbers (ImageNet-256 FID for class-conditional, zero-shot MS-COCO FID for text-to-image where applicable, tokenizer column reports the discrete code source, latency is per-image at the listed resolution, and the per-row citation names the method's source paper).

Autoregressive image-generation leaderboard, assembled across papers rather than from any single head-to-head study. ImageNet-256 FID is class-conditional, COCO-FID is zero-shot at 256 $\times$ 256, and latency is wall-clock seconds per image at the listed deployment resolution, measured on whatever hardware each paper used and therefore not portable between rows. The per-row citation names the method's source paper, not a common measurement protocol. Tokenizer column lists the discrete code source. Empty cells denote not-reported or not-applicable. Models tagged "masked" use parallel iterative decoding, "next-scale" use coarse-to-fine scale autoregression, and "raster-AR" use sequential next-token. Two cells should be read with caution and are discussed in the note below the table: VQ-VAE-2's parameter count and LlamaGen-3B's COCO-FID. The "first AR model to beat diffusion on ImageNet" milestone is VAR-d30 at 2.0B parameters, reaching FID 1.92 by sampling and 1.73 with rejection sampling, against DiT-XL/2's 2.27 at 675M (Peebles and Xie 2023). The comparison is not parameter-matched, so read it as "a 3 $\times$ larger AR model finally cleared the diffusion baseline", not as an efficiency result.
Method	Year	Params	Tokenizer	Decoding	ImageNet-FID	COCO-FID	Latency
PixelRNN (Oord, Kalchbrenner, and Kavukcuoglu 2016)	2016	~50M	Raw 8-bit pixel	raster-AR	3.00 bpd		very slow
PixelCNN (Oord, Kalchbrenner, Vinyals, et al. 2016)	2016	~80M	Raw 8-bit pixel	raster-AR	3.03 bpd		slow
PixelCNN++ (Salimans et al. 2017)	2017	~95M	Raw + logistic mix	raster-AR	2.92 bpd		slow
VQ-VAE-2 (Razavi et al. 2019)	2019	~13B*	Hierarchical VQ	raster-AR	31.11		minutes
iGPT-L (Chen et al. 2020)	2020	1.4B	k-means 9-bit pixel	raster-AR	36.4		minutes
DALL-E (Ramesh et al. 2021)	2021	12B	dVAE 8192	raster-AR		~17	~30s/64 cands
MaskGIT (Chang et al. 2022)	2022	227M	VQ-GAN 1024	masked	6.18		~0.5s
Make-A-Scene (Gafni et al. 2022)	2022	4B	Triple-VQ	raster-AR		11.84	~10s
Parti-20B (Yu et al. 2022)	2022	20B	ViT-VQGAN 8192	raster-AR		7.23	~15s
MAGE-L (Li et al. 2023)	2023	300M	VQ-GAN 1024	masked	6.93		~0.5s
Muse (Chang et al. 2023)	2023	3B+3B	ViT-VQGAN 8192	masked+SR		7.88	1.3s
LlamaGen-3B (Sun et al. 2024)	2024	3.1B	VQ-GAN 16384	raster-AR	2.18	7.2	~2s w/ vLLM
VAR-d30 (Tian et al. 2024)	2024	2.0B	Multi-scale RQ-VAE	next-scale	1.73		~0.3s
Infinity-2B (Han et al. 2025)	2025	2B	LFQ 32-bit (no book)	next-scale		0.69 GE	0.8s @ 1024
InfinityStar (Liu et al. 2025)	2025	8B	3D LFQ	next-scale		83.74 VB	40s/5s 720p
VARGPT (Zhuang et al. 2025)	2025	7B	VAR-style + CLIP	next-scale		14.8	~0.3s+text

(*) The VQ-VAE-2 parameter cell is not a measured figure. The paper reports layer counts and hidden widths for its two PixelCNN-style priors but no aggregate parameter count, so the ~13B entry is an inference someone made downstream and should not be quoted as the paper's number. The LlamaGen-3B COCO-FID cell carries the same warning for a different reason: LlamaGen's text-conditional model is the 775M variant, the 3.1B model is class-conditional only, and the paper evaluates the text model qualitatively without reporting a zero-shot MS-COCO FID. Read that cell as unsourced rather than as a LlamaGen result. "GE" denotes GenEval composite score (text-to-image alignment, higher is better, 0-1) and "VB" denotes VBench composite (text-to-video, higher is better, 0-100). Neither is comparable to FID. Pixel-level rows report bits-per-dimension instead of FID because Inception-v3 features are uninformative at 32 $\times$ 32 resolution.

The table is restricted to image models reporting FID or a text-to-image alignment composite, which leaves out several papers covered later in the chapter because they report on incomparable benchmarks: VideoGPT, TATS, Phenaki and MCVD report FVD on BAIR, UCF-101, Kinetics-600 and Cityscapes, ARDM reports bits-per-character on Text8 and bits-per-dimension on CIFAR-10, and Semantic-Diffusion reports Cityscapes and ADE20K FID under a segmentation-conditioned protocol. Those numbers are given in the sections that discuss each paper. Folding them into one column would imply a comparison the benchmarks do not support.

Three masking regimes that govern autoregressive visual generation. Left: a raster-scan causal mask on a $4 \times 4$ token grid, where the model predicting position 9 (red) sees only the 9 preceding tokens (blue) and is blind to the future (gray). Middle: VAR's block-causal scale mask flips this. Tokens within a scale attend bidirectionally to each other, while only earlier scales are visible across the block boundary, which is what enables parallel decoding inside each scale step. Right: MaskGIT's cosine schedule $\gamma(t) = \cos(\pi t / 2T)$ replaces token-level causality entirely with iterative parallel decoding. At each of $T = 8$ steps the model unmasks a chunk of high-confidence tokens and re-masks the rest, and because the cosine falls slowly at first the early steps commit few tokens and the late steps commit many. The three diagrams capture the dominant axis of design variation in modern AR-image work, which is not what the network does but what its attention is permitted to see, and at what granularity.

References

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. "Neural Machine Translation by Jointly Learning to Align and Translate." International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1409.0473.

Chang, Huiwen, Han Zhang, Jarred Barber, et al. 2023. "Muse: Text-To-Image Generation via Masked Generative Transformers." International Conference on Machine Learning (ICML), January. https://doi.org/10.48550/arXiv.2301.00704.

Chang, Huiwen, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. 2022. "MaskGIT: Masked Generative Image Transformer." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), February. https://doi.org/10.48550/arXiv.2202.04200.

Chen, Mark, Alec Radford, Rewon Child, et al. 2020. "Generative Pretraining From Pixels." Proceedings of the 37th International Conference on Machine Learning, November, 1691-703. https://proceedings.mlr.press/v119/chen20s.html.

Gafni, Oran, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. 2022. "Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors." European Conference on Computer Vision (ECCV), March. https://doi.org/10.48550/arXiv.2203.13131.

Ge, Songwei, Thomas Hayes, Harry Yang, et al. 2022. "Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer." European Conference on Computer Vision (ECCV), September. https://doi.org/10.48550/arXiv.2204.03638.

Han, Jian, Jinlai Liu, Yi Jiang, et al. 2025. Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis. arXiv. https://doi.org/10.48550/arXiv.2412.04431.

Henighan, Tom, Jared Kaplan, Mor Katz, et al. 2020. Scaling Laws for Autoregressive Generative Modeling. arXiv. https://doi.org/10.48550/arXiv.2010.14701.

Hoogeboom, Emiel, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. 2022. "Autoregressive Diffusion Models." International Conference on Learning Representations, February. https://doi.org/10.48550/arXiv.2110.02037.

Hooker, Sara. 2020. The Hardware Lottery. arXiv. https://doi.org/10.48550/arXiv.2009.06489.

Lee, Doyup, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. 2022. "Autoregressive Image Generation Using Residual Quantization." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), March. https://doi.org/10.48550/arXiv.2203.01941.

Li, Tianhong, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. 2023. "MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June. https://doi.org/10.48550/arXiv.2211.09117.

Liu, Jinlai, Jian Han, Bin Yan, et al. 2025. "InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation." Advances in Neural Information Processing Systems, November. https://doi.org/10.48550/arXiv.2511.04675.

Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. "Pixel Recurrent Neural Networks." International Conference on Machine Learning (ICML), August. https://doi.org/10.48550/arXiv.1601.06759.

Oord, Aaron van den, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. 2016. "Conditional Image Generation with PixelCNN Decoders." Advances in Neural Information Processing Systems, June. https://doi.org/10.48550/arXiv.1606.05328.

Oord, Aaron van den, Oriol Vinyals, and Koray Kavukcuoglu. 2017. "Neural Discrete Representation Learning." Advances in Neural Information Processing Systems, May. https://doi.org/10.48550/arXiv.1711.00937.

Peebles, William, and Saining Xie. 2023. Scalable Diffusion Models with Transformers. arXiv. https://doi.org/10.48550/arXiv.2212.09748.

Ramesh, Aditya, Mikhail Pavlov, Gabriel Goh, et al. 2021. "Zero-Shot Text-to-Image Generation." International Conference on Machine Learning (ICML), February. https://doi.org/10.48550/arXiv.2102.12092.

Razavi, Ali, Aaron van den Oord, and Oriol Vinyals. 2019. "Generating Diverse High-Fidelity Images with VQ-VAE-2." Advances in Neural Information Processing Systems, June. https://doi.org/10.48550/arXiv.1906.00446.

Salimans, Tim, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. 2017. "PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications." International Conference on Learning Representations, January. https://doi.org/10.48550/arXiv.1701.05517.

Sun, Peize, Yi Jiang, Shoufa Chen, et al. 2024. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv. https://doi.org/10.48550/arXiv.2406.06525.

Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. 2014. "Sequence to Sequence Learning with Neural Networks." Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/1409.3215.

Tian, Keyu, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. 2024. "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction." Advances in Neural Information Processing Systems, June. https://doi.org/10.48550/arXiv.2404.02905.

Villegas, Ruben, Mohammad Babaeizadeh, Pieter-Jan Kindermans, et al. 2023. "Phenaki: Variable Length Video Generation From Open Domain Textual Description." International Conference on Learning Representations, October. https://doi.org/10.48550/arXiv.2210.02399.

Voleti, Vikram, Alexia Jolicoeur-Martineau, and Christopher Pal. 2022. "MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation." Advances in Neural Information Processing Systems, October. https://doi.org/10.48550/arXiv.2205.09853.

Wang, Weilun, Jianmin Bao, Wengang Zhou, et al. 2023. Semantic Image Synthesis via Diffusion Models. arXiv. https://doi.org/10.48550/arXiv.2207.00050.

Yan, Wilson, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. 2021. VideoGPT: Video Generation Using VQ-VAE and Transformers. arXiv. https://doi.org/10.48550/arXiv.2104.10157.

Yu, Jiahui, Yuanzhong Xu, Jing Yu Koh, et al. 2022. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. arXiv. https://doi.org/10.48550/arXiv.2206.10789.

Zhai, Xiaohua, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. "Scaling Vision Transformers." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June. https://doi.org/10.48550/arXiv.2106.04560.

Zhuang, Xianwei, Yuxin Xie, Yufan Deng, et al. 2025. VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model. arXiv. https://doi.org/10.48550/arXiv.2501.12327.