Vision Transformers: ViT, Self-Attention, and the CNN Comparison

Vision Transformers (ViT) replace the convolutional backbone with a pure attention architecture. The foundational ViT paper (Dosovitskiy et al. 2021) showed that a standard Transformer encoder, applied directly to sequences of image patches, achieves strong image classification when pretrained on sufficient data (ImageNet-21K (Deng et al. 2009) or JFT-300M). ViT-Huge achieved 88.55% top-1 accuracy on ImageNet, matching the best reported CNNs of the time (BiT-L, Noisy Student) at lower pretraining compute. The paper's reading is that convolutional inductive biases are not strictly necessary, because a transformer can learn them from data.

Data efficiency became the focus of subsequent work. DeiT (Touvron et al. 2021) showed ViT can be trained on ImageNet-1K alone using strong regularization, achieving 81.8% top-1 accuracy without external data (83.4% with distillation from a RegNetY-16GF teacher). DeiT introduced knowledge distillation via a distillation token, showing that ViT benefits from CNN-style inductive biases transferred through distillation. DeiT III (Touvron et al. 2022) pushed this further with improved training recipes, achieving 85.2% with ViT-H/14.

Hierarchical designs addressed the computational limitations of global attention. Swin Transformer (Liu et al. 2021) introduced shifted window attention, computing attention within local windows while enabling cross-window connections through shifting. This achieves linear complexity and provides multi-scale feature maps compatible with dense prediction. Swin achieved 87.3% on ImageNet and, at publication, the best reported COCO detection (58.7 box AP) and ADE20K segmentation (53.5 mIoU) figures. Swin Transformer V2 (Ze Liu et al. 2022) scaled to 3B parameters with techniques for training stability (residual post-normalization, scaled cosine attention).

Alternative hierarchical approaches emerged. Pyramid Vision Transformer (PVT) (Wang et al. 2021) uses spatial reduction attention to handle high-resolution feature maps. Twins (Chu et al. 2021) combines locally-grouped attention with global sub-sampled attention. Focal Transformer (Yang et al. 2021) introduces focal attention attending both locally and globally with varying granularity. Nested Hierarchical Transformer (NesT) (Zhang et al. 2021) aggregates tokens in a nested manner to create hierarchical representations.

Hybrid architectures combine CNN and Transformer strengths. Convolutional vision Transformer (CvT) (Wu et al. 2021) introduces convolutions into attention projection. CoAtNet (Dai et al. 2021) systematically combines depthwise convolution and attention, achieving 90.88% ImageNet accuracy with JFT-3B pretraining. MaxViT (Tu et al. 2022) interleaves block and grid attention patterns with convolutions for efficient multi-scale attention. Mobile-Former (Y. Chen et al. 2022) uses parallel mobile and former branches with bidirectional connections. MobileViT (Mehta and Rastegari 2022) unfolds features for transformer processing while maintaining mobile efficiency.

Self-supervised learning proved particularly effective for ViT. DINO (Caron et al. 2021) showed self-supervised ViT features contain semantic segmentation information not present in supervised models, achieving 78.3% k-NN accuracy on ImageNet. A follow-up diagnostic, Vision Transformers Need Registers (Darcet et al. 2024), identified a structural artifact in large pre-trained ViTs: a small fraction of patch tokens develop anomalously high feature norms and attract disproportionate attention weight, producing irregular blobs in attention maps. These high-norm tokens appear preferentially at low-information background patches, not at semantically salient locations. The interpretation is that the model repurposes uninformative patch tokens as global scratch space to store and transmit information that does not fit naturally into the attention pattern of any single semantic patch.

The fix is minimal: append $r = 4$ additional learnable [REG] tokens to the input sequence (alongside the [CLS] token) and include them throughout all layers. These tokens have no spatial correspondence and no loss target; they serve purely as internal workspace. With registers present, the high-norm artifact disappears from patch tokens, attention maps become spatially smooth, and dense prediction improves. The intervention requires no change to the training recipe, the loss function, or the backbone architecture. The paper reports the best figures at publication for self-supervised models on dense visual prediction tasks and improvements in object discovery quality with larger models. The caveat is that register count $r = 4$ is a hyperparameter tuned on DINOv2-scale models; smaller ViTs trained on less data show less severe artifact behaviour and smaller gains from registers. MAE (He et al. 2021) showed that masking 75% of patches and reconstructing provides strong pretraining, with ViT-Huge achieving 87.8% accuracy using only ImageNet-1K. BEiT (Bao et al. 2022) predicted discrete visual tokens from masked patches. SimMIM (Xie et al. 2022) showed simple random masking with direct pixel prediction works well.

Efficiency improvements targeted inference cost. DynamicViT (Rao et al. 2021) dynamically prunes uninformative tokens based on learned importance scores, reducing FLOPs by 31-37% with <0.5% accuracy drop. EfficientFormer (Li et al. 2022) designed latency-driven architectures achieving mobile speed. Token-to-Token ViT (T2T-ViT) (Yuan et al. 2021) progressively tokenizes images to capture local structure.

Understanding ViT behavior revealed important properties. Empirical studies (X. Chen et al. 2021) showed ViT optimizes differently from CNNs, with sensitivity to optimizer, weight decay, and learning rate schedules. "How to train your ViT" (Steiner et al. 2022) provided a systematic study of data, augmentation and regularization for ViT training. Sharpness-aware minimization (SAM) (Foret et al. 2021) improved ViT generalization by optimizing for flatter minima. Adversarial robustness analysis (Naseer et al. 2021) (Shao et al. 2022) found ViT more robust than CNNs due to attention's global context. The data-scale crossover between the two families is measured in the original ViT paper (Dosovitskiy et al. 2021); the narrower claim of (X. Chen et al. 2022) is that a sharpness-aware optimizer lets ViTs beat ResNets on ImageNet-1K from scratch, with neither large-scale pretraining nor strong augmentation.

Cross-attention between scales improved multi-scale reasoning. CrossViT (C.-F. Chen et al. 2021) uses dual-branch architecture with different patch sizes, fusing via cross-attention. This achieves efficient multi-scale processing while maintaining reasonable computational cost.

One axis organises the whole list above. ViT made a single design decision, it deleted the convolutional prior, and every paper that follows pays some of that prior back. DeiT pays it back through the teacher and the augmentation stack. Swin, PVT, Twins and Focal re-impose locality inside the attention pattern itself. CvT, CoAtNet, MaxViT and MobileViT put the convolution back into the block. MAE, BEiT, SimMIM and DINO move the prior into the pretext task, so unlabeled data supplies it instead of the architecture. ConvNeXt runs the argument backwards, keeping the convolution and importing only the recipe. Sorted by where the bias is stored rather than by accuracy, the families read as follows.

The field converged on: (1) hierarchical designs for dense prediction (Swin), (2) strong regularization for data-efficient training (DeiT), (3) self-supervised pretraining for label efficiency (MAE, DINO), (4) hybrid CNN-Transformer architectures for mobile deployment, and (5) training recipes matched to the architectural choice, which move accuracy as much as the choice itself does.

Related-work leaderboard (representative numbers, ImageNet-1K top-1 unless noted):

References