Detection and Segmentation Transformers: DETR and Its Descendants

Detection before DETR relied on two-stage detectors like Faster R-CNN or single-stage detectors like YOLO and RetinaNet. These require hand-designed components, anchor boxes for proposal generation, non-maximum suppression for duplicate removal, and a label-assignment rule deciding which prediction is responsible for which object. Each of those three is a spatial prior, a statement made in advance about where objects sit and which prediction owns which one. DETR (Carion et al. 2020) deleted all three at once by framing detection as set prediction, a transformer encoder-decoder emitting a fixed set of $N$ predictions from learned object queries, trained by one-to-one bipartite matching against the ground truth.

The deletion is what makes the chapter, and the bill for the deletion is what organizes the rest of it. DETR trained for 500 COCO epochs where Faster R-CNN needs 36, and it lost 6.1 AP on small objects while gaining 7.7 on large ones (Carion et al. 2020). Both facts have one cause. A query carrying no spatial prior has to discover from scratch where in the image to attend, and a single stride-32 feature map cannot resolve a small object at any amount of training. So every method after DETR in this chapter puts a prior back, and the binding constraint is that none of them may put NMS back. A prior returned as a reference point, an anchor coordinate, a noised ground-truth box or an auxiliary dense head is differentiable and stays inside training, and is discarded at inference. A prior returned as greedy suppression is neither, and would surrender the one property the 500 epochs bought. Table tbl:detr-prior-ledger is that ledger, and it is the shortest route through the sections that follow.

Segmentation repeats the shape one level up. MaskFormer (Cheng et al. 2021) drops the per-pixel class prior and predicts a set of masks matched exactly the way DETR matches boxes, which is what lets semantic, instance and panoptic segmentation share one framework rather than three. Mask2Former (Cheng et al. 2022) then puts a prior of its own back, each query attends only inside the mask it predicted at the previous layer, and that is the mask-space analogue of a reference point. Underneath both sits the backbone's own prior, the hierarchical stride-4-to-32 pyramid that the Swin Transformer (Liu et al. 2021) supplies with shifted window attention and Swin V2 (Z. Liu et al. 2022) carries to 3B parameters and 1536-pixel inputs. A plain ViT gives one scale and a DETR decoder has nothing to address it with, which is why the backbone chapter and this one are not separable.

The prior ledger. *Puts back* names the spatial prior DETR had deleted and this method restores. *Differentiable form* names the construction that makes it trainable rather than a post-processing heuristic, which is the condition for keeping the NMS-free property. *Bought* gives what the restoration is worth, with the schedule and backbone attached, because these methods are only comparable at matched epochs. Figures are COCO val2017 with ResNet-50 unless the cell states otherwise. The per-row citation names the method's source paper, not the source of the specific figure in the *Bought* cell, and two of those figures should be treated as unverified: the DN-DETR row's 44.4 AP does not match the 44.1 AP reported in that paper's own 50-epoch ResNet-50 table, and the Group DETR row's +1.5 AP is quoted without its baseline configuration. Read the table downward and the chapter is one argument rather than a list of ten papers.
Method	Puts back	Differentiable form	Bought
Deformable DETR (Zhu et al. 2021)	reference point, multi-scale	$K{=}4$ learned offsets around $p_q$ , levels C3-C6	50 epochs, 43.8 AP, AP $_S$ 20.5 $\to$ 26.4
Conditional DETR (Meng et al. 2021)	spatial half of cross-attention	$q_s = f_s(c_q) \cdot p_s$ , 300 learned reference points	50 epochs, 40.9 AP where DETR needs 500 for 42.0
DAB-DETR (S. Liu et al. 2022)	the anchor box itself	sinusoidal PE of $(x,y,w,h)$ , updated per decoder layer	50 epochs, 42.2 AP, and queries you can plot
DN-DETR (Li et al. 2022)	the assignment	noised GT boxes with known targets, matcher bypassed	50 epochs, 44.4 AP, ~4 $\times$ less match switching
DINO (Zhang et al. 2023)	negatives, encoder-picked positions	contrastive denoising, mixed query selection	12 epochs, 49.4 AP; 63.3 test-dev at Swin-L
Group DETR (Chen et al. 2023)	positive-sample density	$G{=}11$ replicated query groups, masked apart	+1.5 AP, training-only, $G$ is dataset-dependent
H-DETR (Jia et al. 2023)	positive-sample density	a second one-to-many head on shared decoder features	~~1.7 AP over Deformable-DETR+~~ at 12 epochs
Co-DETR (Zong et al. 2023)	dense assignment on the encoder	ATSS and RPN auxiliary heads, discarded at inference	66.0 AP test-dev, ViT-L with Objects365
Mask2Former (Cheng et al. 2022)	the region worth attending to	previous layer's mask as a $-\infty$ attention bias	+4.8 PQ over unmasked cross-attention
RT-DETR (Zhao et al. 2024)	query quality	uncertainty-minimal selection, decoupled hybrid encoder	53.1 AP at 108 FPS on T4, NMS-free

References

Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. "End-to-End Object Detection with Transformers." European Conference on Computer Vision (ECCV), May. https://doi.org/10.48550/arXiv.2005.12872.

Chen, Qiang, Xiaokang Chen, Jian Wang, et al. 2023. "Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment." IEEE/CVF International Conference on Computer Vision (ICCV), August. https://doi.org/10.48550/arXiv.2207.13085.

Cheng, Bowen, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. "Masked-Attention Mask Transformer for Universal Image Segmentation." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June. https://doi.org/10.48550/arXiv.2112.01527.

Cheng, Bowen, Alexander G. Schwing, and Alexander Kirillov. 2021. "Per-Pixel Classification Is Not All You Need for Semantic Segmentation." Advances in Neural Information Processing Systems (NeurIPS), October. https://doi.org/10.48550/arXiv.2107.06278.

Jia, Ding, Yuhui Yuan, Haodi He, et al. 2023. "DETRs with Hybrid Matching." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), May. https://doi.org/10.48550/arXiv.2207.13080.

Li, Feng, Hao Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, and Lei Zhang. 2022. "DN-DETR: Accelerate DETR Training by Introducing Query DeNoising." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), December. https://doi.org/10.48550/arXiv.2203.01305.

Liu, Shilong, Feng Li, Hao Zhang, et al. 2022. "DAB-DETR: Dynamic Anchor Boxes Are Better Queries for DETR." International Conference on Learning Representations (ICLR), March. https://doi.org/10.48550/arXiv.2201.12329.

Liu, Ze, Han Hu, Yutong Lin, et al. 2022. Swin Transformer V2: Scaling Up Capacity and Resolution. arXiv. https://doi.org/10.48550/arXiv.2111.09883.

Liu, Ze, Yutong Lin, Yue Cao, et al. 2021. "Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows." IEEE/CVF International Conference on Computer Vision (ICCV), August. https://doi.org/10.48550/arXiv.2103.14030.

Meng, Depu, Xiaokang Chen, Zejia Fan, et al. 2021. "Conditional DETR for Fast Training Convergence." IEEE/CVF International Conference on Computer Vision (ICCV), September. https://doi.org/10.48550/arXiv.2108.06152.

Zhang, Hao, Feng Li, Shilong Liu, et al. 2023. "DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection." International Conference on Learning Representations (ICLR), July. https://doi.org/10.48550/arXiv.2203.03605.

Zhao, Yian, Wenyu Lv, Shangliang Xu, et al. 2024. DETRs Beat YOLOs on Real-Time Object Detection. arXiv. https://doi.org/10.48550/arXiv.2304.08069.

Zhu, Xizhou, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021. "Deformable DETR: Deformable Transformers for End-to-End Object Detection." International Conference on Learning Representations (ICLR), March. https://doi.org/10.48550/arXiv.2010.04159.

Zong, Zhuofan, Guanglu Song, and Yu Liu. 2023. "DETRs with Collaborative Hybrid Assignments Training." IEEE/CVF International Conference on Computer Vision (ICCV), August. https://doi.org/10.48550/arXiv.2211.12860.