← Back to blogs

CNN Object Detection: From LeNet and AlexNet to Modern Detector Backbones

Convolutional backbones and CNN-based object detection

Object detection localizes and classifies multiple objects in images via bounding boxes. Across roughly six years (2014-2020) the literature settled into five overlapping families: two-stage refinement, single-shot dense prediction, focal-loss balanced one-stage, anchor-free per-pixel and keypoint heads, and set-prediction transformers. Each occupies a different point on the accuracy/speed/simplicity frontier, and each family also addressed a characteristic set of well-known problems (scale, imbalance, NMS heuristics, anchor design, slow convergence) walked through paragraph-by-paragraph after the SoTA leaderboard below.

The historical context matters because the same problems recur under new names. Pre-deep-learning detection plateaued near 33% mAP on PASCAL VOC under HOG-feature pipelines1 (Dalal and Triggs 2005) and the deformable part model (Felzenszwalb et al. 2008) (a sliding-window root + parts mixture trained with latent-SVM that defined the pre-CNN state of the art for almost a decade); R-CNN (Girshick et al. 2014) (2014) lifted it to 53.3% by reusing ImageNet (Deng et al. 2009) pretrained CNN features on selective-search proposals, but at ~49 seconds per image. Fast R-CNN (Girshick 2015) (2015) shared backbone computation across proposals for a 213×\times speedup; Faster R-CNN (Ren et al. 2015) (2015) replaced the external proposal stage with a learned Region Proposal Network. Single-shot detectors emerged in parallel: YOLO (Redmon et al. 2016) (2016) reframed detection as direct regression on a 7×\times7 grid at 45 FPS, SSD (Liu et al. 2016) (2016) added multi-scale feature maps. RetinaNet (2017) closed the one-stage/two-stage accuracy gap with focal loss (Lin, Goyal, et al. 2017), while FPN (Lin, Dollár, et al. 2017) became the universal neck. Anchor-free methods (CornerNet (Law and Deng 2018), FCOS (Tian et al. 2019), CenterNet (Zhou et al. 2019)) eliminated the anchor-design hyperparameters in 2019; DETR (Carion et al. 2020) removed NMS via Hungarian matching in 2020 (covered in the transformer chapter).

The five families differ on which problems they accept and which they engineer away. Two-stage detectors handle the foreground/background imbalance via balanced sampling at the RoI head, but inherit a complex multi-stage pipeline and slow inference. One-stage detectors face the imbalance directly through hard mining (SSD), focal loss (RetinaNet), or implicit balanced anchors (YOLO grid). Anchor-free heads sidestep the entire scale-and-aspect anchor-tuning enterprise but still need a label-assignment heuristic (smallest-area box or centerness). Set-prediction transformers bypass NMS at the cost of 500-epoch training. No family dominates: the COCO leaderboard top is a moving boundary between Cascade-style two-stage refinement, EfficientDet-style compound-scaled one-stage, and DINO-DETR-style transformers, and the production-grade default depends on whether latency, throughput, or peak accuracy is the primary constraint23.

CNN object-detection SoTA leaderboard. Method links to the chapter section, with citation attached. Year is the conference/arXiv year. Backbone is the canonical reported configuration; Head/Loss summarises the detection head and the canonical regression/classification loss with key hyperparameters in parentheses. Data names the training set (VOC07+12 trainval / COCO trainval (Lin et al. 2014) / OpenImages). VOC = PASCAL VOC 2007 test mAP at IoU=0.5. COCO columns: AP is the mean over IoU thresholds 0.5:0.05:0.95 on val2017 single-scale; AP50 at IoU=0.5; APS, APM, APL are the standard small (<322 px) / medium (32-962 px) / large (>962 px) splits. FPS is the original paper's reported speed on its own GPU (K40, Titan X, V100, A100); cross-family comparisons unreliable (see fn ref in prose). Dashes mark unreported settings. RPN+RoI(softmax) denotes Faster-style two-stage classification; Focal denotes RetinaNet-style focal cross-entropy; Hungarian denotes DETR-style set matching.
VOC COCO Speed
Method Year Backbone Head/Loss Data mAP AP AP50 APS APM APL FPS
R-CNN (Girshick et al. 2014) 2014 AlexNet SVM+ridge-reg VOC07+12 53.3 - - - - - 0.05
Fast R-CNN (Girshick 2015) 2015 VGG-16 RoI(softmax)+sm-L1 VOC07+12 70.0 19.7 35.9 - - - 0.5
Faster R-CNN (Ren et al. 2015) 2015 VGG-16 RPN+RoI(softmax)+sm-L1 VOC07+12 73.2 21.9 42.7 - - - 5
YOLO v1 (Redmon et al. 2016) 2016 Darknet (custom) grid-MSE(λc\lambda_c=5,λn\lambda_n=0.5) VOC07+12 63.4 - - - - - 45
SSD-512 (Liu et al. 2016) 2016 VGG-16 hard-mining(3:1)+sm-L1 VOC07+12 76.8 28.8 48.5 10.9 31.8 43.5 22
FPN (Faster R-CNN) (Lin, Dollár, et al. 2017) 2017 ResNet-101 RPN+RoI(softmax)+sm-L1 COCO - 36.2 59.1 18.2 39.0 48.2 5
Mask R-CNN (He et al. 2017) 2017 ResNet-101 + FPN RPN+RoI(softmax)+sm-L1+mask COCO - 39.8 62.3 22.1 43.2 51.2 5
RetinaNet (Lin, Goyal, et al. 2017) 2017 ResNet-101 + FPN Focal(γ\gamma=2,α\alpha=0.25)+sm-L1 COCO - 39.1 59.1 21.8 42.7 50.2 5
YOLO v3 (Redmon and Farhadi 2018) 2018 Darknet-53 BCE(per-anchor obj)+sm-L1 COCO - 33.0 57.9 18.3 35.4 41.9 35
Cascade R-CNN (Cai and Vasconcelos 2018) 2018 ResNet-101 + FPN 3-head cascade(0.5/0.6/0.7) COCO - 42.8 62.1 23.7 45.5 55.2 7
PANet (Liu et al. 2018) 2018 ResNeXt-101 PAFPN+RoI(softmax)+sm-L1 COCO - 47.4 67.2 27.2 51.0 60.0 4
CornerNet (Law and Deng 2018) 2019 Hourglass-104 Focal-heatmap+pull/push COCO - 42.1 57.8 20.8 44.8 56.7 4
FCOS (Tian et al. 2019) 2019 ResNet-101 + FPN center-ness+GIoU+focal-cls COCO - 41.5 60.7 24.4 44.8 51.6 17
CenterNet (Zhou et al. 2019) 2019 Hourglass-104 Gaussian-heatmap+L1(size) COCO - 42.1 61.1 19.9 43.0 51.4 7
EfficientDet-D7 (Tan et al. 2020) 2020 EfficientNet-B6 BiFPN+Focal+sm-L1 COCO - 53.7 72.4 35.8 57.0 66.3 6
YOLO v4 (Bochkovskiy et al. 2020) 2020 CSPDarknet-53 CIoU+BCE-obj+Mosaic-aug COCO - 43.5 65.7 26.7 46.7 53.3 65
DETR (Carion et al. 2020) 2020 ResNet-101 Hungarian+CE+L1+GIoU(5,2) COCO - 44.9 64.7 23.7 49.5 62.3 28
Deformable DETR (Zhu et al. 2021) 2021 ResNet-50 Hungarian+CE+L1+GIoU+def-att COCO - 46.2 65.2 28.8 49.2 61.7 19
DINO-DETR (Zhang et al. 2023) 2022 Swin-L Hungarian+denoising+contrast COCO+O365 - 58.5 77.1 41.5 62.7 74.0 10
Co-DETR (Zong et al. 2023) 2023 ViT-L (MAE) hybrid Hungarian+ATSS+Faster COCO+O365 - 66.0 - - - - -

The leaderboard separates six eras and a speed column whose meaning differs from the accuracy columns. Reading the table from left to right gives the family-by-family progression (R-CNN →\to one-stage →\to Focal →\to anchor-free →\to compound-scaled →\to transformer); reading top-to-bottom by Year reveals the COCO AP trajectory from 19.7 (Fast R-CNN, 2015) to 66.0 (Co-DETR, 2023), a 3.4×\times gain spread across backbone scaling4, pyramid sophistication5, and head redesign. Three diagnostics. First, APS almost always lags APM and APL by 15-25 points, the single most persistent failure mode of CNN detectors6. Second, FPS within a family is reliable but cross-family comparisons span four GPU generations7. Third, the gap between best two-stage and best one-stage inverted between 2018 and 2020, eliminating the historical "two-stage = accuracy, one-stage = speed" dichotomy8.

The two-stage R-CNN family defines the era. R-CNN ran 2000 selective-search proposals through an AlexNet/VGG forward pass each, so per-image inference at ~49s was untenable; Fast R-CNN shared a single backbone pass and replaced the per-class SVM cascade with a softmax + smooth-L1 multi-task head, while Faster R-CNN folded proposals into a learned RPN sliding 9 hand-tuned anchors per location9. Mask R-CNN extended the same scaffolding with a per-class binary mask branch and replaced quantising RoIPool with bilinear RoIAlign; the pixel-precise alignment matters far more at strict IoU thresholds10. Cascade R-CNN refines proposals through three heads at IoU thresholds 0.5/0.6/0.7, exploiting the quality-mismatch observation that a single head cannot simultaneously be accurate at all IoU regimes; this is also where label assignment becomes structurally explicit11.

The single-shot SSD/YOLO line trades the proposal stage for direct dense prediction. YOLO partitioned the image into a 7×\times7 grid where each cell predicts B=2B=2 boxes plus class probabilities at 45 FPS but only 63.4 VOC mAP, paying for speed in localization (19% of YOLO's error is localization vs Fast R-CNN's 8%); SSD recovered accuracy by attaching prediction heads to multiple feature-map resolutions (38×\times38 through 1×\times1) so that small objects map to high-resolution layers and large objects to coarse layers, a structurally different solution to scale variance12. SSD also introduced 3:1 hard negative mining as its blunt-instrument response to foreground/background imbalance13, and depends notoriously hard on data augmentation (74.3 →\to 65.5 mAP without augmentation). YOLOv3 added k-means anchor priors and FPN-style three-scale prediction; YOLOv4 split engineering into "Bag of Freebies" (Mosaic, CIoU, label smoothing) and "Bag of Specials" (Mish, DIoU-NMS, PANet neck), reaching 43.5 COCO AP at 65 FPS by combining many small recipes rather than one architectural insight.

RetinaNet + FPN closed the one-stage/two-stage accuracy gap with two orthogonal contributions. Focal loss −(1−pt)γlog⁡pt -(1-p_t)^\gamma \log p_t with γ=2,α=0.25\gamma=2,\alpha=0.25 reweights cross-entropy by prediction confidence so that the cumulative loss from ~100k easy negatives no longer drowns ~100 hard positives14, and the classification head's bias is initialized to −log⁡((1−π)/π),π=0.01 -\log((1-\pi)/\pi),\ \pi=0.01 to make the initial logits match the empirical class prior. Focal loss beat OHEM because it uses every negative with smooth down-weighting rather than a hard cutoff. The second contribution is that RetinaNet sat on top of FPN (Lin, Dollár, et al. 2017), whose top-down + lateral pathway propagates semantic strength to high-resolution feature levels at single-pass cost; APS rose from 14.1 to 19.9 (+5.8) on COCO purely from FPN, and FPN became the universal neck across Faster R-CNN, Mask R-CNN, RetinaNet, FCOS, and EfficientDet15.

CNN detection family evolution from 2015 R-CNN-era detectors to 2023 transformer-based DETR variants, tracking COCO AP vs publication year. Marker shape and colour encode family (two-stage / one-stage / anchor-free / transformer). Three structural observations match the table reading: (i) the two-stage and one-stage clusters stayed within 5 AP of each other from 2017 onward; (ii) anchor-free detectors landed on the same Pareto-front in 2019 without delivering a step change, suggesting that anchor design was a hyperparameter inconvenience rather than a fundamental capacity limit; (iii) transformer-based set-prediction (DETR →\to Deformable →\to DINO →\to Co-DETR) opens a clear gap from 2022 onward, lifting COCO AP from 47 (best CNN) to 66 (Co-DETR), a 19-AP improvement driven by larger backbones (Swin-L, ViT-L MAE) and richer training (denoising, contrastive auxiliaries).

The anchor-free turn (CornerNet, FCOS, CenterNet) made a structural rather than empirical case. FCOS predicts (l,t,r,b)(l,t,r,b) distances at every foreground pixel of the FPN levels and resolves multi-GT overlap by assigning to the smallest-area box, the only heuristic the paper retains16. The centerness branch min⁡(l,r)/max⁡(l,r)⋅min⁡(t,b)/max⁡(t,b) \sqrt{\min(l,r)/\max(l,r) \cdot \min(t,b)/\max(t,b)} suppresses low-quality peripheral predictions for +3.6 AP and is a cheap analogue of objectness. CornerNet detects top-left and bottom-right corner heatmaps with associative-embedding grouping, and CenterNet reduces this further to a single Gaussian-heatmap center per object plus a (w,h)(w,h) regression. Anchor-free heads sidestepped hand-tuned anchor scales/ratios entirely17 and proved that proper loss weighting plus per-level FPN assignment closes the gap; ATSS later showed that adaptive label assignment (mean+std of per-object IoU as the dynamic threshold) brings anchor-based and anchor-free recipes to within 0.1 AP, confirming that the anchor itself was never the load-bearing piece18.

The transformer set-prediction line, covered fully in the next chapter, starts with DETR replacing dense prediction + NMS with 100 learned object queries and Hungarian bipartite matching against the ground-truth set. NMS disappears because each query commits to one GT, eliminating the threshold-sensitive greedy suppression heuristic that limits CNN detectors in dense crowds1920. The cost was a 500-epoch convergence schedule (versus 12-24 for Faster R-CNN); Deformable DETR's sparse multi-scale attention reduced this to 50 epochs, DAB-DETR/DN-DETR added content-aware queries and denoising auxiliaries, and DINO-DETR with Swin-L plus contrastive denoising reaches 58.5 COCO AP21. Co-DETR closed the loop by re-introducing one-to-many auxiliary heads (ATSS, Faster R-CNN) alongside the one-to-one Hungarian head during training, lifting Swin-L COCO val from 58.5 to 59.5 and ViT-L COCO test-dev to 66.0 AP, the current chapter ceiling.

The closing reading is that detection has not so much "solved" any of the listed problems as redistributed them. Scale variance moved from image pyramids into FPN; foreground/background imbalance moved from RPN sampling into focal loss into per-query bipartite assignment; anchor design was first adaptive (ATSS), then dropped (FCOS), then absent (DETR); NMS was softened (Soft-NMS, DIoU-NMS) and then removed (DETR). What remains stubborn is small-object AP (still 15-25 points below large-object AP across the entire table, even at Swin-L scale), cross-paper FPS comparability (every paper reports its own GPU)22, and ImageNet/Object365 pretraining dependence23; the latter starts to dissolve only with self-supervised backbones (MAE, DINO) trained at 108^8+ image scale24.

References

Bochkovskiy, Alexey, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv. https://doi.org/10.48550/arXiv.2004.10934.
Cai, Zhaowei, and Nuno Vasconcelos. 2018. "Cascade R-CNN: Delving into High Quality Object Detection." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), December. https://doi.org/10.48550/arXiv.1712.00726.
Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. "End-to-End Object Detection with Transformers." European Conference on Computer Vision (ECCV), May. https://doi.org/10.48550/arXiv.2005.12872.
Dalal, Navneet, and Bill Triggs. 2005. "Histograms of Oriented Gradients for Human Detection." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2005.177.
Deng, Jia, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. "ImageNet: A Large-Scale Hierarchical Image Database." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2009.5206848.
Donahue, Jeff, Yangqing Jia, Oriol Vinyals, et al. 2014. "DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition." International Conference on Machine Learning (ICML). https://arxiv.org/abs/1310.1531.
Felzenszwalb, Pedro F., David McAllester, and Deva Ramanan. 2008. "A Discriminatively Trained, Multiscale, Deformable Part Model." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2008.4587597.
Girshick, Ross. 2015. "Fast R-CNN." IEEE International Conference on Computer Vision (ICCV), September. https://doi.org/10.48550/arXiv.1504.08083.
Girshick, Ross, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), October. https://doi.org/10.48550/arXiv.1311.2524.
He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. "Mask R-CNN." IEEE International Conference on Computer Vision (ICCV), January. https://doi.org/10.48550/arXiv.1703.06870.
Law, Hei, and Jia Deng. 2018. "CornerNet: Detecting Objects as Paired Keypoints." European Conference on Computer Vision (ECCV), March. https://doi.org/10.48550/arXiv.1808.01244.
Lin, Tsung-Yi, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. "Feature Pyramid Networks for Object Detection." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), April. https://doi.org/10.48550/arXiv.1612.03144.
Lin, Tsung-Yi, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. "Focal Loss for Dense Object Detection." IEEE International Conference on Computer Vision (ICCV), February. https://doi.org/10.48550/arXiv.1708.02002.
Lin, Tsung-Yi, Michael Maire, Serge Belongie, et al. 2014. "Microsoft COCO: Common Objects in Context." European Conference on Computer Vision (ECCV). https://arxiv.org/abs/1405.0312.
Liu, Shu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. 2018. "Path Aggregation Network for Instance Segmentation." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), September. https://doi.org/10.48550/arXiv.1803.01534.
Liu, Wei, Dragomir Anguelov, Dumitru Erhan, et al. 2016. SSD: Single Shot MultiBox Detector. Vol. 9905. https://doi.org/10.1007/978-3-319-46448-0_2.
Redmon, Joseph, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. "You Only Look Once: Unified, Real-Time Object Detection." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), May. https://doi.org/10.48550/arXiv.1506.02640.
Redmon, Joseph, and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. arXiv. https://doi.org/10.48550/arXiv.1804.02767.
Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun. 2015. "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks." Advances in Neural Information Processing Systems (NeurIPS), January. https://doi.org/10.48550/arXiv.1506.01497.
Tan, Mingxing, Ruoming Pang, and Quoc V. Le. 2020. "EfficientDet: Scalable and Efficient Object Detection." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), July. https://doi.org/10.48550/arXiv.1911.09070.
Tian, Zhi, Chunhua Shen, Hao Chen, and Tong He. 2019. "FCOS: Fully Convolutional One-Stage Object Detection." IEEE International Conference on Computer Vision (ICCV), August. https://doi.org/10.48550/arXiv.1904.01355.
Zhang, Hao, Feng Li, Shilong Liu, et al. 2023. "DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection." International Conference on Learning Representations (ICLR), July. https://doi.org/10.48550/arXiv.2203.03605.
Zhou, Xingyi, Dequan Wang, and Philipp Krähenbühl. 2019. Objects as Points. arXiv. https://doi.org/10.48550/arXiv.1904.07850.
Zhu, Xizhou, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021. "Deformable DETR: Deformable Transformers for End-to-End Object Detection." International Conference on Learning Representations (ICLR), March. https://doi.org/10.48550/arXiv.2010.04159.
Zong, Zhuofan, Guanglu Song, and Yu Liu. 2023. "DETRs with Collaborative Hybrid Assignments Training." IEEE/CVF International Conference on Computer Vision (ICCV), August. https://doi.org/10.48550/arXiv.2211.12860.

  1. HOG descriptor parameters (Dalal and Triggs 2005): gradient magnitudes and unsigned orientations (0-180°) are accumulated into 9-bin histograms over dense 8×\times8-pixel cells; cells are grouped into overlapping 2×\times2-cell blocks (16×\times16 pixels), each block normalized by L2-Hys (clip magnitudes at 0.2, renormalize). The concatenated block descriptors form a feature vector of length 9 bins ×\times 4 cells/block ×\times (block count per detection window); on the canonical 64×\times128-pixel pedestrian detection window this gives a 3780-dimensional descriptor. Classification uses a linear SVM trained on the INRIA pedestrian dataset. The paper reports roughly 11% miss rate at 10−110^{-1} false positives per image on INRIA test, at the time significantly better than prior Haar-wavelet-based methods. The 8-pixel cell, 16-pixel block, and 9-bin choices were each determined by systematic grid search on the INRIA training set; the paper notes that doubling cell size (to 16×\times16) drops performance substantially because fine gradient structure at pedestrian limb boundaries is lost. HOG was the dominant pedestrian and general object descriptor until R-CNN (2014) replaced it.↩︎

  2. Two-stage vs one-stage trade-off: two-stage detectors filter proposals before classification, gaining accuracy on hard examples at the cost of inference latency; one-stage detectors predict densely in a single forward pass, paying for speed with the foreground/background imbalance problem. The "RetinaNet matches Faster R-CNN" result was the first empirical point at which one-stage stopped trading accuracy for speed; the EfficientDet line later pushed the Pareto frontier further.↩︎

  3. Cross-paper FPS incomparability: each detector paper reports throughput on its own hardware (K40, Titan X, V100, A100), test resolution, and batch size, with TensorRT/half-precision sometimes folded in. Within-family FPS comparisons are reliable; cross-family comparisons require re-benchmarking on a single platform, which the literature rarely provides.↩︎

  4. ImageNet-pretrained backbone dependence: detection pipelines are not trained from scratch; the entire family inherits ImageNet-pretrained ResNet/Swin features. The DeCAF result (Donahue et al. 2014) was the first systematic demonstration that mid-layer activations of a supervised ImageNet CNN transfer as off-the-shelf features to downstream tasks, and that observation seeded the entire pretrain-then-finetune paradigm that detection inherits. Scratch training (Detectron2 GroupNorm recipe, ×6\times 6 schedule) closes the gap on COCO but at 6×\times compute, and self-supervised backbones (MoCo v3, MAE) now match supervised pretraining for downstream detection.↩︎

  5. Scale variance: detection must localize objects spanning two orders of magnitude in pixel area within the same image (a 30-pixel pedestrian and a 300-pixel bus). Single-scale prediction misses one extreme; image pyramids cost N-fold compute. Addressed by feature pyramids (FPN, PANet, BiFPN), multi-scale anchors (Faster R-CNN, SSD), per-level FCOS regression ranges, and dynamic multi-scale training (YOLOv3, EfficientDet).↩︎

  6. Scale variance: detection must localize objects spanning two orders of magnitude in pixel area within the same image (a 30-pixel pedestrian and a 300-pixel bus). Single-scale prediction misses one extreme; image pyramids cost N-fold compute. Addressed by feature pyramids (FPN, PANet, BiFPN), multi-scale anchors (Faster R-CNN, SSD), per-level FCOS regression ranges, and dynamic multi-scale training (YOLOv3, EfficientDet).↩︎

  7. Cross-paper FPS incomparability: each detector paper reports throughput on its own hardware (K40, Titan X, V100, A100), test resolution, and batch size, with TensorRT/half-precision sometimes folded in. Within-family FPS comparisons are reliable; cross-family comparisons require re-benchmarking on a single platform, which the literature rarely provides.↩︎

  8. Two-stage vs one-stage trade-off: two-stage detectors filter proposals before classification, gaining accuracy on hard examples at the cost of inference latency; one-stage detectors predict densely in a single forward pass, paying for speed with the foreground/background imbalance problem. The "RetinaNet matches Faster R-CNN" result was the first empirical point at which one-stage stopped trading accuracy for speed; the EfficientDet line later pushed the Pareto frontier further.↩︎

  9. Anchor design: hand-tuned scale and aspect-ratio priors (Faster R-CNN's 3×\times3 = 9 anchors, SSD's per-level 4 or 6, RetinaNet's 3×\times3 across 5 levels). The choice transfers poorly across datasets and adds three hyperparameters per level. Anchor-free methods (FCOS, CornerNet, CenterNet) regress from a single per-pixel reference; YOLOv3 derives priors via k-means clustering on the dataset.↩︎

  10. Scale variance: detection must localize objects spanning two orders of magnitude in pixel area within the same image (a 30-pixel pedestrian and a 300-pixel bus). Single-scale prediction misses one extreme; image pyramids cost N-fold compute. Addressed by feature pyramids (FPN, PANet, BiFPN), multi-scale anchors (Faster R-CNN, SSD), per-level FCOS regression ranges, and dynamic multi-scale training (YOLOv3, EfficientDet).↩︎

  11. Label assignment ambiguity: which anchor (or per-pixel location) is responsible for which ground-truth box. Faster R-CNN uses IoU 0.7/0.3 thresholds; FCOS uses smallest-area box for overlapping GTs; ATSS adaptively picks the threshold per object via mean+std of candidate IoUs; OTA and DETR cast assignment as bipartite matching. The choice matters by 1-3 AP and remains an active research area.↩︎

  12. Scale variance: detection must localize objects spanning two orders of magnitude in pixel area within the same image (a 30-pixel pedestrian and a 300-pixel bus). Single-scale prediction misses one extreme; image pyramids cost N-fold compute. Addressed by feature pyramids (FPN, PANet, BiFPN), multi-scale anchors (Faster R-CNN, SSD), per-level FCOS regression ranges, and dynamic multi-scale training (YOLOv3, EfficientDet).↩︎

  13. Foreground/background imbalance: a 600×\times800 image with FPN produces ~100k anchor locations against ~10 ground-truth objects; the resulting ~1:1000 ratio means easy negatives dominate gradients under plain cross-entropy. Two-stage detectors filter via the RPN to ~1:3; SSD uses 3:1 hard negative mining; RetinaNet introduced focal loss to down-weight easy examples by (1−pt)γ (1-p_t)^\gamma . The classification-conv bias is initialized to −log⁡((1−π)/π),π=0.01 -\log((1-\pi)/\pi),\ \pi=0.01 so initial predictions match the empirical prior.↩︎

  14. Foreground/background imbalance: a 600×\times800 image with FPN produces ~100k anchor locations against ~10 ground-truth objects; the resulting ~1:1000 ratio means easy negatives dominate gradients under plain cross-entropy. Two-stage detectors filter via the RPN to ~1:3; SSD uses 3:1 hard negative mining; RetinaNet introduced focal loss to down-weight easy examples by (1−pt)γ (1-p_t)^\gamma . The classification-conv bias is initialized to −log⁡((1−π)/π),π=0.01 -\log((1-\pi)/\pi),\ \pi=0.01 so initial predictions match the empirical prior.↩︎

  15. Scale variance: detection must localize objects spanning two orders of magnitude in pixel area within the same image (a 30-pixel pedestrian and a 300-pixel bus). Single-scale prediction misses one extreme; image pyramids cost N-fold compute. Addressed by feature pyramids (FPN, PANet, BiFPN), multi-scale anchors (Faster R-CNN, SSD), per-level FCOS regression ranges, and dynamic multi-scale training (YOLOv3, EfficientDet).↩︎

  16. Label assignment ambiguity: which anchor (or per-pixel location) is responsible for which ground-truth box. Faster R-CNN uses IoU 0.7/0.3 thresholds; FCOS uses smallest-area box for overlapping GTs; ATSS adaptively picks the threshold per object via mean+std of candidate IoUs; OTA and DETR cast assignment as bipartite matching. The choice matters by 1-3 AP and remains an active research area.↩︎

  17. Anchor design: hand-tuned scale and aspect-ratio priors (Faster R-CNN's 3×\times3 = 9 anchors, SSD's per-level 4 or 6, RetinaNet's 3×\times3 across 5 levels). The choice transfers poorly across datasets and adds three hyperparameters per level. Anchor-free methods (FCOS, CornerNet, CenterNet) regress from a single per-pixel reference; YOLOv3 derives priors via k-means clustering on the dataset.↩︎

  18. Label assignment ambiguity: which anchor (or per-pixel location) is responsible for which ground-truth box. Faster R-CNN uses IoU 0.7/0.3 thresholds; FCOS uses smallest-area box for overlapping GTs; ATSS adaptively picks the threshold per object via mean+std of candidate IoUs; OTA and DETR cast assignment as bipartite matching. The choice matters by 1-3 AP and remains an active research area.↩︎

  19. NMS heuristic: hand-crafted greedy post-processing that suppresses overlapping detections by IoU. Threshold-sensitive, fails in dense crowds, and not differentiable. Soft-NMS, DIoU-NMS, and learned NMS soften it; DETR-style Hungarian matching removes it entirely by predicting a fixed-size set with one-to-one assignment.↩︎

  20. Occlusion and crowding: NMS suppresses overlapping detections of the same class even when they correspond to distinct instances. Soft-NMS preserves recall in crowded scenes; CrowdDet and Repulsion Loss explicitly model crowd density. Set-prediction transformers handle occlusion by construction since each query is one-to-one matched to one GT.↩︎

  21. Slow convergence (DETR): the original DETR required 500 COCO epochs (versus 12-24 for Faster R-CNN) due to unstable Hungarian-matching gradients and the absence of explicit object priors. Deformable-DETR, DAB-DETR, and DN-DETR cut this to ~50 epochs by introducing reference points, content-aware queries, and denoising auxiliaries.↩︎

  22. Cross-paper FPS incomparability: each detector paper reports throughput on its own hardware (K40, Titan X, V100, A100), test resolution, and batch size, with TensorRT/half-precision sometimes folded in. Within-family FPS comparisons are reliable; cross-family comparisons require re-benchmarking on a single platform, which the literature rarely provides.↩︎

  23. ImageNet-pretrained backbone dependence: detection pipelines are not trained from scratch; the entire family inherits ImageNet-pretrained ResNet/Swin features. The DeCAF result (Donahue et al. 2014) was the first systematic demonstration that mid-layer activations of a supervised ImageNet CNN transfer as off-the-shelf features to downstream tasks, and that observation seeded the entire pretrain-then-finetune paradigm that detection inherits. Scratch training (Detectron2 GroupNorm recipe, ×6\times 6 schedule) closes the gap on COCO but at 6×\times compute, and self-supervised backbones (MoCo v3, MAE) now match supervised pretraining for downstream detection.↩︎

  24. Long-tail class imbalance: COCO is roughly balanced (80 classes), but production data and LVIS exhibit power-law distributions. Detector heads trained on balanced data fail on rare classes; recipes like Equalization Loss, Seesaw Loss, and decoupled classifier finetuning address this orthogonally to the foreground/background problem.↩︎