Self-Supervised Representation Learning: Instance Discrimination and InfoNCE

Self-supervised learning emerged as a paradigm to bridge the gap between unsupervised feature learning and supervised representation learning. Early approaches focused on pretext tasks such as image rotation prediction, jigsaw puzzle solving, and colorization; the field then evolved toward contrastive learning, non-contrastive and clustering siamese variants, masked modeling, self-distillation, and language supervision. Each paradigm has characteristic strengths and well-known open problems, walked through paragraph-by-paragraph after the SoTA leaderboard below.

The historical context is important for understanding the field's trajectory. Early SSL methods (2014-2018) produced features useful for downstream tasks but fell substantially behind supervised pretraining; the bar to beat was set by DeCAF (Donahue et al. 2014), which first showed that mid-layer activations of a supervised ImageNet CNN transfer as off-the-shelf features to almost any downstream classification task, founding the pretrain-then-finetune paradigm that SSL would later challenge. The 2019-2020 period saw SSL close the gap with supervised learning on ImageNet linear probing; by 2021-2022, with DINO and MAE on Vision Transformers, SSL began to exceed supervised pretraining on transfer tasks like COCO detection and ADE20K segmentation. The 2023-2024 era split along two lines. On the vision-only line DINOv2 (Oquab et al. 2024) reached supervised parity with frozen features, so the backbone stopped being a starting point and became a fixed component. On the image-text line OpenCLIP (Cherti et al. 2023) reproduced CLIP on public LAION data and measured its scaling exponent, and SigLIP (Zhai et al. 2023) replaced softmax normalisation with a pairwise sigmoid so the loss no longer needs a global view of the batch. The CLIP line has no single successor, it has several parallel ones, and that is the shape of the era: SSL became the default pretraining stage rather than one more competing method.

The specific pretext tasks explored in early SSL reveal the field's progression. Rotation prediction (Gidaris et al. 2018) trained the network to classify 4-way rotations (0, 90, 180, 270 degrees) on AlexNet, with a training trick that included all four rotated copies of each image in the same minibatch to avoid BatchNorm drift from rotation-mean shifts; rotations were implemented via flip+transpose rather than interpolation to avoid aliasing artifacts that would otherwise let the network cheat on low-level cues. RotNet reached 54.0% on ImageNet and 54.4% PASCAL VOC 2007 detection (only 2.4 points below supervised ImageNet pretraining); the 54.0% is not the same measurement as the per-layer frozen-conv linear probes that DeepCluster's Table 1 reports for the same network, so the two should not be read against each other without checking which protocol each paper used[fn:s0-probe-protocol]. Its failure mode is rotation-invariant classes like jellyfish or aerial views, for which the canonical-orientation prior provides no learning signal[fn:s0-semantic-narrow][fn:s0-invariance-fixed]. Jigsaw puzzle solving (Noroozi and Favaro 2016) predicted which permutation of a 3x3 tile grid had been applied, reducing the intractable 9!=362,880 permutation space to 100 or 1000 Hamming-well-separated permutations. Noroozi and Favaro's Context-Free Network (CFN) used a Siamese-ennead architecture (9 shared-weight CNN branches each seeing one tile, with cross-tile information only at the final FC fusion) and inserted gaps between tiles to block edge-alignment shortcuts. A further subtlety is the defense against chromatic aberration: natural lenses induce color fringing toward the image edges, which would trivially localize tiles; countered with per-tile channel-mean subtraction or grayscale training[fn:s0-shortcut-vulnerable]. Jigsaw reached 45.7% ImageNet linear probing and 51.8% VOC detection. Colorization (Zhang et al. 2016) reached 39.6% via a 313-way classification formulation (predicting a discretized bin of the ab color space rather than regressing continuous color), with class rebalancing to compensate for the heavy-tail distribution of natural colors; the 39.6% figure is specifically the cross-channel encoder variant where $L \to ab$ and $ab \to L$ are jointly trained[fn:s0-semantic-narrow]. These methods showed that self-generated supervision could learn useful features, but their reliance on specific geometric or chromatic priors limited generalization; modern methods (instance discrimination, masked modeling) avoid these priors with more generic learning objectives.

The table groups the literature into seven families, walked through in chronological order. Early pretext methods (RotNet, Jigsaw, Colorization) reach modest LP via hand-crafted self-supervisory tasks but suffer the same structural limitations: a hand-crafted signal is too narrow for high-level features[fn:s0-semantic-narrow] and is vulnerable to low-level shortcuts the network exploits in place of real understanding[fn:s0-shortcut-vulnerable].

Contrastive methods learn representations by maximizing agreement between augmented views of the same image while pushing apart views of different images (T. Chen et al. 2020). Instance discrimination established the foundation by treating each image as its own class; queue-based memory banks and momentum encoders (He et al. 2020) enabled efficient contrastive learning at scale by sidestepping the large-batch dependence[fn:s0-large-batch] that simple end-to-end variants like SimCLR require, at the cost of a queue whose features drift from the current encoder during training[fn:s0-queue-stale]. A deeper limitation cuts across the entire contrastive family: the augmentation pipeline declares which transformations should be invariant rather than letting the network learn them[fn:s0-invariance-fixed].

Non-contrastive methods including BYOL, SimSiam, and Barlow Twins achieve competitive results without explicit negative pairs, challenging the assumption that negatives are necessary for avoiding collapse. They rely on architectural asymmetries (a predictor head, stop-gradient, or a momentum encoder) or on redundancy reduction objectives to prevent degenerate solutions; why these recipes avoid the trivial constant solution remains contested[fn:s0-collapse-mystery] and the methods are notoriously hyperparameter-sensitive[fn:s0-hp-fragile].

Masked modeling, inspired by BERT in NLP (Bao et al. 2022), trains models to reconstruct masked portions of images. MAE (He et al. 2022) showed that simple pixel reconstruction with high masking ratios produces excellent representations; BEiT (Bao et al. 2022) introduced a discrete tokenizer trained offline as a separate dVAE pre-stage, which adds compute and constrains downstream by fixing the visual vocabulary[fn:s0-tokenizer-offline]. A persistent diagnostic for the masked modeling family is the linear-probe / fine-tune disagreement: MAE reports weak LP yet strong FT, while BEiT and its derivatives sit elsewhere on the same axis[fn:s0-lp-ft-gap]. Most 2021+ recipes (MAE, BEiT, MoCo v3, DINO) are also ViT-only and have not transferred well to ResNets[fn:s0-vit-only].

Self-distillation methods (DINO, iBOT, DINOv2) and clustering methods (SwAV via Sinkhorn assignment + multi-crop) form a third strand: a teacher-student loop without negatives where the teacher is an EMA of the student. DINOv2 (Oquab et al. 2024) combined self-distillation with masked modeling and large-scale scaling (ViT-g/14, 142M curated images) to produce the first SSL backbone matching supervised on every transfer benchmark; the price is production-scale compute beyond academic budgets and a closed-source data-curation pipeline[fn:s0-compute-cost].

Vision-language language supervision methods like CLIP (Radford et al. 2021) and ALIGN (Jia et al. 2021) use natural language supervision to learn transferable visual representations, enabling zero-shot transfer and catalyzing research in multimodal learning. They also inherit web-text bias[fn:s0-web-bias] and the same compute scaling limits[fn:s0-compute-cost].

The table separates four eras: pretext-based (2016-2018), contrastive (2018-2021), non-contrastive + clustering + masked (2020-2022), and the modern "best of both" line (DINOv2, iBOT, MAE-derivatives). Three readings. First, the contrastive-to-non-contrastive transition was empirical: BYOL and SimSiam dropped negatives entirely with no degradation, refuting the prior assumption that negatives are essential. Second, masked image modeling is not Pareto-dominated by contrastive: MAE has lower linear-probe but higher fine-tuning and downstream segmentation, indicating its representations encode useful structure that is harder to read off linearly. Third, the gap to supervised pretraining (ResNet-50: 76.2% IN1k FT) closed around BYOL/SwAV (2020) for ResNets and around DINO (2021) for ViTs; by 2024 DINOv2 exceeds supervised on every transfer benchmark tested.

The leaderboard records outcomes, not decisions. Three choices fix a method: which pair of things must agree, which space that agreement is measured in, and what stops the agreement from being bought by a constant output. The first is the augmentation or masking policy, the second is the projection target, the third is the collapse mechanism dissected in Understanding Representation Collapse. Backbone and training corpus are separate axes, and the delta ledger under ImageNet Linear Probing shows they move the numbers as hard as the objective does, which is why they have to be held fixed when a method is read against its predecessor. The next table is the leaderboard's other half. It says what changed between two rows where the leaderboard says only by how much, and the rest of this chapter is a walk down its three columns.

Two readings follow from the columns rather than from the numbers. The target space column is where the field actually moved: from a 128-d sphere in 2018-2020, to a large prototype simplex in 2020-2021, to raw pixels in 2021-2022, and to another network's features in 2022 onward. The anti-collapse column shrinks as the target space grows, because a target the model does not produce cannot be gamed by a constant, which is why the masked and pretext rows need no mechanism at all and the joint-embedding rows need one apiece.

Encyclopedia references: self-supervised learning, Vision Transformer, SimCLR, MAE, DINO, DINOv2.

References