Vision-Language Models: CLIP, ALIGN, and SigLIP

The foundation of modern VLMs was established by CLIP (Radford et al. 2021), which showed that contrastive learning on 400 million image-text pairs enables zero-shot transfer to diverse visual tasks.[fn:s0-clip-impl] CLIP's text tower inherits a long lineage: distributed word representations trained by contrasting context-word co-occurrences against negative samples (the word2vec construction of (Mikolov et al. 2013), which introduced negative sampling as a replacement for hierarchical softmax, frequent-word subsampling, and phrase representations, making billion-token corpora trainable on a single machine) were the first widely-deployed evidence that geometry in a learned embedding space reflects semantic structure, and the InfoNCE loss CLIP uses is essentially the same noise-contrastive estimator generalised to cross-modal positive pairs. ALIGN (Jia et al. 2021) showed that noisy web-scale data can substitute for curation when sufficient scale is achieved. SigLIP (Zhai et al. 2023) introduced a simpler sigmoid loss that scales more efficiently than softmax-based contrastive learning. The contrastive pre-training lineage also includes earlier work like VirTex (2020, caption-based supervision), ConVIRT (2020, medical domain), and UniCL (2022, unified classification and contrastive learning), with CLIP's influence coming from its combination of scale, simplicity, and public release. The longer arc reaches back to Baby Talk (Kulkarni et al. 2011), the first end-to-end pipeline that generated natural-language descriptions of unseen images from hand-engineered object/attribute/preposition detectors fed into a tree-CRF; that paper's central claim that visual content can be linearised into compositional language predates CLIP-style contrastive coupling by a decade and frames every subsequent VLM benchmark. Subsequent refinements include OpenCLIP (open reimplementation with documented data), FILIP (fine-grained token-level alignment), and MetaCLIP (careful data curation matching CLIP's original recipe).

Self-supervised visual learning reached new heights with DINOv2 (Oquab et al. 2024), which produces all-purpose visual features through careful data curation and training at scale (below). Chinese CLIP (Yang et al. 2023) extended contrastive pre-training to Chinese language.

The BLIP series (Li et al. 2022; J. Li et al. 2023; Xue et al. 2024) introduced bootstrapping approaches that filter noisy captions and enable efficient adaptation of frozen encoders (BLIP, BLIP-2). InstructBLIP (Dai et al. 2023) showed the power of instruction tuning for vision-language models. Parallel efforts in this space include ALBEF (2021, introduced the unified encoder-decoder design with image-text matching and contrastive objectives), OFA (2022, unified multimodal sequence-to-sequence learning), and mPLUG (2022-2023, modular cross-attention with skip connections). The BLIP line's distinguishing contribution was CapFilt-style bootstrapping, which has since become a standard technique across the VLM training pipeline.

Flamingo (Alayrac et al. 2022) pioneered few-shot learning in VLMs through interleaved image-text inputs. LLaVA (Liu et al. 2023, 2024; Lee et al. 2024) established simple yet effective approaches for visual instruction tuning. CogVLM (Wang et al. 2024) introduced visual expert modules for deep vision-language fusion, while Qwen-VL (Bai et al. 2023) showed strong multilingual capabilities. Contemporary open-source VLMs that extend or compete with these lines include InternVL (2024, high-resolution scaling), IDEFICS (2023-2024, open Flamingo reproduction), MiniGPT-4/v2 (2023, early LLaVA-style open demonstrator), mPLUG-Owl (2023-2024, modular design with visual abstractor), and PaliGemma (2024, Google's open SigLIP + Gemma combination).

Efficient VLMs for deployment include MobileVLM (Chu et al. 2023), TinyGPT-V (Yuan et al. 2024), and LLaVA-Phi (Zhu et al. 2024); see Efficient Vision-Language Models. Frontier systems like GPT-4V (OpenAI 2023) and Gemini (Team et al. 2023, 2024) lead most published leaderboards in multimodal understanding. Additional closed frontier systems include Claude 3.5 Sonnet (Anthropic), GPT-4o (OpenAI, successor to GPT-4V with stronger multimodal integration), and Grok-Vision (xAI). The closed-vs-open gap has narrowed substantially: as of late 2024, the best open models (Qwen2-VL-72B, InternVL-2-76B) match or exceed GPT-4V on most academic benchmarks, though closed models retain a lead in qualitative usability, reasoning depth, and safety behavior.

Recent systematic surveys include Zhang et al.'s "Vision-Language Models for Vision Tasks: A Survey" and the MMLMs survey by Yin et al., neither of which has an entry in this bibliography; both provide broader coverage complementing the focus here. Reading pathways: for contrastive pre-training, start with CLIP and follow forward through SigLIP; for generative VLMs, follow BLIP -> BLIP-2 -> LLaVA -> LLaVA-NeXT; for frontier systems, read Gemini and GPT-4V system cards alongside technical reports.

SoTA Leaderboard

GQA was dropped to keep the table readable at A4 width. Instruction-tuned VLM GQA scores cluster between 49 and 67 (InstructBLIP-13B 49.5, Qwen-VL-Chat 57.5, LLaVA-1.5-13B 63.3, CogVLM 65.2, LLaVA-NeXT-34B 67.1) and are visited per-model in the chapter sections below. The " - " entries are not failures: contrastive dual-encoders cannot answer free-form VQA questions without a generative head; instruction VLMs are not optimized for ImageNet zero-shot prompting; closed frontier systems (GPT-4V, Gemini) do not publish architectural details.[fn:s0-problem-vlm-comparability]

The contrastive era (rows 1-7) opens with CLIP, whose ViT-L/14-336 set the 76.2% ImageNet zero-shot bar that anchored the field for two years. ALIGN matched it the same year using 1.8B noisy web pairs, validating that curation could be replaced by scale. FLIP introduced patch-level masking to halve training compute at modest accuracy cost. OpenCLIP reproduced the recipe openly on LAION-2B, climbing to 80.1% with the ViT-G/14 model and giving the academic community its first matched-spec CLIP variant. EVA-CLIP layered MIM-pretrained vision backbones onto LAION to push to 82.0%, while SigLIP swapped softmax InfoNCE for sigmoid loss and reached 83.0% on WebLI-10B; the SigLIP recipe has since become the default vision tower for downstream VLMs (PaliGemma, LLaVA-NeXT-Sigmoid). Chinese-CLIP (row 7) extended the pattern to non-English, with bilingual retrieval scores no English-trained CLIP can match.

The generative and bootstrapping era (rows 8-14) introduced unified objectives for retrieval, captioning, and VQA simultaneously. BLIP's CapFilt cleaned web captions and reached 78.2 VQAv2 with only 129M training pairs. CoCa fused contrastive and captioning losses in one decoder, hitting 86.3 ImageNet zero-shot (the contrastive ceiling at the time) and 82.3 fine-tuned VQAv2. BLIP-2's Q-Former showed that frozen-encoder/frozen-LLM bridging trains for a fraction of full-finetune cost and still reaches 65.0 zero-shot VQAv2 with FlanT5-XXL. BEiT-3 unified vision and language as a single Multiway Transformer, taking 84.0 fine-tuned VQAv2 with only 35M training pairs by reusing the same backbone for image, text, and image-text inputs. Flamingo and its open clone IDEFICS-80B introduced gated cross-attention and Perceiver resamplers to enable few-shot multimodal in-context learning, while Florence-2 went the opposite direction (dense prediction unification with DaViT and a BART-style decoder).

The instruction era (rows 15-22) replaced fine-tuning with visual instruction tuning. MiniGPT-4 showed the basic recipe, LLaVA-1.5 cleaned it up with a 2-layer MLP projector and 336px CLIP, reaching 80.0 VQAv2 on a 1.2M-sample budget. InstructBLIP turned 26 datasets into instruction format atop the BLIP-2 Q-Former. Qwen-VL added bounding-box supervision and bilingual training. CogVLM introduced "visual expert" parallel layers (separate QKV/FFN per Transformer block) for deeper fusion, reaching 82.3 VQAv2 without forgetting language ability. LLaVA-NeXT scaled to dynamic resolution and 34B LLM backbones (83.7 VQAv2). The closed frontier (GPT-4V, Gemini-1.5 Pro) reports lower academic VQAv2 (77.2 / 73.2) than the best open-source models on the same benchmark, but holds qualitative leads on long-context, document, and reasoning evaluations not tabulated here; the closed-vs-open gap that mattered most in 2023 has been substantially closed by late 2024 on extractive benchmarks while remaining wide on open-ended, multi-capability evaluations.

References