This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| ViT (Vision Transformer) | |
|---|---|
| Name | ViT (Vision Transformer) |
| Introduced | 2020 |
| Authors | Alexey Dosovitskiy; Lucas Beyer; Alexander Kolesnikov; Dirk Weissenborn; Xiaohua Zhai; Thomas Unterthiner; Mostafa Dehghani; Matthias Minderer; Georg Heigold; Sylvain Gelly; Jakob Uszkoreit; Neil Houlsby |
| Institution | Google Research; DeepMind; University of Toronto |
| Keyword | computer vision; neural network; transformer |
ViT (Vision Transformer) is a deep learning model that applies the Transformer architecture, originally developed for Natural Language Processing tasks such as GPT-3 and BERT, to image recognition by treating image patches as token sequences. Proposed in 2020 by researchers associated with Google Research and DeepMind, ViT sparked broad interest across research groups including Facebook AI Research, MIT CSAIL, Stanford AI Lab, and University of Toronto. It influenced follow-on work at institutions like OpenAI, Carnegie Mellon University, ETH Zurich, University of Oxford, and industrial labs such as Microsoft Research and NVIDIA Research.
ViT reframes image classification by segmenting an input image into fixed-size patches, projecting them into linear embeddings, and processing the resulting sequence with the Transformer encoder invented by researchers at Google Research (notably Vaswani et al.). The model sidesteps convolutional inductive biases emphasized in architectures like AlexNet, VGG, ResNet, and Inception and draws conceptual lineage from sequence models such as BERT, Transformer-XL, and XLNet. Early demonstrations compared ViT against benchmarks established by ImageNet, COCO, ADE20K, and spurred comparisons with hybrid designs developed at labs including DeepMind, Facebook AI Research, and Google Brain.
ViT divides images into non-overlapping patches (e.g., 16×16) and flattens each into a vector that is linearly projected to form patch embeddings; positional embeddings are then added similar to approaches used in BERT and GPT-2. The core consists of stacked self-attention and feed-forward blocks much like the Transformer encoder, using multi-head attention, layer normalization, and residual connections as in work from Google Research and OpenAI. A learnable classification token aggregates global information analogous to sequence-level representations in BERT; output heads mirror practices from models such as ResNet and fine-tuning regimes developed in papers from Stanford and Carnegie Mellon University. Implementation details often borrow optimizers and schedulers popularized by teams at Facebook AI Research and Microsoft Research.
Training ViT at scale typically requires large datasets such as ImageNet-21k, JFT-300M, and web-scale collections compiled by labs like Google Research and OpenAI, or self-supervised pretraining frameworks pioneered by SimCLR (Google Research), MoCo (Facebook AI Research), and BYOL (researchers at DeepMind and Google. Optimization strategies include Adam and AdamW optimizers advanced in research from Facebook AI Research and Microsoft Research, learning rate schedules such as cosine annealing from work by Loshchilov and Hutter, and regularization methods like stochastic depth and mixup inspired by contributions from Facebook AI Research and Google Brain. Techniques for data augmentation and distillation reference innovations from Google Research, Stanford, and Berkeley AI Research.
ViT spawned numerous derivatives including hybrid CNN-Transformer models developed at Facebook AI Research and Google Research; hierarchical designs like Swin Transformer (Microsoft Research Asia); efficient variants such as DeiT from Facebook AI Research and distillation techniques introduced by teams at Facebook AI Research and Facebook AI; and locality-aware models influenced by work at ETH Zurich, Oxford, and University of Cambridge. Extensions also include vision-language hybrids in projects across OpenAI, Google Research, and Microsoft Research (e.g., multimodal models akin to CLIP), self-supervised adaptations by DeepMind and Google Brain, and resource-constrained versions developed at NVIDIA Research and ARM Research.
On image classification benchmarks like ImageNet and transfer tasks such as CIFAR-10, CIFAR-100, and VTAB, ViT and its successors have achieved state-of-the-art or competitive results when pretrained on large datasets used by Google Research and DeepMind. Comparative studies from Stanford, MIT, and Berkeley AI Research evaluated ViT against convolutional baselines such as ResNet and attention-based hybrids from Facebook AI Research and reported strengths in scaling and transfer but weaknesses on small-data regimes noted by researchers at ETH Zurich and University of Oxford. Benchmark suites maintained by Papers with Code and leaders at AI2 showcased metric trends across FLOPs, latency profiles studied by NVIDIA Research and energy considerations explored at Microsoft Research.
ViT variants have been applied across domains including medical imaging at groups like Mayo Clinic and Johns Hopkins University; remote sensing collaborations involving NASA and ESA; autonomous driving research at Waymo, Tesla, and Uber ATG; satellite imagery analysis by Planet Labs and Maxar Technologies; and multimodal systems in projects at OpenAI, Google DeepMind, and Microsoft Research integrating vision with language for tasks akin to Visual Question Answering and image captioning explored by Facebook AI Research and Stanford. Industrial deployments and startups in healthcare, agriculture, and surveillance have experimented with ViT-based backbones developed by teams at NVIDIA, Amazon Web Services, and Intel Labs.
Challenges include data efficiency and large-scale pretraining requirements noted by researchers at Google Research and DeepMind; robustness and adversarial vulnerabilities studied at ETH Zurich and MIT; interpretability concerns discussed by Stanford and Berkeley AI Research; and computational cost and memory footprint assessed by NVIDIA Research and Microsoft Research. Open problems involve domain adaptation work from CMU and Oxford, theoretical understanding pursued at Google DeepMind and Harvard University, and standards for benchmarking inspired by Papers with Code and initiatives at AI2 and NeurIPS.
Category:Computer vision models