Attention Is All You Need

Attention Is All You Need
Yuening Jia · CC BY-SA 3.0 · source
Title	Attention Is All You Need
Authors	Ashish Vaswani; Noam Shazeer; Niki Parmar; Jakob Uszkoreit; Llion Jones; Aidan N. Gomez; Lukasz Kaiser; Illia Polosukhin
Year	2017
Venue	NIPS (NeurIPS)

Attention Is All You Need

"Attention Is All You Need" is a 2017 paper introducing the Transformer, a neural network architecture that replaces recurrent and convolutional structures with self-attention mechanisms. It had major impact on subsequent work in natural language processing, machine translation, and large-scale pretrained models, influencing research at organizations such as Google, OpenAI, Facebook AI Research, DeepMind, and institutions like Stanford University, Massachusetts Institute of Technology, and University of Toronto. The paper was presented at NeurIPS and quickly became foundational for models developed by teams at Google Brain, Microsoft Research, and commercial deployments at Amazon Web Services and IBM Research.

Background and Motivation

The paper was motivated by limitations observed in sequence models used in systems such as the Bahdanau et al. 2015 encoder–decoder frameworks and by work from groups including Yoshua Bengio's lab, the Google Brain team, and researchers associated with Carnegie Mellon University and University of Oxford. Prior dominant approaches involved architectures like Long Short-Term Memory and gated recurrent units used in projects from Facebook, DeepMind, and Microsoft Research, as well as convolutional sequence models explored at Facebook AI Research and by researchers affiliated with Google DeepMind. The authors argued that attention mechanisms, building on earlier attention research from Bahdanau, Luong et al., and related efforts at Stanford University and University of Montreal, could yield better parallelization and improved long-range dependency modeling than models in work by groups at University of Edinburgh and TITANIC-era sequence research.

Transformer Architecture

The Transformer architecture uses multi-head self-attention, positional encodings, residual connections, and layer normalization in an encoder–decoder stack; these design choices echo techniques used by researchers at Google, Oxford University, Imperial College London, and labs like DeepMind and Microsoft Research. Key components include scaled dot-product attention, inspired by matrix methods common in labs at MIT and ETH Zurich, and multi-head attention that parallels parallel computation trends in systems developed by NVIDIA and Intel. The encoder and decoder each consist of repeated layers; encoder layers focus on self-attention and feed-forward networks, while decoder layers add masked self-attention and encoder–decoder attention, reflecting design evolution discussed at conferences like ACL and ICML and in work from CMU and Berkeley AI Research.

Training and Optimization

The paper describes training on parallel corpora and optimization with Adam, learning-rate schedules, label smoothing, and dropout—techniques used widely across groups from Google Brain, OpenAI, Facebook AI Research, and Microsoft Research. Training experiments leveraged hardware ecosystems from NVIDIA GPUs and Google's TPUs, techniques that intersect with large-scale training efforts at Amazon Web Services and compute projects at Lawrence Berkeley National Laboratory. The authors used datasets and evaluation metrics standard in machine translation research such as WMT and BLEU scores, building on benchmark traditions established by teams at University of Edinburgh, University of Edinburgh Machine Translation Group, and research presented at EMNLP and COLING.

Performance and Applications

Transformers achieved state-of-the-art results in machine translation and became the basis for downstream models like BERT, GPT, and T5, developed by teams at Google Research, OpenAI, and Facebook AI Research. The architecture enabled rapid progress in tasks assessed at venues such as NAACL, ACL, and EMNLP and catalyzed applications in industry by companies including Amazon, Microsoft, Apple Inc., and Salesforce. Beyond translation, Transformers have been adapted for speech by groups at DeepMind and Facebook AI Research, for vision by researchers at Facebook AI Research and Google Research, and for multimodal systems investigated at Microsoft Research and Stanford University.

Analysis and Extensions

Subsequent analysis and extensions built on the original paper include sparse attention variants, efficient Transformers from groups at Google Research and Facebook AI Research, and scaling studies by OpenAI and DeepMind comparing model families. Architectures like Transformer-XL, Sparse Transformer, and Reformer were proposed by teams affiliated with Google Brain, Carnegie Mellon University, and ETH Zurich, while pretrained encoder and decoder variants such as BERT and GPT advanced benchmarks in work from Google Research and OpenAI. Theoretical analysis by researchers at MIT, Princeton University, and Harvard University examined expressivity and inductive biases, and system-level improvements were driven by engineering at NVIDIA, Google, and Intel.

Criticism and Limitations

Critiques have focused on computational cost, data and energy demands highlighted by studies from Stanford University and University of Massachusetts Amherst, and issues of bias and safety raised by researchers at OpenAI, Google Research, and AI Now Institute. Concerns about interpretability motivated methods from groups at Carnegie Mellon University and ETH Zurich, and limitations for low-resource languages were discussed in work from University of Edinburgh and Johns Hopkins University. Debates at venues like NeurIPS and ICML continue about environmental impact, fairness, and governance, engaging institutions including UNESCO and policy units at European Commission.

Category:Neural network architectures