RoBERTa

RoBERTa
Name	RoBERTa
Type	Transformer-based language model
Introduced	2019
Developers	Facebook AI Research
Architecture	Transformer encoder
Parameters	up to 355 million (base), 1.5 billion (large variants)
License	research use

RoBERTa is a transformer-based masked language model developed by Facebook AI Research that builds on the architecture of earlier models to improve pretraining and downstream performance. It reevaluated pretraining choices, scaling, and data usage to achieve stronger results on natural language understanding benchmarks. The model influenced subsequent work across industry and academia, affecting research at organizations and projects worldwide.

Background

RoBERTa emerged from a lineage of transformer research including Google Research, OpenAI, Stanford University, DeepMind, Microsoft Research, Carnegie Mellon University, Massachusetts Institute of Technology, University of California, Berkeley, University of Oxford, University of Cambridge, ETH Zurich, University of Toronto, University College London, Princeton University, Columbia University, Yale University, Cornell University, University of Washington, University of Pennsylvania, University of Edinburgh, Johns Hopkins University, University of Michigan, New York University, Imperial College London, Tsinghua University, Peking University, National University of Singapore, Stanford AI Lab, Berkeley AI Research, Allen Institute for AI, NVIDIA, Google Brain, OpenAI Gym, Hugging Face, ACL Anthology, NeurIPS, ICLR, ICML, EMNLP, AAAI, SIGIR, Reuters, The New York Times, BBC News, Nature (journal), Science (journal), ArXiv.

Model architecture

RoBERTa uses the Transformer encoder architecture introduced by researchers at Google Research and popularized through implementations at OpenAI and Stanford University. It retains multi-head self-attention and position-wise feed-forward networks that echo structures used by models from DeepMind and Microsoft Research. Variants align in parameterization with sizes investigated by teams at NVIDIA and hardware optimization groups at Intel Corporation and AMD to target inference on accelerators used by Amazon Web Services, Google Cloud Platform, and Microsoft Azure. The model design reflects architectural choices discussed at conferences like NeurIPS, ICLR, and ICML and implemented in toolkits from Hugging Face, PyTorch, and TensorFlow.

Pretraining and training procedure

The pretraining regime for RoBERTa reexamined strategies from earlier work at Google Research and OpenAI, adjusting masking, sequence length, batch size, and training time. It trained on large corpora sourced from datasets and providers including Common Crawl, Wikipedia, BookCorpus, OpenWebText, Toronto BookCorpus, Project Gutenberg, Wikimedia Foundation, and web-scale collections curated by teams at Facebook AI Research and collaborators at Hugging Face. Training protocols used optimizers and schedulers discussed in papers from Stanford University and Carnegie Mellon University, and leveraged distributed training frameworks developed at NVIDIA and Facebook across clusters similar to those used by Google and Microsoft. Evaluation employed benchmarks such as GLUE, SQuAD, SuperGLUE, and tasks highlighted at EMNLP and ACL Anthology.

Performance and benchmarks

RoBERTa produced gains on standardized benchmarks originally used to evaluate models from Google Research, OpenAI, and Microsoft Research. Reported improvements appeared on GLUE, SQuAD, SuperGLUE, and tasks featured in workshops at NeurIPS and ICLR. Comparative studies contrasted RoBERTa with models like those from OpenAI and research groups at DeepMind and Google Brain, and influenced leaderboard entries hosted by organizations such as Hugging Face and datasets curated by Stanford University. Performance assessments were widely discussed in venues including Nature (journal), Science (journal), and computational linguistics conferences like ACL Anthology.

Variants and adaptations

Following its release, RoBERTa inspired adaptations and fine-tuned variants created by teams at Hugging Face, Microsoft Research, Allen Institute for AI, NVIDIA, Amazon Web Services, Google Research, Stanford University, Carnegie Mellon University, University of Washington, University of Toronto, ETH Zurich, University of Cambridge, and industry groups at Facebook AI Research. Community-maintained checkpoints and distilled versions were developed by practitioners associated with Hugging Face and research consortia participating in ICLR and EMNLP. Domain-specific adaptations addressed domains represented in corpora from Reuters, The New York Times, PubMed, arXiv, IEEE Xplore, and legal texts used by institutions like Harvard University and Yale University.

Applications and use cases

RoBERTa and its forks have been applied by teams at Google, Microsoft, Amazon, Facebook, Apple Inc., IBM, Salesforce, Baidu, Alibaba Group, Tencent, NVIDIA, Hugging Face, Stanford University, Massachusetts Institute of Technology, Johns Hopkins University, University of Oxford, Imperial College London, Columbia University, University of Toronto, National University of Singapore, Peking University, Tsinghua University, Carnegie Mellon University, University of Michigan, and Cornell University for tasks including question answering on datasets like SQuAD, sentiment classification for outlets such as BBC News and The New York Times, information extraction for repositories like PubMed and arXiv, and retrieval-augmented tasks evaluated at conferences like SIGIR and NeurIPS.

Limitations and ethical considerations

Limitations and ethical concerns were raised by scholars from Harvard University, Yale University, Stanford University, University of Oxford, University of Cambridge, MIT Media Lab, AI Now Institute, Electronic Frontier Foundation, ACLU, United Nations, and policy groups participating in discussions at NeurIPS and ICLR. Issues include potential bias traced to corpora from Common Crawl, privacy implications discussed by researchers at Carnegie Mellon University and Johns Hopkins University, environmental cost analyses by teams at University of Massachusetts Amherst and University of Edinburgh, and misuse scenarios considered by OpenAI, Google DeepMind, and Facebook AI Research. Mitigation strategies have been proposed in workshops at ACL Anthology, NeurIPS, ICLR, and forums involving regulators and institutions such as European Commission and U.S. National Institute of Standards and Technology.

Category:Language models