Turing-NLG

Turing-NLG was a large-scale autoregressive language model developed by Microsoft and announced in February 2020. It was a significant milestone in the AI arms race for natural language generation, briefly holding the record for the largest publicly detailed model of its kind. The model demonstrated advanced capabilities in tasks like text summarization, question answering, and open-domain dialogue.

● Overview

Turing-NLG represented a major advancement in the scale of generative artificial intelligence models available from a major technology corporation. It was part of a broader initiative within Microsoft Research to push the boundaries of deep learning for natural language processing (NLP). The model's release followed notable predecessors like OpenAI's GPT-2 and preceded the explosive growth in model size seen with later systems such as GPT-3 and Google's PaLM. Its development was closely tied to the Microsoft Turing project, which aimed to unify AI model advancements across the company's product suite, including improvements to the Bing search engine and the Microsoft Office productivity suite.

● Development and architecture

The development of Turing-NLG was led by teams at Microsoft Research AI and leveraged the company's substantial investment in Azure cloud infrastructure. Architecturally, it was a decoder-only model based on the Transformer architecture, similar to the GPT-2 framework pioneered by OpenAI. With 17.2 billion parameters, it substantially exceeded the 1.5 billion parameters of the largest GPT-2 variant. Training utilized a novel distributed training technique across hundreds of NVIDIA V100 GPUs to manage the immense computational load, a feat documented in research presented at conferences like NeurIPS. The model was pre-trained on a massive corpus of text data from the internet, including sources like Wikipedia, news articles, and books.

● Capabilities and performance

Benchmark evaluations demonstrated that Turing-NLG achieved state-of-the-art results on several prominent NLP benchmarks at the time of its release. It excelled in the LAMBADA dataset for word prediction, the RACE dataset for reading comprehension, and the WikiText benchmark for language modeling. The model showed particularly strong performance in zero-shot learning and few-shot learning scenarios, where it could perform new tasks with minimal specific examples. Its text generation was notably coherent and contextually relevant over long passages, enabling sophisticated dialogue system prototypes and complex document summarization that surpassed earlier models like BERT.

● Applications and impact

While not released as a standalone public product, Turing-NLG's technology was integrated into various Microsoft services and informed future projects. Its capabilities directly enhanced features in Microsoft Office, such as advanced autocomplete and ideas in Word, and improved answer generation in the Bing search engine. The model also served as a foundational research platform for exploring AI safety and bias mitigation techniques within large language models. Its development accelerated internal projects at Microsoft, culminating in the unified Microsoft Turing model family and influencing the approach to massive models seen in subsequent efforts like the Megatron-Turing NLG collaboration with NVIDIA.

● Limitations and criticism

Despite its technical achievements, Turing-NLG exhibited limitations common to large language models of its era. It could generate factually incorrect or nonsensical text, and its outputs sometimes reflected and amplified societal biases present in its training data. The model's enormous computational cost for training and inference raised concerns about the environmental impact of artificial intelligence and the carbon footprint of such research. Furthermore, its limited public accessibility, in contrast to some open-source models, sparked discussions about corporate control of AI and the democratization of artificial intelligence. These critiques were part of a broader ethical debate highlighted by organizations like the Partnership on AI and researchers at institutions like the University of Washington.

Category:Microsoft software Category:Natural language processing Category:Artificial intelligence projects