PyTorch (machine learning)

PyTorch (machine learning)
Name	PyTorch
Developer	Meta Platforms
Initial release	2016
Latest release	(varies)
Programming language	C++, Python
License	BSD-style

PyTorch (machine learning)

PyTorch is an open-source deep learning framework for tensor computation and neural network development. It provides dynamic computation graphs, automatic differentiation, and a Pythonic interface used across research and industry. The project intersects with organizations and projects such as Meta Platforms, Facebook AI Research, Microsoft Research, Amazon Web Services, and NVIDIA while influencing work at institutions like Stanford University, MIT, Carnegie Mellon University, University of California, Berkeley.

History

PyTorch originated from research at Facebook AI Research and was unveiled in 2016 during a period of rapid growth in frameworks alongside TensorFlow, Theano, Caffe, and MXNet. Early contributors included researchers affiliated with NYU, University of Oxford, and University of Toronto, and the project drew on ideas from predecessors such as Torch (machine learning), Lua (programming language), and systems used at Google Research. Subsequent development involved collaborations with companies like NVIDIA and Microsoft and academic labs including DeepMind, OpenAI, ETH Zurich, Max Planck Society, and Tsinghua University. PyTorch’s evolution paralleled advances reported at conferences like NeurIPS, ICML, CVPR, ACL (conference), and ICLR, and integrations were showcased in workshops at SIGGRAPH and KDD. Major releases incorporated features inspired by projects such as ONNX, Apache TVM, and XLA, and governance shifted toward wider community involvement with contributors from Google, Intel, AMD, and research groups at Berkeley AI Research.

Architecture and design

PyTorch’s core is implemented in C++ with a front-end API in Python (programming language), enabling bindings used in environments like Jupyter Notebook, Google Colab, and Microsoft Azure Notebooks. The design emphasizes dynamic computation graphs resembling approaches used at Chainer and contrasts with static graphs popularized by TensorFlow. Automatic differentiation is provided by a system influenced by academic work at University of Toronto and libraries used at OpenAI; the autograd engine traces operations similar to techniques discussed at NeurIPS workshops. Backend execution leverages libraries from NVIDIA such as cuDNN and CUDA, and supports interoperability with compilers and runtimes like LLVM, Intel MKL, and ROCm from AMD. Device abstraction accommodates CPUs from Intel and ARM cores used in devices by Apple Inc. and Qualcomm.

Features and components

PyTorch includes primitives for tensors, neural modules, and optimizers, with modules reflecting design patterns from Torch (machine learning), Lasagne, and Keras. Key components include an autograd engine, a module system analogous to constructs used at Microsoft Research, and utilities for data handling comparable to frameworks at Amazon Web Services and Alibaba Group. Distributed training features integrate concepts from Horovod and systems demonstrated by Google Research and Facebook AI Research, enabling parameter servers and data-parallel strategies employed in projects at Uber AI Labs and Pinterest Research. Model serialization and export use formats and tools interoperable with ONNX, facilitating deployment to platforms such as TensorRT, OpenVINO, and runtimes employed by Apple and Samsung. Visualization and monitoring integrate with tools like TensorBoard, MLflow, and telemetry systems used in pipelines at Netflix and Airbnb.

Ecosystem and libraries

A broad ecosystem surrounds PyTorch, including higher-level libraries and toolkits from organizations such as Hugging Face, fast.ai, Detectron2 by Facebook AI Research, and scientific projects at Scikit-learn-adjacent groups. Natural language processing work ties to models and libraries influenced by teams at Google DeepMind, OpenAI, Allen Institute for AI, and datasets curated by Stanford NLP Group and Berkeley AI Research. Computer vision stacks draw from research at Microsoft Research Asia, University of Oxford Visual Geometry Group, and projects like OpenCV. Reinforcement learning libraries reflect codebases developed at DeepMind, OpenAI, and Berkeley AI Research. Community-driven packages are maintained by contributors from GitHub, GitLab, and research labs at Princeton University and Yale University.

Performance and benchmarking

Performance tuning relies on hardware-optimized libraries from NVIDIA (for example cuBLAS and cuDNN), compiler technologies from LLVM-affiliated projects, and kernel optimizations found in research at Intel Labs and AMD Research. Benchmarks compare PyTorch against TensorFlow, MXNet, and JAX in studies presented at ICLR and NeurIPS, with speed and memory metrics reported by teams at Facebook AI Research, Google Research, Microsoft Research, and Stanford University. Multi-node scaling is informed by distributed systems research from Amazon Web Services, Google Cloud Platform, and Microsoft Azure, and evaluated using workloads from industrial users such as Baidu Research, Alibaba DAMO Academy, and Tencent AI Lab.

Adoption and industry use

PyTorch is used extensively across industry and academia: researchers at MIT, Harvard University, Columbia University, and Caltech publish models implemented in the framework, while companies like Meta Platforms, Microsoft, Amazon, NVIDIA, Apple Inc., Tesla, Inc., Salesforce, Uber Technologies, Spotify, Netflix, Adobe Systems, and Siemens integrate PyTorch into products and services. It powers applications in areas pioneered by teams at Google Research and DeepMind such as computer vision, natural language processing, speech recognition, and recommender systems, and is deployed in production on cloud offerings by Amazon Web Services, Google Cloud Platform, Microsoft Azure, and edge platforms developed by ARM Holdings and Intel Corporation.

Licensing and governance

The project is distributed under a permissive BSD-style license and has governance involving corporate stakeholders like Meta Platforms and contributors from Microsoft, NVIDIA, Intel, and academic institutions including Stanford University and ETH Zurich. Community governance models echo practices from large open-source initiatives such as Linux Kernel and Apache Software Foundation, with contributions managed via platforms such as GitHub and discussions at events like NeurIPS and ICML.

Category:Machine learning frameworks