Boltzmann machine

Boltzmann machine
Public domain · source
Name	Boltzmann machine
Invented by	Geoffrey Hinton, David Ackley, Terrence Sejnowski
Year	1985
Field	Machine learning, Neural network
Related	Hopfield network, Restricted Boltzmann machine, Deep belief network, Energy-based model

Boltzmann machine is a class of stochastic recurrent neural network originally proposed by Geoffrey Hinton, David Ackley, and Terrence Sejnowski in 1985. It combines ideas from statistical physics and connectionist models to perform probabilistic modeling, associative memory, and unsupervised learning, and influenced later architectures such as Restricted Boltzmann machine, Deep belief network, and Variational autoencoder. Boltzmann machines have been studied within communities around University of Toronto, Carnegie Mellon University, and California Institute of Technology and have connections to concepts from Ludwig Boltzmann's statistical mechanics, John von Neumann's computing theories, and the Turing Award-winning work of several contributors.

Introduction

A Boltzmann machine is a network of symmetrically connected stochastic binary units which settle into a thermal equilibrium described by a Boltzmann distribution. The model was introduced in the mid-1980s by Geoffrey Hinton, David Ackley, and Terrence Sejnowski while they were affiliated with institutions such as Carnegie Mellon University and University of California, San Diego. It generalizes earlier work on content-addressable memory such as the Hopfield network, and influenced subsequent developments by researchers at Massachusetts Institute of Technology, Stanford University, and Google DeepMind. Boltzmann machines are energy-based models whose learning is driven by gradients of likelihood and has historical ties to statistical physics via scholars like Ludwig Boltzmann and methodological links to probabilistic modeling advanced at Bell Labs and IBM Research.

Structure and variants

A standard Boltzmann machine consists of visible units and hidden units with undirected symmetric weights; early expositions appeared in papers from Geoffrey Hinton, David Ackley, and Terrence Sejnowski. The fully connected form is computationally expensive, prompting variants such as the Restricted Boltzmann Machine (RBM) popularized by Geoffrey Hinton and used in systems by Microsoft Research and Amazon Web Services. Other variants include Gaussian-Boltzmann models employed in collaborations involving University of Toronto researchers, conditional Boltzmann machines explored by teams at Adobe Research and Toyota Research Institute, and deep compositions like Deep Belief Networks developed by groups including Hinton and colleagues at University of Toronto and Google. Architectures inspired by Boltzmann machines intersect with models produced at Facebook AI Research and theoretical investigations at Princeton University and Harvard University.

Learning and training algorithms

Training Boltzmann machines involves adjusting symmetric weights to maximize data likelihood via gradient estimates that compare model statistics to data statistics; foundational algorithms were developed by Ackley, Hinton, and Sejnowski. Exact learning requires computing expectations over an intractable partition function, which led to approximate methods including Gibbs sampling used in implementations from Stanford Linear Accelerator Center experiments and contrastive divergence introduced by Geoffrey Hinton and applied in systems at Google. Persistent contrastive divergence and stochastic maximum likelihood were developed and evaluated by teams at Carnegie Mellon University and University of Toronto to improve mixing and convergence. Advanced training strategies combine ideas from Yann LeCun's optimization work, Yoshua Bengio's deep learning research, and regularization techniques seen in studies at Massachusetts Institute of Technology and École Polytechnique Fédérale de Lausanne.

Applications

Boltzmann machines and their variants have been applied across domains by researchers at institutions such as Stanford University, MIT, Bell Labs, and industrial labs like IBM Research, Google, Microsoft Research, and Facebook AI Research. RBMs have been used for collaborative filtering in recommendation systems developed by teams at Netflix and Amazon, feature learning in speech processing evaluated at Carnegie Mellon University and Microsoft Research, and pretraining layers in image recognition pipelines examined at University of Toronto and Google Brain. Conditional forms and temporal extensions have been applied in sequence modeling for projects at Adobe Research, Toyota Research Institute, and DeepMind. Theoretical use-cases include modelling biological neural populations studied at California Institute of Technology and Harvard Medical School and connections to statistical physics explored at Princeton University.

Theoretical properties and analysis

Boltzmann machines define a probability distribution via an energy function whose stationary distribution is the Boltzmann distribution; this formalism ties to classical work by Ludwig Boltzmann and to contemporary probabilistic inference developed at Bell Labs and IBM Research. The complexity of exact inference links to computational complexity results studied at MIT and Stanford University, while approximation bounds and convergence analyses have been pursued by researchers at Carnegie Mellon University, University of Toronto, and Princeton University. Relationships to variational methods and Markov chain Monte Carlo techniques connect Boltzmann machines to frameworks advanced by Radford Neal and Christopher Bishop and to subsequent deep probabilistic models developed by Yoshua Bengio and Geoffrey Hinton. Properties such as representational capacity, mixing times, and phase transitions have been analyzed in collaborations involving École Normale Supérieure and Max Planck Institute scholars.

Practical implementations and software tools

Implementations of Boltzmann machines and RBMs appear in libraries and frameworks maintained by organizations including Google, Facebook, Microsoft, and open-source communities centered around projects at GitHub and Apache Software Foundation. Toolkits and research codebases have been published by labs at University of Toronto, Carnegie Mellon University, and Stanford University and integrated into platforms such as TensorFlow, PyTorch, and Theano. Industrial adopters like IBM Research and Microsoft Research provided reference implementations and benchmarks, while community packages curated on GitHub and deployment examples in services by Amazon Web Services and Google Cloud Platform assist reproducibility. Ongoing development continues in academic groups at University of Cambridge, ETH Zurich, and Imperial College London.

Category:Neural networks