belief propagation

belief propagation
Name	belief propagation
Type	algorithm
Field	Stanford University, University of California, Berkeley, Massachusetts Institute of Technology
Introduced	1980s
Developer	Judea Pearl, Richard Karp, David MacKay
Applications	ImageNet, Google, Facebook, Tesla, Inc.

belief propagation

Introduction

Belief propagation is an iterative message-passing algorithm for performing inference on graphical models. It was popularized by researchers at University of California, Berkeley and Massachusetts Institute of Technology and has ties to work by Judea Pearl, David MacKay, and contributors at Bell Labs and AT&T Research. The method operates on factor graphs, Bayesian networks and Markov random fields and is connected historically to research at Stanford University, Princeton University, Carnegie Mellon University, University of Cambridge, Oxford University, Harvard University, Yale University, Columbia University, University of Toronto, ETH Zurich, Max Planck Society, INRIA, Microsoft Research, IBM Research, and Google DeepMind.

Algorithmic Formulation

The algorithm represents a joint distribution via local potentials and iteratively updates beliefs using messages exchanged between variable and factor nodes, a paradigm refined in work led by Judea Pearl, Ross Shachter, Lauritzen and Spiegelhalter, and later formalized in coding theory by Robert G. Gallager, David J.C. MacKay, and collaborators at Bell Labs. Each update step uses products and marginalizations analogous to operations used in algorithms from Richard Karp and techniques appearing in Viterbi algorithm analyses. Implementations often borrow data structures and scheduling strategies from projects at Sun Microsystems and Oracle Corporation and from software libraries originating at Apache Software Foundation, NumPy, SciPy, TensorFlow, and PyTorch groups.

Extensions and Variants

Many extensions modify message computation or topology: loopy variants used in practice follow empirical work by researchers at AT&T Research and Bell Labs; expectation propagation links to methods from Harrison’s group and ideas explored at University College London; generalized belief propagation developed by Yedidia, Freeman, and Weiss builds on cluster variational techniques connected with research at Cornell University, California Institute of Technology, and University of Washington. Other variants include tree-reweighted belief propagation tied to work by Martin Wainwright and Michael I. Jordan and survey propagation inspired by studies at Los Alamos National Laboratory and Princeton University related to satisfiability problems investigated by groups at DIMACS and Bell Labs.

Convergence and Correctness

Convergence analysis has deep links to results by Judea Pearl, Martin Wainwright, Michael I. Jordan, Stuart Russell, and theoretical foundations echoing complexity characterizations from Richard Karp and Leslie Lamport. Exactness on trees follows from work dating to the 1980s credited to Judea Pearl and algorithms used in coding theory by Robert G. Gallager and David J.C. MacKay. For loopy graphs, results by Yedidia, Freeman, and Weiss and Martin Wainwright relate fixed points to variational principles akin to those studied at Princeton University and University of California, Berkeley. Rigorous bounds and counterexamples appear in literature from MIT Press authors and research groups at ETH Zurich, INRIA, Max Planck Society, and IBM Research.

Applications

Belief propagation has been applied widely across domains explored at institutions such as NASA, European Space Agency, National Institutes of Health, Centers for Disease Control and Prevention, World Health Organization, Samsung Electronics, NVIDIA, Intel Corporation, and Sony Corporation. Notable application areas include decoding of error-correcting codes pioneered at Bell Labs and used in standards managed by 3GPP and IEEE, stereo vision and image restoration advanced in research from Adobe Systems, Microsoft Research, Google Research, and Adobe Research, natural language processing influenced by groups at Google, Facebook AI Research, OpenAI, and DeepMind, and probabilistic reasoning in robotics developed at Carnegie Mellon University and Stanford University. Other deployments are in bioinformatics studies at Broad Institute and Wellcome Trust Sanger Institute, in statistical physics interactions studied at Los Alamos National Laboratory and CERN, and in econometric modeling influenced by work at London School of Economics and University of Chicago.

Practical Implementation Considerations

Practical implementations draw on software engineering practices from GitHub projects and continuous integration patterns used at Google, Facebook, Microsoft, and Netflix. Efficient implementations use sparse linear algebra libraries from Intel Math Kernel Library, distributed frameworks developed at Apache Hadoop and Apache Spark, and GPU acceleration strategies pioneered by NVIDIA and implemented in CUDA toolchains. Numerical stability, scheduling, damping, and parallelization strategies are often informed by benchmarking at Stanford DAWN, Berkeley RISELab, and industrial labs like DeepMind and Google Brain.

Category:Algorithms