Markov chain

Markov chain
Joxemai4 · CC BY-SA 3.0 · source
Name	Markov chain
Field	Probability theory
Introduced	1906
Named after	Andrey Markov

Markov chain is a mathematical model describing a sequence of random states where the probability of each state depends only on the immediately preceding state. It originated in the work of Andrey Markov and later influenced research by Andrey Kolmogorov, Andrey Nikolaevich Kolmogorov, Aleksandr Lyapunov, Paul Lévy, and practitioners in Princeton University and Cambridge University. The concept underpins techniques in Harvard University statistics, Bell Labs information theory, IBM computing, and applications at NASA and Google.

Definition and basic properties

A Markov chain is defined on a state space with a transition mechanism satisfying the Markov property introduced by Andrey Markov and formalized by Andrey Kolmogorov in his work on stochastic processes. Core properties include irreducibility, recurrence, transience, and periodicity studied by researchers at Moscow State University, University of Oxford, Massachusetts Institute of Technology, Stanford University, and ETH Zurich. In finite chains, classification into absorbing and ergodic types was developed alongside applications in Bell Labs and AT&T research labs. The Chapman–Kolmogorov equations, named after Sydney Chapman and Andrey Kolmogorov, describe compositional transition probabilities, while stationarity concepts relate to equilibrium measures used by John von Neumann and Norbert Wiener.

Examples and classifications

Simple examples include random walks analyzed by George Pólya and Gábor Szegő on lattices considered at University of Göttingen and Princeton University, birth–death processes studied by Agner Krarup Erlang and F. H. C. Crick-adjacent fields, and finite-state Markov chains used in models by Alan Turing and Claude Shannon. Chains are classified as discrete-time versus continuous-time, finite versus countable versus uncountable state space, reversible explored by Ludwig Boltzmann and J. W. Gibbs influences, and periodic versus aperiodic examples used by Erdős–Rényi network models at Hungarian Academy of Sciences and Institut Henri Poincaré.

Transition matrices and long-term behavior

Transition matrices, studied in linear algebra contexts by Issai Schur and John von Neumann, determine single-step dynamics; eigenvalues and eigenvectors relate to steady-state distributions as in work by Perron and Frobenius. The Perron–Frobenius theorem, used in analyses at University of Cambridge and University of California, Berkeley, yields existence of dominant eigenvectors for positive matrices; mixing times and spectral gaps were investigated by Persi Diaconis and David Aldous at Harvard University and University of California, San Diego. Ergodic theorems connecting time averages to ensemble averages were proved in frameworks developed by George Dantzig and Andrey Kolmogorov, influencing statistical mechanics at Los Alamos National Laboratory and econometric models at University of Chicago.

Continuous-time Markov chains

Continuous-time processes were formalized by Andrey Kolmogorov and further developed by Kiyosi Itô and Henry McKean with generator matrices and infinitesimal generators analogous to operators studied at Institute for Advanced Study and Princeton University. Birth–death and queueing models trace to Agner Krarup Erlang and found extensive use at AT&T and Bell Labs; the Chapman–Kolmogorov forward and backward equations are analogous to master equations in studies by Enrico Fermi and Richard Feynman at Los Alamos National Laboratory and California Institute of Technology.

Applications and algorithms

Markov-type models power algorithms such as Markov chain Monte Carlo implemented in work by Nicholas Metropolis, W.K. Hastings, and Radford Neal for Bayesian computation at Los Alamos National Laboratory, University of Toronto, and University of Cambridge. Applications span speech recognition research at Bell Labs and AT&T by Fred Jelinek and L. R. Rabiner, hidden Markov models used in genomics at Cold Spring Harbor Laboratory and Wellcome Trust Sanger Institute, PageRank developed at Stanford University by Larry Page and Sergey Brin for Google, and reliability models in civil engineering projects associated with MIT and ETH Zurich. Algorithms for mixing, sampling, and inference have been refined by groups at Microsoft Research, Facebook, Amazon, and Alphabet Inc..

Theoretical results and extensions

Extensions include Markov processes with continuous state spaces analyzed by Kiyosi Itô and Shizuo Kakutani, interacting particle systems studied by Thomas Liggett at University of Illinois at Urbana–Champaign, and coupling methods developed by Persi Diaconis and David Aldous. Large deviations theory connecting to chains was advanced by S. R. S. Varadhan and Frank Spitzer; martingale techniques from Jean-Pierre Kahane and Joseph Doob provide foundational tools. Modern research at Institut de Mathématiques de Jussieu, Courant Institute, Fields Institute, and Perimeter Institute explores non-reversible dynamics, quasi-stationary distributions, and connections to optimization methods used at DeepMind and OpenAI.

Category:Stochastic processes