This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| MDL | |
|---|---|
| Name | MDL |
| Invented by | Jorma Rissanen |
| Introduced | 1978 |
| Field | Information theory; Statistics; Computer science |
| Related | Minimum description length principle; Kolmogorov complexity; Occam's razor |
MDL is an information-theoretic formalism for model selection and inductive inference that frames learning as data compression. It proposes choosing models that yield the shortest overall description of data plus model, unifying ideas from Jorma Rissanen, Andrey Kolmogorov, Ray Solomonoff, Claude Shannon, and Hermann Weyl. MDL has influenced work across machine learning, statistical inference, bioinformatics, signal processing, and philosophy of science.
MDL interprets model selection as a trade-off between model complexity and data fit by encoding both model and residuals into binary descriptions. It formalizes parsimony akin to Occam's razor and operationalizes Kolmogorov complexity through practical coding schemes inspired by Shannon coding and algorithmic probability developed by Ray Solomonoff. Variants split into two-level and full or two-part MDL formulations, linking to likelihood-based criteria such as the Akaike information criterion and the Bayesian information criterion while maintaining a distinct information-theoretic grounding influenced by Jorma Rissanen's original work.
MDL rests on minimizing the combined code-length L(model) + L(data | model). Practical methods approximate uncomputable ideals from Andrey Kolmogorov's theory using universal codes, stochastic complexity, and normalized maximum likelihood (NML). Key methods include two-part codes, prequential coding inspired by Peter Grünwald and Dawid's prequential principle, and Bayesian mixtures linking to Thomas Bayes and Bruno de Finetti's exchangeability. Connections to compression algorithms such as Lempel–Ziv illustrate operational encodings; connections to hypothesis testing and minimum message length (MML) relate to work by Chris Wallace and David Boulton. MDL incorporates model classes, parameter penalization, and code design, with practical approximations using Bayesian information criteria and penalized likelihood methods informed by Vapnik–Chervonenkis theory and structural risk minimization from Vladimir Vapnik.
MDL has been applied in diverse domains: model selection in linear regression and generalized linear models, structure learning for Bayesian networks, sequence prediction in natural language processing, motif discovery in genomics and bioinformatics, change-point detection in signal processing, image segmentation in computer vision, and time-series forecasting in econometrics. It informs algorithmic approaches in data compression tools and influences model comparison in empirical sciences such as astronomy and neuroscience. MDL-derived criteria have been used for selecting features in high-dimensional settings related to George E. P. Box's modeling ethos and for evaluating competing theories in philosophy of science.
Practical MDL implementations appear in statistical software for model selection, compression libraries, and specialized toolkits. Implementations leverage normalized maximum likelihood estimators and prequential codes in packages for R (programming language), Python (programming language), and tools integrated into MATLAB toolboxes. Compression-based modelers adapt algorithms from Lempel–Ziv families and integrate with Bayesian computation platforms such as Stan (software) and JAGS (program) for approximate code-length evaluation. Open-source projects and academic repositories provide implementations for sequence modeling, change-point detection, and network inference used by researchers at institutions like Massachusetts Institute of Technology, Stanford University, and University of Cambridge.
MDL is praised for a principled tie between compression and inference and its avoidance of explicit priors, but it faces criticisms and practical challenges. Critiques highlight sensitivity to code design, difficulties approximating Kolmogorov-optimal codes, and computational burdens linked to normalized maximum likelihood evaluation. Debates contrast MDL with Bayesian model averaging championed by Thomas Bayes-inspired approaches and with frequentist model selection criteria like Akaike information criterion proponents such as Hirotugu Akaike. Empirical comparisons across domains (e.g., genomics vs. econometrics) show mixed outcomes, and methodological work, including contributions from Peter Grünwald and Chris Wallace, addresses robustness, consistency, and applicability to high-dimensional problems encountered at research centers like Carnegie Mellon University and Princeton University.
The MDL concept traces to foundational work linking algorithmic information and induction by Andrey Kolmogorov and Ray Solomonoff in the 1960s, with Jorma Rissanen formalizing MDL in the late 1970s and elaborating it in seminal texts through the 1980s and 1990s. Subsequent development involved contributions from Peter Grünwald, Chris Wallace, David Wallace, Ian Hacking's philosophical context, and cross-pollination with Bayesian and frequentist traditions represented by Thomas Bayes and Hirotugu Akaike. MDL research advanced in machine learning communities at University of California, Berkeley, University of Toronto, and European centers such as University of Amsterdam and Ecole Polytechnique Federale de Lausanne, leading to modern connections with compression-based learning, information criteria, and contemporary work on universal codes and algorithmic statistics.
Category:Inference methods