This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| Ridge regression | |
|---|---|
| Name | Ridge regression |
| Type | Regularization technique |
| Introduced | 1970s |
| Key people | Arthur E. Hoerl, Robert W. Kennard |
| Related | Linear regression, Tikhonov regularization, Lasso (statistics) |
Ridge regression Ridge regression is a linear estimation method that adds a quadratic penalty to ordinary least squares to stabilize coefficient estimates in the presence of multicollinearity and ill-conditioning. It was popularized by Arthur E. Hoerl and Robert W. Kennard and is mathematically equivalent to early work by Andrey Tikhonov in inverse problems; its development intersects with methods used at Bell Labs, AT&T, and in applied work at Bell Telephone Laboratories and General Electric.
Ridge regression emerged as a response to unstable parameter estimates when fitting Linear regression models to data with nearly collinear predictors or high-dimensional design matrices encountered in projects at General Electric and in econometric studies conducted at National Bureau of Economic Research. Its conceptual roots trace to Tikhonov regularization used in geophysics and signal processing at institutions such as Leningrad State University and later formalized in statistics in work by Hoerl and Kennard at Ohio State University. Ridge connects historically and practically to penalized methods such as Lasso (statistics) developed by researchers at AT&T Bell Laboratories and to Bayesian formulations related to priors studied by scholars at Harvard University and Stanford University.
Let X denote an n-by-p design matrix and y an n-vector of responses as in classical setups at University of Chicago econometrics seminars. Ridge estimates minimize the penalized sum of squares popularized in papers from Journal of the Royal Statistical Society and meetings at Institute of Mathematical Statistics: argmin_beta ||y - X beta||_2^2 + lambda ||beta||_2^2, where lambda >= 0 is the penalty parameter introduced in early expositions at General Electric Research Laboratory. Equivalently, the ridge solution can be written (X^T X + lambda I_p)^{-1} X^T y, a shift used in numerical linear algebra discussions at Massachusetts Institute of Technology and in textbooks influenced by authors from Princeton University and Cambridge University. The connection to Tikhonov regularization is explicit in works presented at International Congress of Mathematicians sessions on inverse problems.
Computational approaches for ridge involve solving linear systems encountered in applied projects at Bell Labs and in software developed at SAS Institute and The R Project for Statistical Computing. Direct formulas exploit matrix decompositions such as the eigendecomposition and the singular value decomposition used in computational treatments at Stanford University and ETH Zurich. For large p, iterative solvers and conjugate gradient methods described in computational mathematics courses at Oxford University and École Polytechnique Fédérale de Lausanne are common. Implementation details appear in software packages originating from CRAN contributions and in numerical libraries from Netlib and Numerical Recipes authors affiliated with Cambridge University Press.
Ridge yields biased but lower-variance estimators, a bias–variance tradeoff discussed in expository lectures at Columbia University and formalized in asymptotic analyses presented at Institute of Mathematical Statistics conferences. Its admissibility and minimaxity under quadratic loss relate to results by researchers connected to University of California, Berkeley and to the James–Stein estimator studied at University of Chicago. Ridge shrinks coefficients toward zero isotropically, unlike selection methods reviewed in monographs from Oxford University Press and in comparative studies at National Institutes of Health workshops. Theoretical guarantees for prediction risk and conditioning improvements are treated in work by teams at Carnegie Mellon University and University of Toronto.
Choosing lambda is critical and studied across methodologies promoted at International Biometric Society meetings and in applied work at Centers for Disease Control and Prevention. Common approaches include generalized cross-validation (GCV) championed in studies from University of Wisconsin–Madison, k-fold cross-validation used in computational experiments at Stanford University, and information criteria adaptations influenced by research at Yale University and Princeton University. Bayesian interpretations tie lambda selection to prior hyperparameter tuning in hierarchical models explored at Harvard University and University of Michigan.
Ridge inspired a family of penalized estimators including Lasso (statistics), elastic net developed by researchers with affiliations to University of Toronto and Harvard Medical School, and generalized ridge regression introduced in statistical theory seminars at Pennsylvania State University. Kernel ridge regression connects to kernel methods advanced at Bell Labs and in machine learning groups at Carnegie Mellon University and University of California, Berkeley. Bayesian ridge places Gaussian priors on coefficients as in work from Columbia University and University of Washington, while generalized linear model extensions were developed in collaborations involving Johns Hopkins University and Imperial College London.
Ridge has been applied in econometrics projects at National Bureau of Economic Research and in genomics studies at Broad Institute, where high-dimensional predictors and collinearity are pervasive. Signal processing applications tie to early uses at Bell Labs and inverse-problem analyses at Leningrad State University. In chemometrics, practitioners from Shell Oil Company and teams at Shell research divisions used ridge-like penalties. Machine learning benchmarks performed by groups at Google and Microsoft Research compare ridge to modern regularizers, and climate modeling efforts at National Aeronautics and Space Administration and NOAA use ridge-based stabilization in parameter estimation.
Category:Statistical estimation