Ridge regression is a method of estimating the coefficients of multiple-regression models in scenarios where the independent variables are highly correlated.[1] It has been used in many fields including econometrics, chemistry, and engineering.[2] Also known as Tikhonov regularization, named for Andrey Tikhonov, it is a method of regularization of ill-posed problems.[a] It is particularly useful to mitigate the problem of multicollinearity in linear regression, which commonly occurs in models with large numbers of parameters.[3] In general, the method provides improved efficiency in parameter estimation problems in exchange for a tolerable amount of bias (see bias–variance tradeoff).[4]
The theory was first introduced by Hoerl and Kennard in 1970 in their Technometrics papers "Ridge regressions: biased estimation of nonorthogonal problems" and "Ridge regressions: applications in nonorthogonal problems".[5][6][1] This was the result of ten years of research into the field of ridge analysis.[7]
Ridge regression was developed as a possible solution to the imprecision of least square estimators when linear regression models have some multicollinear (highly correlated) independent variables—by creating a ridge regression estimator (RR). This provides a more precise ridge parameters estimate, as its variance and mean square estimator are often smaller than the least square estimators previously derived.[8][2]
^ abHilt, Donald E.; Seegrist, Donald W. (1977). Ridge, a computer program for calculating ridge regression estimates. doi:10.5962/bhl.title.68934.[page needed]
^ abGruber, Marvin (1998). Improving Efficiency by Shrinkage: The James--Stein and Ridge Regression Estimators. CRC Press. p. 2. ISBN 978-0-8247-0156-7.
^Kennedy, Peter (2003). A Guide to Econometrics (Fifth ed.). Cambridge: The MIT Press. pp. 205–206. ISBN 0-262-61183-X.
^Gruber, Marvin (1998). Improving Efficiency by Shrinkage: The James–Stein and Ridge Regression Estimators. Boca Raton: CRC Press. pp. 7–15. ISBN 0-8247-0156-9.
^Hoerl, Arthur E.; Kennard, Robert W. (1970). "Ridge Regression: Biased Estimation for Nonorthogonal Problems". Technometrics. 12 (1): 55–67. doi:10.2307/1267351. JSTOR 1267351.
^Hoerl, Arthur E.; Kennard, Robert W. (1970). "Ridge Regression: Applications to Nonorthogonal Problems". Technometrics. 12 (1): 69–82. doi:10.2307/1267352. JSTOR 1267352.
^Beck, James Vere; Arnold, Kenneth J. (1977). Parameter Estimation in Engineering and Science. James Beck. p. 287. ISBN 978-0-471-06118-2.
^Jolliffe, I. T. (2006). Principal Component Analysis. Springer Science & Business Media. p. 178. ISBN 978-0-387-22440-4.
Cite error: There are <ref group=lower-alpha> tags or {{efn}} templates on this page, but the references will not show without a {{reflist|group=lower-alpha}} template or {{notelist}} template (see the help page).
Ridgeregression is a method of estimating the coefficients of multiple-regression models in scenarios where the independent variables are highly correlated...
linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where...
_{0}=c\mathbf {I} } is called ridgeregression. A similar analysis can be performed for the general case of the multivariate regression and part of this provides...
process prior is known as Gaussian process regression, or kriging; extending Gaussian process regression to multiple target variables is known as cokriging...
least-angle regression algorithm. An important difference between lasso regression and Tikhonov regularization is that lasso regression forces more entries...
is an advantage of Lasso over ridgeregression, as driving parameters to zero deselects the features from the regression. Thus, Lasso automatically selects...
logistic regression models, the elastic net is a regularized regression method that linearly combines the L1 and L2 penalties of the lasso and ridge methods...
(e.g., nonparametric regression). Regression analysis is primarily used for two conceptually distinct purposes. First, regression analysis is widely used...
Partial least squares regression (PLS regression) is a statistical method that bears some relation to principal components regression; instead of finding...
Poisson regression is a generalized linear model form of regression analysis used to model count data and contingency tables. Poisson regression assumes...
^{\mathsf {T}}\mathbf {y} .} Optimal instruments regression is an extension of classical IV regression to the situation where E[εi | zi] = 0. Total least...
especially in the case of a simple linear regression, in which there is a single regressor on the right side of the regression equation. The OLS estimator is consistent...
Quantile regression is a type of regression analysis used in statistics and econometrics. Whereas the method of least squares estimates the conditional...
Local regression or local polynomial regression, also known as moving regression, is a generalization of the moving average and polynomial regression. Its...
In statistics, polynomial regression is a form of regression analysis in which the relationship between the independent variable x and the dependent variable...
In statistics, specifically regression analysis, a binary regression estimates a relationship between one or more explanatory variables and a single output...
In statistics, nonlinear regression is a form of regression analysis in which observational data are modeled by a function which is a nonlinear combination...
In statistics, simple linear regression (SLR) is a linear regression model with a single explanatory variable. That is, it concerns two-dimensional sample...
(WLS), also known as weighted linear regression, is a generalization of ordinary least squares and linear regression in which knowledge of the unequal variance...
In statistics, ordinal regression, also called ordinal classification, is a type of regression analysis used for predicting an ordinal variable, i.e....
principal components analysis (PCA), canonical correlation analysis, ridgeregression, spectral clustering, linear adaptive filters and many others. Most...
In statistics and numerical analysis, isotonic regression or monotonic regression is the technique of fitting a free-form line to a sequence of observations...
Density Based Empirical Likelihood Ratio tests In regression analysis, more specifically regression validation, the following topics relate to goodness...
In statistics, multinomial logistic regression is a classification method that generalizes logistic regression to multiclass problems, i.e. with more than...
In statistics, binomial regression is a regression analysis technique in which the response (often referred to as Y) has a binomial distribution: it is...
Nonparametric regression is a category of regression analysis in which the predictor does not take a predetermined form but is constructed according to...
used for estimating the unknown regression coefficients in a standard linear regression model. In PCR, instead of regressing the dependent variable on the...
Global Information