Improved ridge estimation approach to address overfitting: Simulation and financial data analysis

1. Introduction

The multiple linear regression model (MLRM) is one of the most widely used statistical modeling tools, owing to its mathematical simplicity and effectiveness in prediction and statistical inference (Khan et al., 2024a; Lipovetsky and Conklin, 2005). The ordinary least squares (OLS) method is typically preferred for parameter estimation in regression analysis, provided that certain classical assumptions are met. However, in many real-world applications, these assumptions, particularly the requirement of negligible multicollinearity among predictors, are often violated (Schroeder et al., 1990). Condition number and variance inflation factors (VIF) are widely used to detect the degree of multicollinearity in a data. Multicollinearity can lead to unstable and unreliable OLS estimates; therefore, it must be appropriately addressed. To this end, various alternative estimation techniques have been proposed in the literature. These include ridge regression (RR) (Hoerl and Kennard, 1970a, 1970b), principal component regression (Massy, 1965), elastic net regression (Zou and Trevor, 2005), raised regression (Garcia et al., 2011; Salmerón‐Gómez et al., 2025), and residualization methods (Garcia et al., 2020), among others. However, RR is very popular amongst all the alternative methods for its computational convenience, attractive mathematical properties and interpretability (Belsley, 1991). It treats multicollinearity in a principled, stable, and information-preserving way, without deleting or changing the predictors, while improving prediction accuracy. The RR is a precise estimation method as it allows all the considered covariates to be included in the regression model with shrunken regression coefficients, at the cost of some bias in the multiple regression model (Dar and Chand, 2024).

Consider the standard MLRM, as represented in Eq. (1)

Where y is a column vector of observations on response variable, X is fixed matrix of predictors, β is a column vector of unknown regression coefficients and ϵ is a column vector of random errors such that $E\left(\epsilon \right)=0$ and $E\left(\epsilon {\epsilon }^{\prime }\right)= {\sigma }^2{I}_n$. Where, prime denotes the transpose of the vector while $I_n$ is an identity matrix of order n. The OLS estimates of β can be given as represented in Eq. (2):

Where $\stackrel{}{\text{β}}^$ is an unbiased estimator of β, its accuracy largely depends on the characteristics of $X\text{'}X$ matrix. In the presence of near-perfect collinearity among two or more predictors, the variance of the regression coefficients can increases substantially, leading to unstable parameter estimates that cannot be reliably interpreted. From a mathematical perspective, near-collinearity renders the $X\text{'}X$ matrix ill-conditioned, meaning that its determinant approaches zero in case of standardized data. Consequently, attempts to compute its inverse result in numerical instability and uncertain coefficient estimates. Exact collinearity arises when at least one predictor is an exact linear combination of other predictors, in which case $X\text{'}X$ is not of full rank, the determinant of $X\text{'}X$ equals zero, and the matrix is non-invertible. To address the problem of ill-conditioning (Hoerl and Kennard, 1970a, 1970b) proposed the ridge regression (RR) estimator as represented in Eq. (3):

where k is showing ridge or shrinkage penalty and I is an identity matrix of the same order as $X\text{'}X$. The ridge regression is a biased estimation, i.e. $E\left({\text{β}}_k\right)\ne \text{β}$, contrary to unbiased OLS estimation. Ridge regression (RR) seeks to improve upon ordinary least squares by trading a small amount of bias for a significant reduction in variance. This is achieved through a ridge penalty parameter, k. The optimal value of k is selected to minimize the mean squared error (MSE), which represents the expected squared difference between the estimate and the true parameter. Minimizing the MSE automatically finds the ideal balance between bias and variance, as represented in Eq. (4)

This embodies the core bias-variance trade-off: accepting a small amount of bias is beneficial if it significantly reduces variance, leading to a lower overall MSE.

The selection of an optimal ridge parameter k is a well-studied challenge. A consensus in the literature confirms that no single ridge estimator performs optimally under all conditions; rather, its efficacy is contingent upon key data characteristics such as the degree of multicollinearity, error variance, sample size, and the number of predictors (McDonald, 2009). Consequently, the process of selecting k remains both a science, reliant on analytical methods, and an art, requiring diagnostic interpretation. This has led to the proposal of a multitude of estimation methods, including:(Hoerl et al., 1975; Hoerl and Kennard, 1970b; Jegede et al., 2022; Khalaf et al., 2013; Khalaf and Shukur, 2005; Kibria, 2003; Mcdonald et al., 1975; Suhail et al., 2020; Wichern and Gilbert, 1978). Lipovetsky and Conklin, 2005 observed that the selection of the ridge penalty parameter is constrained by its inverse relationship with the model’s goodness of fit. To address this limitation and to enhance the orthogonality between the predicted values and the residuals, they proposed a two-parameter ridge (TPR) estimator, represented as Eq. (5):

where, in Eq. (6)

Subsequently, numerous researchers have proposed improvements to the two-parameter ridge regression model (Akhtar and Alharthi, 2025; Alharthi and Akhtar, 2025; Asar and Erişoğlu, 2016; Dorugade, 2019; Khan et al., 2024b, 2023; Khan and Alharthi, 2025; Lipovetsky, 2006; Adewale F. Lukman et al., 2019; A. F. Lukman et al., 2019; Owolabi et al., 2022; Ozkale and Selahattin, 2007; Wu and Yang, 2013; Yang and Chang, 2010; Yasin et al., 2021). However, while existing ridge estimators often perform well in specific scenarios, they frequently lack robustness and adaptability across diverse datasets. To address this limitation, this study proposes three improved ridge estimators (IREs), which utilizes an auto-adjusted ridge penalty. The rationale for the proposed estimator is explained in detail in Section to follow. Its performance is evaluated across a range of scenarios via extensive Monte Carlo simulations, using the minimum MSE as the criterion. The applicability of the proposed estimator is also demonstrated using two real-world datasets.

This article proceeds as follows: Section 2 presents the statistical methodology, our proposed estimators and a review of existing ridge estimators. Section 3 describes the simulation design, and Section 4 discusses the results. The application to real dataset is evaluated in Section 5, and concluding remarks are offered in Section 6.

2. Statistical Methodology

The canonical form of model (1) is rewritten as represented in Eq. (7)

where $\text{¥}=XQ,\psi =Q\text{‘β},andQ\text{‘}Q=I_p$. The matrix Q contains eigen vectors of $\left(\text{X’X}\right)$. Moreover, $\Lambda =Q\text{‘}X\text{‘}XQ$ is a matrix of order p and contains eigen values of matrix $\text{X’X}$ on its diagonals.

The regression coefficients defined in Eq. (2-4) are expressed in canonical form respectively in Eqs. (8-10):

3. Simulation Study

This section details the data generation process for the empirical evaluation. Data was simulated by varying key factors to assess estimator performance under a range of conditions. These factors include:

Pair-wise correlation between predictors (ρ): 0.90, 0.95, 0.99, 0.999

Error variance (σ²): 1, 5, 10

Sample size (n): 15, 30, 50,100, 200

Number of predictors (p): 3, 5, 7, 10,15

The predictors were generated using the method described by (Akhtar and Alharthi, 2025; Dar and Chand, 2024; Mcdonald et al., 1975; Suhail et al., 2021) are represented in Eqs. (24-26)

Where, w_ji is pseudo random numbers generated from standard normal distribution.

The response variable is generated as:

The coefficients ${\psi }_i$ ​ are calculated based on the most favorable (MF) direction, following the methodology of (Halawa and El Bassiouni, 2000; Mcdonald et al., 1975; Newhouse JP and Oman SD, 1971). The intercept term, ${\psi }_o$,was set to zero without loss of generality. The random error term, ${\epsilon }_i$ ​, was generated from a normal distribution with a mean of 0 and variance ${\sigma }^2$. The simulations were replicated 5,000 times, and the estimated mean squared error (EMSE) was calculated as follows

All calculations were performed using the R programming language. The EMSEs for all estimators are summarized in (Tables 1-13 )

4. Simulation results discussion

Based on a comprehensive simulation study, the following key findings were established:

• i. Superior performance of the IRE estimator: The proposed improved ridge estimator1 (IRE1) demonstrated a consistent and significant advantage over all existing alternatives, achieving the lowest MSE across every simulated condition as evident from (Tables 1-13). This superiority was robust to variations in sample size, error variance, and the number of predictors. In contrast, the performance of the OLS estimator degraded severely under multicollinearity, a result consistent with established literature (Akhtar and Alharthi, 2025; Khan et al., 2024a; Kibria, 2003).

• ii. Robustness to multicollinearity: The simulations confirm that increased multicollinearity elevates the MSE for both OLS and traditional ridge estimators (Tables 1-13), as documented in (Suhail et al., 2020; Yasin et al., 2021). Notably, the IRE1 exhibits a unique and robust inverse relationship with multicollinearity as evident from (Table 1-13) the MSE of IRE1 decreases as correlation among predictors intensifies. This adaptive behavior is a direct result of IRE1’s data-driven mechanism for dynamically tuning the ridge penalty.

• iii. Stability under noisy conditions: A positive relationship was observed between error variance and the MSE of all estimators. However, IRE1’s MSE remained exceptionally stable and low compared to the more pronounced sensitivity of OLS and other ridge estimators, underscoring its robustness in high-noise environments.

• iv. Effect of model dimension: Introducing additional predictor variables in multicollinear settings increased the MSE for all estimators. The OLS estimator was most adversely affected, exhibiting the most rapid degradation in performance. The ridge-type estimators, including IRE1, proved more resilient to increases in model dimensions, a finding that aligns with prior research (Ali et al., 2021; Majid et al., 2022; Suhail et al., 2021; Yasin et al., 2021).

• v. Consistency across sample sizes: As predicted by asymptotic theory, larger sample sizes reduced the MSE for all estimators. Crucially, ridge-type estimators and IRE1 maintained their performance advantage over OLS regardless of sample size, confirming their effectiveness even in large-sample settings (Kibria, 2003).

5. Applications

To demonstrate the real-world applicability of our proposed estimators and methodology, we employ a well-known dataset in econometrics: the Longley dataset (Gujarati and Porter, 2009). Comprising 16 annual observations from 1947 to 1962, it was originally used to test the numerical accuracy of least-squares algorithms, as the high correlations between its variables posed a significant computational challenge for early computers. Beyond its original purpose, the dataset’s enduring legacy is its perfect illustration of multicollinearity and its consequences. The data set contains five predictors.

Considering our regression model for Longley set as represented in Eq. (27):

In the above linear regression model (25), following variable definitions are used, y (number of people employed, in millions), x₁ (GNP implicit price deflator), x₂ (GNP, millions of dollars), x₃ (number of people unemployed, in millions), x₄ (number of people in the armed forces) and x₅ (noninstitutionalized population over 14 years of age).

Diagnostic tests confirm severe multicollinearity among the predictors. The eigenvalues of the S’S matrix are 3.610, 1.175, 0.199, 0.015, and 0.001, with the first eigenvalue accounting for over 90% of the total variation, that results in a condition number of 3,786. Furthermore, the variance inflation factor (VIF) for most predictors significantly exceeds the threshold of 10, with values of 130.829 (s1), 639.050 (s2), 10.787 (s3), 2.506 (s4), and 339.012 (s5). These findings, supported by the pair-wise correlations illustrated in (Fig. 1), unequivocally indicate a severe multicollinearity problem.

Close

Fig. 1.

Pair-wise correlation for Longley data.

Export to PPT

The EMSE and regression coefficients for all estimators are presented in (Table 14). The results show that all ridge-type estimators substantially outperform OLS, consistent with the findings of (Kibria, 2003; Suhail et al., 2020). Among them, the proposed IRE1 estimator achieves the lowest EMSE. Additionally, as noted by (Gujarati and Porter, 2009), multicollinearity can cause OLS coefficient estimates to have counter-intuitive signs; this is observed in our results for the coefficients ${\psi }_1$, ${\psi }_3$ and ${\psi }_4$.

6. Conclusions

In this article, we proposed IRE₁, IRE₂, and IRE₃ aimed at improving existing ridge estimation techniques for more efficient estimation of regression coefficients in the presence of the common issue of multicollinearity. Given that ridge regression involves a bias–variance tradeoff, the selection of an optimal ridge penalty is crucial for achieving reliable estimates. To address this, the proposed estimators adaptively adjusts its penalty parameters based on key characteristics of the data, such as the degree of multicollinearity and the number of predictors. Through extensive simulation studies and real-world data applications, the IRE₁ & IRE₃ demonstrated significant improvements in performance over conventional ridge estimators, particularly in scenarios with high multicollinearity. These findings suggest that the dynamic nature of our estimator offers both flexibility and robustness in practical modeling contexts. As a potential avenue for future research, we recommend investigating the performance of the IRE₁ & IRE₃ in regression models that simultaneously exhibit multicollinearity and heteroscedasticity, which are often encountered in real-world data analysis.