Some Ridge Biasing Parameter for Linear Regression Model and Their Performances on Kibria-Lukman Estimator

Multiple Linear Regression (MLR) extends the simple linear regression framework by incorporating two or more explanatory variables into a single predictive model for a continuous response variable. The general form of the model is expressed as follows:

    (1)

For i =

are regression coefficients,

 are the independent variables,

is the dependent variable and

is the stochastic error term. In matrix form, the M equations can be written as:

Where y denotes an n × 1 vector of observed response, β represents a p × 1 vector of unknown regression coefficients, X is an n × p matrix of observed explanatory variables and e is an n × 1 vector of random error terms assumed to follow a multivariate normal distribution with mean vector 0 and covariance matrix σ²I_n, where I_n is an identity matrix of order n. The Ordinary Least Square (OLS) estimator of β is therefore expressed as:

The covariance matrix of  β

The Ridge regression (RR), originally introduced in 1970 by Hoerl and Kennard, was developed to address the issue of multicollinearity commonly encountered in engineering and other empirical data analyses. Their pioneering study revealed that the introduction of a positive ridge parameter ???? leads to a ridge regression estimator whose Mean Squared Error (MSE) is lower than the variance of the Ordinary Least Squares (OLS) estimator, thereby achieving greater estimation efficiency through an optimal bias–variance trade-off. Consequently, the ridge regression estimator (RRE) is defined as follow:

Where M= [I_p+kZ-1]-1

1.2 The Kibria Lukman Estimator

The newly formulated one-parameter estimator is obtained by optimizing the following objective function, designed to balance bias and variance in the estimation process:

Minimization of the objective function with respect to β  leads to the corresponding normal equations.

In this formulation, ???? represents a nonnegative constant. Solving the preceding equation with respect to ???? produces the explicit form of the proposed estimator as:

1. Statistical Methodology

2.1 Canonical Form

The canonical form of the model is:

Where A = XP and α= P’β. Here, P  is an orthogonal matrix such that

The ridge estimator (RE) of α is:

Where 

Thus the MSE of the propose estimator can be written as:

Finally, the MSE of Kibria-Lukman estimator after using the above stated definitions can be written as:

Differentiating Pk  with respect to k gives and setting (∂p(k)∂k)=0 , we obtain

The optimal value of k in (16) depends on the unknown parameter σ2 and α2 These two estimators are replaced with unbiased estimate.

2.2 Biasing Parameters

In accordance with the methodology introduced by Hoerl, Kennard, and Baldwin, the harmonic mean formulation corresponding to Equation (16) is defined as:

As proposed by Özkale and Kaciranlar, the minimum form of equation (16) is expressed as:

Where the λi are eigenvalues of the matrix X'X , αi  is the i^th element of α , and           

We next examine the existing methods proposed in the literature for determining the value of k. Hoerl and Kennard (1970) recommended estimating k as (here denoted by kHK

Here, αmax2

Hoerl et al. (1975) defined the ridge parameter ???? (denoted here by kHKB

Lawless and Wang (1976) derived biasing parameter k to be (denoted here by kLW

Hocking, Speed and Lynn (1976) also derived shrinkage parameter k to be (denoted here by kHSL

Kibria (2003) suggested alternative biasing estimators for k derived from the geometric mean (GM) and median of  σ2/αi2 These estimators are expressed as follows:

Based on modification of kHK, Khalaf and Shukur (2005) suggested k to be (denoted by  kKS

Where λmax is the maximum eigen value of the matrix X'X . Building on the work of Kibria (2003) and Khalaf and Shukur (2005), Alkhamisi, Khalaf, and Shukur (2006) proposed the following three estimators for k karith

Drawing upon the geometric mean and square-root methods proposed by Khalaf and Shukur (2005), Kibria (2003), and Alkhamisi et al. (2006), Muniz and Kibria (2009) developed seven new estimators for the ridge parameter k:

Following the square-root transformation methodology of Alkhamisi and Shukur (2008), Muniz et al. (2012) introduced five new estimators for the ridge parameter k:

Where,
Khalaf (2012), based on modification of kHK, proposed k to be (denoted by kGKK

Where λmax  and λmin  are the maximum and minimum eigenvalues of matrix X'X  respectively.

Nomura (1988) proposed estimating the ridge parameter k as (denoted by kHMO

3 The Proposed Biasing Estimator

Following the modification of Khalaf (2012), a new biasing parameter was proposed and defines as:

Following the work of ozkale and kaciranlar (2007), the maximum version and the median of (16) is proposed and define as:

Where λmax  and λmin are the maximum and minimum eigenvalues of  X'X respectively.

4. Simulation Study

The primary objective of this study is to evaluate and compare the performance of various ridge biasing parameter estimators, with the aim of recommending efficient and reliable options for practical applications. Since a purely theoretical comparison among these estimators is not feasible, a simulation study was conducted using R software. The design of the simulation experiment was structured around factors that are expected to influence the statistical properties of the estimators under consideration, as well as the evaluation criteria employed to assess their performance. Given that the degree of multicollinearity among the explanatory variables (X’s) plays a critical role in the behavior of ridge-type estimators, we adopted the data generation approach proposed by Kibria and Lukman (2020), Oladapo et al (2022,2023 and 2024), Idowu et al (2022 and 2023) and Owolabi et al (2022) where the explanatory variables were simulated using the following relationship:

Where z_ij represent independent standard normal pseudo-random numbers, and let γ denote the correlation between any two explanatory variables X, with values γ = 0.80, 0.90, 0.95, 0.99 for p = 5. These variables are standardized such that X’X and X’y are expressed in correlation forms. The n observations of y are generated according to the following equation:

Where the e_i ~NIID (0, σ²). And β'β=1

RESULT

This section reports the findings of the Monte Carlo simulation, focusing on the comparative performance of different biasing parameters against the Ordinary Least Squares (OLS) estimator in terms of Mean Squared Error (MSE). The key outcomes are illustrated through graphical analyses, providing a visual summary of estimator efficiency. Furthermore, detailed quantitative results for the five most efficient biasing parameters are presented in Tables 1 and 2, respectively.

Table 1: MSE Comparison With Various Biasing Parameter When n=50

Table 2: MSE Comparison With Various Biasing Parameter When   n=100

5.1 Performances with Respect To Sigma (σ)

Tables 1 and 2 present the Mean Squared Error (MSE) of the selected ridge biasing parameters as a function of σ, for sample sizes n=50,100 and correlation levels γ=0.8,0.9,0.95. The findings reveal that the MSE generally increases with higher values of σ. notably, all ridge-type estimators exhibit smaller MSEs compared to the Ordinary Least Squares (OLS) estimator, indicating improved estimation efficiency. Specifically, when σ=3, the Ridge_kgk estimator demonstrates superior performance relative to other biasing parameters in terms of lower MSE. A similar pattern is observed at σ=5 and σ=10 where Ridge_kgk consistently outperforms its counterparts. For clarity, Figure 1 illustrates the behavior of the estimators as a function of σ for γ=0.8 and n=50.

Figure 1: Performance as a function of sigma (σ) when n = 50 and γ = 0.8

5.2 Performance with Respect To The Correlation Coefficient (γ)

The Mean Squared Errors (MSEs) of the selected estimators were further examined as a function of the correlation coefficient (γ) for given values of n, σ, and p. To enhance interpretability, the performance of the biasing parameters as a function of γ is depicted in Figure 2. The findings reveal that an increase in the correlation among explanatory variables leads to a corresponding rise in the MSE of ridge-type estimators. Nevertheless, all ridge estimators maintain smaller MSEs compared to the Ordinary Least Squares (OLS) estimator, confirming their efficiency in handling multicollinearity. For relatively low correlation levels (e.g., γ=0.8), the Ridge_kgk estimator exhibits superior performance with the smallest MSE. A similar dominance of Ridge_kgk is observed for γ=0.9 and γ=0.95, based on the minimum MSE criterion. Furthermore, even at a high correlation level (γ=0.99), Ridge_kgk continues to outperform other biasing parameters, demonstrating its robustness under severe multicollinearity.

Figure 2: Performance as a function of correlation coefficient (γ) when n=50 and σ =3

5.3 Performances with Respect To Sample Size (n)

The Mean Squared Errors (MSEs) of the selected ridge biasing parameters were assessed as a function of the sample size (n) for fixed values of γ=0.8,0.9,0.95, and 0.99, with p=5 and σ=3,5, and 10. The results indicate a clear inverse relationship between sample size and MSE, demonstrating that estimator efficiency improves as n increases. Across all simulation design, the ridge-type estimators consistently outperformed the Ordinary Least Squares (OLS) estimator, achieving notably smaller MSEs. For smaller sample sizes (e.g., n=50), the Ridge_kgk estimator exhibited superior performance relative to other biasing parameters. Similarly, for larger sample sizes (e.g., n=100), Ridge_kgk maintained its dominance, yielding the minimum MSE among the compared estimators.

Figure 3: Performance as a function of sample size (n) when γ = 0.8 and σ = 3