Abstract

Multicollinearity is a significant issue in multiple linear regression that occurs when two or more independent variables are highly correlated. This correlation undermines the reliability and stability of regression estimates, making it challenging to isolate and interpret the individual effect of each predictor variable. In the presence of multicollinearity, traditional estimation methods like Ordinary Least Squares (OLS) become less effective, often resulting in inflated standard errors and less reliable statistical inference. When multicollinearity exists, biased estimation techniques such as Ridge regression, the Liu estimator, and Principal Component-based estimators are frequently used. These estimators provide more stable and interpretable results when independent variables are correlated. Other estimators that are a combination of existing estimators have been formed. These include the Principal Component Ridge (PCRE) and Principal Component Liu (PCLIU) estimators. They further mitigate the adverse effects of multicollinearity. This study evaluates the performance of PCRE and PCLIU estimators under varying degrees of multicollinearity, sample sizes, and error variances. Seven distinct forms of the biasing parameter k, along with their generalized versions, are examined in this analysis. Originally introduced in 2019 by Fayose and Ayinde, these forms include the maximum, minimum, arithmetic mean, geometric mean, harmonic mean, mid-range, and median. Monte Carlo simulations, repeated 1,000 times, were conducted on regression models with four and seven predictors, across five levels of multicollinearity, three error variances, and eight sample sizes. The Mean Square Error (MSE) criterion was used for evaluation. Results indicate that the maximum form of the Principal Component Ridge estimator consistently outperforms others in terms of efficiency.

References

• Gujarati, D. N. (2021). Essentials of econometrics. Sage Publications.
• Kibria, B. G., & Lukman, A. F. (2020). A new ridge‐type estimator for the linear regression model: simulations and applications. Scientifica, 2020(1), 9758378.
• Paul, R. K. (2006). Multicollinearity: Causes, effects and remedies. IASRI, New Delhi, 1(1), 58–65.
• Montgomery, D. C., Peck, E. A., & Vining, G. G. (2021). Introduction to linear regression analysis. John Wiley & Sons.
• Khalaf, G., & Iguernane, M. (2016). Multicollinearity and a ridge parameter estimation approach. Journal of Modern Applied Statistical Methods, 15(2), 25. https://doi.org/10.22237/jmasm/1478002980
• Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67. https://doi.org/10.1080/00401706.1970.10488634
• Liu, K. (1993). A new class of biased estimate in linear regression. Communications in Statistics - Theory and Methods, 22(2), 393–402.
• Muniz, G., & Kibria, B. M. G. (2009). On Some Ridge Regression Estimators: An Empirical Comparisons. Communications in Statistics - Simulation and Computation, 38(3), 621–630. https://doi.org/10.1080/03610910802592838
• Hoerl, A. E., Kannard, R. W., & Baldwin, K. F. (1975). Ridge regression: some simulations. Communications in Statistics, 4(2), 105–123. https://doi.org/10.1080/03610927508827232
• McDonald, G. C., & Galarneau, D. I. (1975). A Monte Carlo Evaluation of Some Ridge-Type Estimators. Journal of the American Statistical Association, 70(350), 407–416. https://doi.org/10.1080/01621459.1975.10479882
• Lawless F., and Wang P., (1976). A simulation study of ridge and other regression estimators. Communications in Statistics-Theory and Methods 5, p.307-323.
• Dempster, A. P., Schatzoff, M., & Wermuth, N. (1977). A Simulation Study of Alternatives to Ordinary Least Squares. Journal of the American Statistical Association, 72(357), 77–91. https://doi.org/10.1080/01621459.1977.10479910
• Troskie, CG* & Chalton, DO. "A Bayesian estimate for the constants in ridge regression." South African Statistical Journal 30, no. 2 (1996): 119-137.
• Firinguetti, L. (1999). A generalized ridge regression estimator and its finite sample properties: A generalized ridge regression estimator. Communications in Statistics - Theory and Methods, 28(5), 1217–1229. https://doi.org/10.1080/03610929908832353
• Kibria, B. M. G. (2003). Performance of some new ridge regression estimators. Communications in Statistics - Simulation and Computation, 32(2), 419–435. https://doi.org/10.1081/SAC-120017499
• Khalaf, G., & Shukur, G. (2005). Choosing Ridge Parameter for Regression Problems. Communications in Statistics - Theory and Methods, 34(5), 1177–1182. https://doi.org/10.1081/STA-200056836
• Batah, F. S. M., Ramanathan, T. V., & Gore, S. D. (2008). The efficiency of modified jackknife and ridge type regression estimators: a comparison. Surveys in Mathematics and its Applications, 3, 111-122.
• Kibria, B. G., & Banik, S. (2016). Some ridge regression estimators and their performances. Journal of Modern Applied statistical methods, 15, 206-238.
• Lukman, A. F., & Ayinde, K. (2017). Review and classifications of the ridge parameter estimation techniques. Hacettepe Journal of Mathematics and Statistics, 46(5), 953-967.
• Lukman, A. F., Ayinde, K., Oludoun, O., & Onate, C. A. (2020). Combining modified ridge-type and principal component regression estimators. Scientific African, 9, e00536.
• Fayose, T. S., & Ayinde, K. (2019). Different forms biasing parameter for generalized ridge regression estimator. International Journal of Computer Applications, 181(37), 2-29.
• Baye, M. R., & Parker, D. F. (1984). Combining ridge and principal component regression:a money demand illustration. Communications in Statistics - Theory and Methods, 13(2), 197–205. https://doi.org/10.1080/03610928408828675
• Nomura, M., & Ohkubo, T. (1985). A note on combining ridge and principal component regression. Communications in Statistics - Theory and Methods, 14(10), 2489–2493. https://doi.org/10.1080/0361092850882905
• Sarkar, N. (1996). Mean square error matrix comparison of some estimators in linear regressions with multicollinearity. Statist. Probab. Lett. 30:133–138.
• Kaçıranlar, S., & Sakallıoğlu, S. (2001). Combining the Liu Estimator and The Principal Component Regression Estimator. Communications in Statistics - Theory and Methods, 30(12), 2699–2705. https://doi.org/10.1081/STA-100108454
• Ozkale, M. R., & Kaciranlar, S. (2007). Comparisons of the unbiased ridge estimation to the other estimations. Communications in Statistics Theory and Methods, 36(4), 707.
• Batah, F. Sh. M., Özkale, M. R., & Gore, S. D. (2009). Combining Unbiased Ridge and Principal Component Regression Estimators. Communications in Statistics - Theory and Methods, 38(13), 2201–2209. https://doi.org/10.1080/03610920802503396
• Crouse, R. H., Chun Jin, & Hanumara, R. C. (1995). Unbiased ridge estimation with prior information and ridge trace. Communications in Statistics - Theory and Methods, 24(9), 2341–2354. https://doi.org/10.1080/03610929508831620
• Adegoke, A. S., Adewuyi, E., Ayinde, K., & Lukman, A. F. (2016). A comparative study of some robust ridge and liu estimators. Science World Journal, 11(4), 16-20.
• Huang, D., Huang, J., & Bai, D. (2023). Combination of the modified Kibria–Lukman and the principal component regression estimators. Communications in Statistics - Simulation and Computation, 1–16. https://doi.org/10.1080/03610918.2023.2292970
• Özbey, F., & Kaçiranlar, S. (2015). Evaluation of the predictive performance of the Liu estimator. Communications in Statistics-Theory and Methods, 44(10), 1981-1993.
• Lukman, A. F., Haadi, A., Ayinde, K., Onate, C. A., Gbadamosi, B., & Oladejo, N. K. (2018). Some improved generalized ridge estimators and their comparison. WSEAS Transactions on Mathematics, 17, 369–376.
• Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6), 417–441. https://doi.org/10.1037/h0071325
• Pongpiachan, S., Wang, Q., Apiratikul, R., Tipmanee, D., Li, L., Xing, L., ... & Poshyachinda, S. (2024). Combined use of principal component analysis/multiple linear regression analysis and artificial neural network to assess the impact of meteorological parameters on fluctuation of selected PM2.5-bound elements. Plos one, 19(3), e0287187.
• Weeraratne, N., Hunt, L., & Kurz, J. (2024). Challenges of principal component analysis in high-dimensional settings when n < p. Preprint. https://doi.org/10.21203/rs.3.rs-4033858/v1
• Ayinde, K., Apata, E. O., & Alaba, O. O. (2012). Estimators of linear regression model and prediction under some assumptions violation. https://doi.org/ 10.4236/ojs.2012.25069
• Wichern, D. W., & Churchill, G. A. (1978). A comparison of ridge estimators. Technometrics, 20(3), 301–311. https://doi.org/10.1080/00401706.1978.10489675
• Gibbons, D. G. (1981). A simulation study of some ridge estimators. Journal of the American Statistical Association, 76(373), 131-139
• Dorugade, A. V., & Kashid, D. N. (2010). Alternative method for choosing ridge parameter for regression. Applied Mathematical Sciences, 4(9), 447–456
• Dorugade, A. V. (2016). Adjusted ridge estimator and comparison with Kibria’s method in linear regression. Journal of the Association of Arab Universities for Basic and Applied Sciences, 21, 96–102.
• Amalare, A. A., Ayinde, K., & Onanuga, K. (2023). Parameter Estimation of Linear Regression Model with Multicollinearity and Heteroscedasticity Problems. The Journals of the Nigerian Association of Mathematical Physics, 65, 207-216.