Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter
Variational estimation method is a deterministic approximation technique which involves Bayesian framework while giving a point estimate instead of the usual Bayesian interval estimation. The linear regression model, which has always been a popular model, can benefit from the implementation of varia...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
EDP Sciences
2024-01-01
|
| Series: | ITM Web of Conferences |
| Subjects: | |
| Online Access: | https://www.itm-conferences.org/articles/itmconf/pdf/2024/04/itmconf_iscpms2024_01010.pdf |
| _version_ | 1827372054502965248 |
|---|---|
| author | Widyaningsih Yekti Rizka Hakiim Nur Siswantining Titin |
| author_facet | Widyaningsih Yekti Rizka Hakiim Nur Siswantining Titin |
| author_sort | Widyaningsih Yekti |
| collection | DOAJ |
| description | Variational estimation method is a deterministic approximation technique which involves Bayesian framework while giving a point estimate instead of the usual Bayesian interval estimation. The linear regression model, which has always been a popular model, can benefit from the implementation of variational estimation method. In this paper, the theoretical basis on why variational method can reduce overfitting in linear regression is reviewed. Based on the review, in theory, variational method is more robust to overfitting than MLE. This paper also performed a simulation study. The simulation is done in a manner such that the simulation represents the situation of predicting for new or hidden data. The simulation starts from generating random explanatory data and generates the appropriate response data based on linear regression equation. Then, the randomly generated data is used to estimate the linear regression parameters. The simulation is performed to compare the parameters estimation results from variational method with the method of MLE. The comparison is done using the estimation values and the squared differences between true parameters value and the estimates. Empirical findings show that both methods have relatively close estimate values. It can be seen as the simulation study concludes that both variational and ML yield rather close parameters estimates for simple linear regression case. The estimates closeness gets more obvious as the sample size grows. The study also found that Variational method has performs better in terms of parameters estimation in linear regression when the sample size is small or the data has large variance. |
| first_indexed | 2024-03-08T10:50:52Z |
| format | Article |
| id | doaj.art-ea2e62d69f304882b762a40bbc608429 |
| institution | Directory Open Access Journal |
| issn | 2271-2097 |
| language | English |
| last_indexed | 2024-03-08T10:50:52Z |
| publishDate | 2024-01-01 |
| publisher | EDP Sciences |
| record_format | Article |
| series | ITM Web of Conferences |
| spelling | doaj.art-ea2e62d69f304882b762a40bbc6084292024-01-26T16:48:07ZengEDP SciencesITM Web of Conferences2271-20972024-01-01610101010.1051/itmconf/20246101010itmconf_iscpms2024_01010Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameterWidyaningsih Yekti0Rizka Hakiim Nur1Siswantining Titin2Department of Mathematics, Faculty of Mathematics and Natural Sciences (FMIPA), Universitas IndonesiaDepartment of Mathematics, Faculty of Mathematics and Natural Sciences (FMIPA), Universitas IndonesiaDepartment of Mathematics, Faculty of Mathematics and Natural Sciences (FMIPA), Universitas IndonesiaVariational estimation method is a deterministic approximation technique which involves Bayesian framework while giving a point estimate instead of the usual Bayesian interval estimation. The linear regression model, which has always been a popular model, can benefit from the implementation of variational estimation method. In this paper, the theoretical basis on why variational method can reduce overfitting in linear regression is reviewed. Based on the review, in theory, variational method is more robust to overfitting than MLE. This paper also performed a simulation study. The simulation is done in a manner such that the simulation represents the situation of predicting for new or hidden data. The simulation starts from generating random explanatory data and generates the appropriate response data based on linear regression equation. Then, the randomly generated data is used to estimate the linear regression parameters. The simulation is performed to compare the parameters estimation results from variational method with the method of MLE. The comparison is done using the estimation values and the squared differences between true parameters value and the estimates. Empirical findings show that both methods have relatively close estimate values. It can be seen as the simulation study concludes that both variational and ML yield rather close parameters estimates for simple linear regression case. The estimates closeness gets more obvious as the sample size grows. The study also found that Variational method has performs better in terms of parameters estimation in linear regression when the sample size is small or the data has large variance.https://www.itm-conferences.org/articles/itmconf/pdf/2024/04/itmconf_iscpms2024_01010.pdfbayesian regressionmaximum likelihoodoverfittingsimulation |
| spellingShingle | Widyaningsih Yekti Rizka Hakiim Nur Siswantining Titin Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter ITM Web of Conferences bayesian regression maximum likelihood overfitting simulation |
| title | Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter |
| title_full | Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter |
| title_fullStr | Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter |
| title_full_unstemmed | Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter |
| title_short | Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter |
| title_sort | performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter |
| topic | bayesian regression maximum likelihood overfitting simulation |
| url | https://www.itm-conferences.org/articles/itmconf/pdf/2024/04/itmconf_iscpms2024_01010.pdf |
| work_keys_str_mv | AT widyaningsihyekti performancecomparisonbetweenmaximumlikelihoodestimationandvariationalmethodforestimatingsimplelinearregressionparameter AT rizkahakiimnur performancecomparisonbetweenmaximumlikelihoodestimationandvariationalmethodforestimatingsimplelinearregressionparameter AT siswantiningtitin performancecomparisonbetweenmaximumlikelihoodestimationandvariationalmethodforestimatingsimplelinearregressionparameter |