Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression
In this paper, a new nonsmooth optimization-based algorithm for solving large-scale regression problems is introduced. The regression problem is modeled as fully-connected feedforward neural networks with one hidden layer, piecewise linear activation, and the <inline-formula><math xmlns=&qu...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Algorithms |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-4893/16/9/444 |
| _version_ | 1797581606984089600 |
|---|---|
| author | Napsu Karmitsa Sona Taheri Kaisa Joki Pauliina Paasivirta Adil M. Bagirov Marko M. Mäkelä |
| author_facet | Napsu Karmitsa Sona Taheri Kaisa Joki Pauliina Paasivirta Adil M. Bagirov Marko M. Mäkelä |
| author_sort | Napsu Karmitsa |
| collection | DOAJ |
| description | In this paper, a new nonsmooth optimization-based algorithm for solving large-scale regression problems is introduced. The regression problem is modeled as fully-connected feedforward neural networks with one hidden layer, piecewise linear activation, and the <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mn>1</mn></msub></semantics></math></inline-formula>-loss functions. A modified version of the limited memory bundle method is applied to minimize this nonsmooth objective. In addition, a novel constructive approach for automated determination of the proper number of hidden nodes is developed. Finally, large real-world data sets are used to evaluate the proposed algorithm and to compare it with some state-of-the-art neural network algorithms for regression. The results demonstrate the superiority of the proposed algorithm as a predictive tool in most data sets used in numerical experiments. |
| first_indexed | 2024-03-10T23:07:46Z |
| format | Article |
| id | doaj.art-ab4197fef93247a4b6c6d09ea6ed168f |
| institution | Directory Open Access Journal |
| issn | 1999-4893 |
| language | English |
| last_indexed | 2024-03-10T23:07:46Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Algorithms |
| spelling | doaj.art-ab4197fef93247a4b6c6d09ea6ed168f2023-11-19T09:13:13ZengMDPI AGAlgorithms1999-48932023-09-0116944410.3390/a16090444Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale RegressionNapsu Karmitsa0Sona Taheri1Kaisa Joki2Pauliina Paasivirta3Adil M. Bagirov4Marko M. Mäkelä5Department of Computing, University of Turku, FI-20014 Turku, FinlandSchool of Science, RMIT University, Melbourne 3000, AustraliaDepartment of Mathematics and Statistics, University of Turku, FI-20014 Turku, FinlandSiili Solutions Oyj, FI-60100 Seinäjoki, FinlandCentre for Smart Analytics, Federation University Australia, Ballarat 3350, AustraliaDepartment of Mathematics and Statistics, University of Turku, FI-20014 Turku, FinlandIn this paper, a new nonsmooth optimization-based algorithm for solving large-scale regression problems is introduced. The regression problem is modeled as fully-connected feedforward neural networks with one hidden layer, piecewise linear activation, and the <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mn>1</mn></msub></semantics></math></inline-formula>-loss functions. A modified version of the limited memory bundle method is applied to minimize this nonsmooth objective. In addition, a novel constructive approach for automated determination of the proper number of hidden nodes is developed. Finally, large real-world data sets are used to evaluate the proposed algorithm and to compare it with some state-of-the-art neural network algorithms for regression. The results demonstrate the superiority of the proposed algorithm as a predictive tool in most data sets used in numerical experiments.https://www.mdpi.com/1999-4893/16/9/444machine learningregression analysisneural networksL1-loss functionnonsmooth optimization |
| spellingShingle | Napsu Karmitsa Sona Taheri Kaisa Joki Pauliina Paasivirta Adil M. Bagirov Marko M. Mäkelä Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression Algorithms machine learning regression analysis neural networks L1-loss function nonsmooth optimization |
| title | Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression |
| title_full | Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression |
| title_fullStr | Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression |
| title_full_unstemmed | Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression |
| title_short | Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression |
| title_sort | nonsmooth optimization based hyperparameter free neural networks for large scale regression |
| topic | machine learning regression analysis neural networks L1-loss function nonsmooth optimization |
| url | https://www.mdpi.com/1999-4893/16/9/444 |
| work_keys_str_mv | AT napsukarmitsa nonsmoothoptimizationbasedhyperparameterfreeneuralnetworksforlargescaleregression AT sonataheri nonsmoothoptimizationbasedhyperparameterfreeneuralnetworksforlargescaleregression AT kaisajoki nonsmoothoptimizationbasedhyperparameterfreeneuralnetworksforlargescaleregression AT pauliinapaasivirta nonsmoothoptimizationbasedhyperparameterfreeneuralnetworksforlargescaleregression AT adilmbagirov nonsmoothoptimizationbasedhyperparameterfreeneuralnetworksforlargescaleregression AT markommakela nonsmoothoptimizationbasedhyperparameterfreeneuralnetworksforlargescaleregression |