Activation functions performance in multilayer perceptron for time series forecasting
Activation functions are important hyperparameters in neural networks, applied to calculate the weighted sum of inputs and biases and determine whether a neuron can be activated. Choosing the most suitable activation function can assist neural networks in training faster without sacrificing accuracy...
| Main Authors: | , , |
|---|---|
| Format: | Conference or Workshop Item |
| Language: | English English |
| Published: |
AIP Publishing
2024
|
| Subjects: | |
| Online Access: | http://umpir.ump.edu.my/id/eprint/42460/1/2024%20Activation%20Functions%20Performance%20in%20Multilayer%20Perceptron%20for%20Time%20Series%20Forecasting.pdf http://umpir.ump.edu.my/id/eprint/42460/2/Activation%20functions%20performance%20in%20multilayer%20perceptron%20for%20time%20series%20forecasting.pdf |
| _version_ | 1825815882892312576 |
|---|---|
| author | Nur Haizum, Abd Rahman Yin, Chin Hui Hani Syahida, Zulkafli |
| author_facet | Nur Haizum, Abd Rahman Yin, Chin Hui Hani Syahida, Zulkafli |
| author_sort | Nur Haizum, Abd Rahman |
| collection | UMP |
| description | Activation functions are important hyperparameters in neural networks, applied to calculate the weighted sum of inputs and biases and determine whether a neuron can be activated. Choosing the most suitable activation function can assist neural networks in training faster without sacrificing accuracy. This study aims to evaluate the performance of three activation functions, Sigmoid, Hyperbolic Tangent (Tanh), and Rectified Linear Unit (ReLU) in the hidden layer of Multilayer Perceptron (MLP) for time series forecasting. To evaluate the activation functions, three simulated non-linear time series were generated using the Threshold Autoregressive (TAR) model, and two real datasets, the Canadian Lynx series and Wolf’s Sunspot data, were employed. The Mean Square Error (MSE) and Mean Absolute Error (MAE) were computed to measure the performance accuracy. The analysis of the real data revealed that the Tanh function exhibited the lowest MSE and MAE, with values of 1.345 and 0.945, respectively. The Sigmoid function yielded MSE and MAE values of 1.520 and 1.005, while the ReLU function resulted in values of 1.562 and 1.018. These findings align with the simulation results, confirming that the Tanh function is the most effective for time series forecasting. Therefore, it is recommended to replace the commonly used Sigmoid function with Tanh for an accurate forecast. |
| first_indexed | 2024-09-25T03:52:43Z |
| format | Conference or Workshop Item |
| id | UMPir42460 |
| institution | Universiti Malaysia Pahang |
| language | English English |
| last_indexed | 2024-09-25T03:52:43Z |
| publishDate | 2024 |
| publisher | AIP Publishing |
| record_format | dspace |
| spelling | UMPir424602024-09-05T04:21:14Z http://umpir.ump.edu.my/id/eprint/42460/ Activation functions performance in multilayer perceptron for time series forecasting Nur Haizum, Abd Rahman Yin, Chin Hui Hani Syahida, Zulkafli QA Mathematics Activation functions are important hyperparameters in neural networks, applied to calculate the weighted sum of inputs and biases and determine whether a neuron can be activated. Choosing the most suitable activation function can assist neural networks in training faster without sacrificing accuracy. This study aims to evaluate the performance of three activation functions, Sigmoid, Hyperbolic Tangent (Tanh), and Rectified Linear Unit (ReLU) in the hidden layer of Multilayer Perceptron (MLP) for time series forecasting. To evaluate the activation functions, three simulated non-linear time series were generated using the Threshold Autoregressive (TAR) model, and two real datasets, the Canadian Lynx series and Wolf’s Sunspot data, were employed. The Mean Square Error (MSE) and Mean Absolute Error (MAE) were computed to measure the performance accuracy. The analysis of the real data revealed that the Tanh function exhibited the lowest MSE and MAE, with values of 1.345 and 0.945, respectively. The Sigmoid function yielded MSE and MAE values of 1.520 and 1.005, while the ReLU function resulted in values of 1.562 and 1.018. These findings align with the simulation results, confirming that the Tanh function is the most effective for time series forecasting. Therefore, it is recommended to replace the commonly used Sigmoid function with Tanh for an accurate forecast. AIP Publishing 2024-08 Conference or Workshop Item PeerReviewed pdf en http://umpir.ump.edu.my/id/eprint/42460/1/2024%20Activation%20Functions%20Performance%20in%20Multilayer%20Perceptron%20for%20Time%20Series%20Forecasting.pdf pdf en http://umpir.ump.edu.my/id/eprint/42460/2/Activation%20functions%20performance%20in%20multilayer%20perceptron%20for%20time%20series%20forecasting.pdf Nur Haizum, Abd Rahman and Yin, Chin Hui and Hani Syahida, Zulkafli (2024) Activation functions performance in multilayer perceptron for time series forecasting. In: AIP Conference Proceedings. The 6th ISM International Statistical Conference (ISM-VI) 2023 , 19–20 September 2023 , Shah Alam, Malaysia. pp. 1-10., 3123 (1). ISBN 978-0-7354-5030-1 (Published) https://doi.org/10.1063/5.0223864 |
| spellingShingle | QA Mathematics Nur Haizum, Abd Rahman Yin, Chin Hui Hani Syahida, Zulkafli Activation functions performance in multilayer perceptron for time series forecasting |
| title | Activation functions performance in multilayer perceptron for time series forecasting |
| title_full | Activation functions performance in multilayer perceptron for time series forecasting |
| title_fullStr | Activation functions performance in multilayer perceptron for time series forecasting |
| title_full_unstemmed | Activation functions performance in multilayer perceptron for time series forecasting |
| title_short | Activation functions performance in multilayer perceptron for time series forecasting |
| title_sort | activation functions performance in multilayer perceptron for time series forecasting |
| topic | QA Mathematics |
| url | http://umpir.ump.edu.my/id/eprint/42460/1/2024%20Activation%20Functions%20Performance%20in%20Multilayer%20Perceptron%20for%20Time%20Series%20Forecasting.pdf http://umpir.ump.edu.my/id/eprint/42460/2/Activation%20functions%20performance%20in%20multilayer%20perceptron%20for%20time%20series%20forecasting.pdf |
| work_keys_str_mv | AT nurhaizumabdrahman activationfunctionsperformanceinmultilayerperceptronfortimeseriesforecasting AT yinchinhui activationfunctionsperformanceinmultilayerperceptronfortimeseriesforecasting AT hanisyahidazulkafli activationfunctionsperformanceinmultilayerperceptronfortimeseriesforecasting |