Noise influence on recurrent neural network with nonlinear neurons
The purpose of this study is to establish the features of noise propagation and accumulation in a recurrent neural network using a simplified echo network as an example. In this work, we studied the influence of activation function of artificial neurons and the connection matrices between them. Met...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Saratov State University
2023-07-01
|
| Series: | Известия высших учебных заведений: Прикладная нелинейная динамика |
| Subjects: | |
| Online Access: | https://andjournal.sgu.ru/sites/andjournal.sgu.ru/files/text-pdf/2023/07/and_2023-4_7moskvitin-semenova_484-500.pdf |
| _version_ | 1797769064669511680 |
|---|---|
| author | Moskvitin, Viktor Максимович Semenova, Nadezhda Игоревна |
| author_facet | Moskvitin, Viktor Максимович Semenova, Nadezhda Игоревна |
| author_sort | Moskvitin, Viktor Максимович |
| collection | DOAJ |
| description | The purpose of this study is to establish the features of noise propagation and accumulation in a recurrent neural network using a simplified echo network as an example. In this work, we studied the influence of activation function of artificial neurons and the connection matrices between them. Methods. We have considered white Gaussian noise sources. We used additive, multiplicative and mixed noise depending on how the noise is introduced into artificial neurons. The noise impact was estimated using the dispersion (variance) of the output signal. Results. It is shown that the activation function plays a significant role in noise accumulation. Two nonlinear activation functions have been considered: the hyperbolic tangent and the sigmoid function with range form 0 to 1. It is shown that some types of noise are suppressed in the case of the second function. As a result of considering the influence of coupling matrices, it was found that diagonal coupling matrices with a large blurring coefficient lead to less noise accumulation in the echo network reservoir with an increase in the reservoir memory influence. Conclusion. It is shown that activation functions of the form of sigmoid with range from 0 to 1 are suitable for suppressing multiplicative and mixed noise. The accumulation of noise in the reservoir was considered for three types of coupling matrices inside the reservoir: a uniform matrix, a band matrix with a small blurring coefficient, and a band matrix with a large blurring coefficient. It has been found that the band matrix echo networks with a high blurring coefficient accumulates the least noise. This holds for both additive and multiplicative noise. |
| first_indexed | 2024-03-12T21:02:25Z |
| format | Article |
| id | doaj.art-d9eafe98650647a0948843c36e64dc0c |
| institution | Directory Open Access Journal |
| issn | 0869-6632 2542-1905 |
| language | English |
| last_indexed | 2024-03-12T21:02:25Z |
| publishDate | 2023-07-01 |
| publisher | Saratov State University |
| record_format | Article |
| series | Известия высших учебных заведений: Прикладная нелинейная динамика |
| spelling | doaj.art-d9eafe98650647a0948843c36e64dc0c2023-07-31T04:28:19ZengSaratov State UniversityИзвестия высших учебных заведений: Прикладная нелинейная динамика0869-66322542-19052023-07-0131448450010.18500/0869-6632-003052Noise influence on recurrent neural network with nonlinear neuronsMoskvitin, Viktor Максимович0Semenova, Nadezhda Игоревна1Saratov State University, ul. Astrakhanskaya, 83, Saratov, 410012, RussiaSaratov State University, ul. Astrakhanskaya, 83, Saratov, 410012, RussiaThe purpose of this study is to establish the features of noise propagation and accumulation in a recurrent neural network using a simplified echo network as an example. In this work, we studied the influence of activation function of artificial neurons and the connection matrices between them. Methods. We have considered white Gaussian noise sources. We used additive, multiplicative and mixed noise depending on how the noise is introduced into artificial neurons. The noise impact was estimated using the dispersion (variance) of the output signal. Results. It is shown that the activation function plays a significant role in noise accumulation. Two nonlinear activation functions have been considered: the hyperbolic tangent and the sigmoid function with range form 0 to 1. It is shown that some types of noise are suppressed in the case of the second function. As a result of considering the influence of coupling matrices, it was found that diagonal coupling matrices with a large blurring coefficient lead to less noise accumulation in the echo network reservoir with an increase in the reservoir memory influence. Conclusion. It is shown that activation functions of the form of sigmoid with range from 0 to 1 are suitable for suppressing multiplicative and mixed noise. The accumulation of noise in the reservoir was considered for three types of coupling matrices inside the reservoir: a uniform matrix, a band matrix with a small blurring coefficient, and a band matrix with a large blurring coefficient. It has been found that the band matrix echo networks with a high blurring coefficient accumulates the least noise. This holds for both additive and multiplicative noise.https://andjournal.sgu.ru/sites/andjournal.sgu.ru/files/text-pdf/2023/07/and_2023-4_7moskvitin-semenova_484-500.pdfneural networksrecurrent neural networksecho-state networksnoise influencewhite noisenonlinear activation function |
| spellingShingle | Moskvitin, Viktor Максимович Semenova, Nadezhda Игоревна Noise influence on recurrent neural network with nonlinear neurons Известия высших учебных заведений: Прикладная нелинейная динамика neural networks recurrent neural networks echo-state networks noise influence white noise nonlinear activation function |
| title | Noise influence on recurrent neural network with nonlinear neurons |
| title_full | Noise influence on recurrent neural network with nonlinear neurons |
| title_fullStr | Noise influence on recurrent neural network with nonlinear neurons |
| title_full_unstemmed | Noise influence on recurrent neural network with nonlinear neurons |
| title_short | Noise influence on recurrent neural network with nonlinear neurons |
| title_sort | noise influence on recurrent neural network with nonlinear neurons |
| topic | neural networks recurrent neural networks echo-state networks noise influence white noise nonlinear activation function |
| url | https://andjournal.sgu.ru/sites/andjournal.sgu.ru/files/text-pdf/2023/07/and_2023-4_7moskvitin-semenova_484-500.pdf |
| work_keys_str_mv | AT moskvitinviktormaksimovič noiseinfluenceonrecurrentneuralnetworkwithnonlinearneurons AT semenovanadezhdaigorevna noiseinfluenceonrecurrentneuralnetworkwithnonlinearneurons |