Rational neural networks
We consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds in terms of network complexity and prove that rational neural...
| Main Authors: | , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
Conference on Neural Information Processing Systems
2020
|
| _version_ | 1797067623866826752 |
|---|---|
| author | Boullé, N Nakatsukasa, Y Townsend, A |
| author_facet | Boullé, N Nakatsukasa, Y Townsend, A |
| author_sort | Boullé, N |
| collection | OXFORD |
| description | We consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds in terms of network complexity and prove that rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth. The flexibility and smoothness of rational activation functions make them an attractive alternative to ReLU, as we demonstrate with numerical experiments. |
| first_indexed | 2024-03-06T21:59:00Z |
| format | Conference item |
| id | oxford-uuid:4df37d2b-1104-4ead-955f-02c95ed9a388 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-06T21:59:00Z |
| publishDate | 2020 |
| publisher | Conference on Neural Information Processing Systems |
| record_format | dspace |
| spelling | oxford-uuid:4df37d2b-1104-4ead-955f-02c95ed9a3882022-03-26T15:58:26ZRational neural networksConference itemhttp://purl.org/coar/resource_type/c_c94fuuid:4df37d2b-1104-4ead-955f-02c95ed9a388EnglishSymplectic ElementsConference on Neural Information Processing Systems2020Boullé, NNakatsukasa, YTownsend, AWe consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds in terms of network complexity and prove that rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth. The flexibility and smoothness of rational activation functions make them an attractive alternative to ReLU, as we demonstrate with numerical experiments. |
| spellingShingle | Boullé, N Nakatsukasa, Y Townsend, A Rational neural networks |
| title | Rational neural networks |
| title_full | Rational neural networks |
| title_fullStr | Rational neural networks |
| title_full_unstemmed | Rational neural networks |
| title_short | Rational neural networks |
| title_sort | rational neural networks |
| work_keys_str_mv | AT boullen rationalneuralnetworks AT nakatsukasay rationalneuralnetworks AT townsenda rationalneuralnetworks |