Smooth Function Approximation by Deep Neural Networks with General Activation Functions
There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2019-06-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/21/7/627 |
| _version_ | 1798002516298825728 |
|---|---|
| author | Ilsang Ohn Yongdai Kim |
| author_facet | Ilsang Ohn Yongdai Kim |
| author_sort | Ilsang Ohn |
| collection | DOAJ |
| description | There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functions. This class of activation functions includes most of frequently used activation functions. We derive the required depth, width and sparsity of a deep neural network to approximate any Hölder smooth function upto a given approximation error for the large class of activation functions. Based on our approximation error analysis, we derive the minimax optimality of the deep neural network estimators with the general activation functions in both regression and classification problems. |
| first_indexed | 2024-04-11T11:53:20Z |
| format | Article |
| id | doaj.art-b679be97d72f46f9a1a7bf79a638b155 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-04-11T11:53:20Z |
| publishDate | 2019-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-b679be97d72f46f9a1a7bf79a638b1552022-12-22T04:25:15ZengMDPI AGEntropy1099-43002019-06-0121762710.3390/e21070627e21070627Smooth Function Approximation by Deep Neural Networks with General Activation FunctionsIlsang Ohn0Yongdai Kim1Department of Statistics, Seoul National University, Seoul 08826, KoreaDepartment of Statistics, Seoul National University, Seoul 08826, KoreaThere has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functions. This class of activation functions includes most of frequently used activation functions. We derive the required depth, width and sparsity of a deep neural network to approximate any Hölder smooth function upto a given approximation error for the large class of activation functions. Based on our approximation error analysis, we derive the minimax optimality of the deep neural network estimators with the general activation functions in both regression and classification problems.https://www.mdpi.com/1099-4300/21/7/627function approximationdeep neural networksactivation functionsHölder continuityconvergence rates |
| spellingShingle | Ilsang Ohn Yongdai Kim Smooth Function Approximation by Deep Neural Networks with General Activation Functions Entropy function approximation deep neural networks activation functions Hölder continuity convergence rates |
| title | Smooth Function Approximation by Deep Neural Networks with General Activation Functions |
| title_full | Smooth Function Approximation by Deep Neural Networks with General Activation Functions |
| title_fullStr | Smooth Function Approximation by Deep Neural Networks with General Activation Functions |
| title_full_unstemmed | Smooth Function Approximation by Deep Neural Networks with General Activation Functions |
| title_short | Smooth Function Approximation by Deep Neural Networks with General Activation Functions |
| title_sort | smooth function approximation by deep neural networks with general activation functions |
| topic | function approximation deep neural networks activation functions Hölder continuity convergence rates |
| url | https://www.mdpi.com/1099-4300/21/7/627 |
| work_keys_str_mv | AT ilsangohn smoothfunctionapproximationbydeepneuralnetworkswithgeneralactivationfunctions AT yongdaikim smoothfunctionapproximationbydeepneuralnetworkswithgeneralactivationfunctions |