TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equations
The architecture of deep neural networks is commonly determined via trial and error, resulting in inefficiency and a lack of architecture interpretability. Recent research shows that numerical solutions of ordinary differential equations have demonstrated great potential for designing network archit...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-10-01
|
| Series: | Alexandria Engineering Journal |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1110016823007834 |
| _version_ | 1797659885722140672 |
|---|---|
| author | Haoyu Chu Shikui Wei Shunli Zhang Yao Zhao |
| author_facet | Haoyu Chu Shikui Wei Shunli Zhang Yao Zhao |
| author_sort | Haoyu Chu |
| collection | DOAJ |
| description | The architecture of deep neural networks is commonly determined via trial and error, resulting in inefficiency and a lack of architecture interpretability. Recent research shows that numerical solutions of ordinary differential equations have demonstrated great potential for designing network architectures. Implicit methods are generally more stable and accurate than explicit methods but are yet to be applied to neural network design. Unlike explicit methods that directly calculate the current state based on the previous state, implicit methods have to solve a nonlinear equation to get the approximate solution, which discourages using implicit methods in network design. In this paper, we propose replacing Euler's method with the trapezoidal rule, a scheme that is a second-order implicit method with much lower local truncation errors. The crux of our approach lies in first altering the trapezoidal rule from its implicit form to the explicit form and then designing the corresponding neural network. By introducing the multi-channel convolution and layer-sharing mechanism, we propose a new architecture called TrapezoidalNet. We expect TrapezoidalNet can outperform other deep neural networks based on explicit methods such as ResNet, FitResNet, FractalNet, and LM-ResNet on image recognition tasks. Experimental results on CIFAR10/100 datasets verify the effectiveness of TrapezoidalNet. |
| first_indexed | 2024-03-11T18:21:42Z |
| format | Article |
| id | doaj.art-055f205465d045f3b83047c368c008ee |
| institution | Directory Open Access Journal |
| issn | 1110-0168 |
| language | English |
| last_indexed | 2024-03-11T18:21:42Z |
| publishDate | 2023-10-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Alexandria Engineering Journal |
| spelling | doaj.art-055f205465d045f3b83047c368c008ee2023-10-15T04:36:45ZengElsevierAlexandria Engineering Journal1110-01682023-10-01815563TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equationsHaoyu Chu0Shikui Wei1Shunli Zhang2Yao Zhao3Institute of Information Science, Beijing Jiaotong University, Beijing 100044, China; Beijing Key Laboratory of Advanced Information Science and Network Technology, Beijing 100044, China; Graduate School of Information Science and Technology, Osaka University, Osaka 5600043, JapanInstitute of Information Science, Beijing Jiaotong University, Beijing 100044, China; Beijing Key Laboratory of Advanced Information Science and Network Technology, Beijing 100044, China; Corresponding author at: Institute of Information Science, Beijing Jiaotong University, Beijing 100044, China.School of Software Engineering, Beijing Jiaotong University, Beijing 100044, ChinaInstitute of Information Science, Beijing Jiaotong University, Beijing 100044, China; Beijing Key Laboratory of Advanced Information Science and Network Technology, Beijing 100044, ChinaThe architecture of deep neural networks is commonly determined via trial and error, resulting in inefficiency and a lack of architecture interpretability. Recent research shows that numerical solutions of ordinary differential equations have demonstrated great potential for designing network architectures. Implicit methods are generally more stable and accurate than explicit methods but are yet to be applied to neural network design. Unlike explicit methods that directly calculate the current state based on the previous state, implicit methods have to solve a nonlinear equation to get the approximate solution, which discourages using implicit methods in network design. In this paper, we propose replacing Euler's method with the trapezoidal rule, a scheme that is a second-order implicit method with much lower local truncation errors. The crux of our approach lies in first altering the trapezoidal rule from its implicit form to the explicit form and then designing the corresponding neural network. By introducing the multi-channel convolution and layer-sharing mechanism, we propose a new architecture called TrapezoidalNet. We expect TrapezoidalNet can outperform other deep neural networks based on explicit methods such as ResNet, FitResNet, FractalNet, and LM-ResNet on image recognition tasks. Experimental results on CIFAR10/100 datasets verify the effectiveness of TrapezoidalNet.http://www.sciencedirect.com/science/article/pii/S1110016823007834Neural network designOrdinary differential equationsNumerical approximation methodsImage classifications |
| spellingShingle | Haoyu Chu Shikui Wei Shunli Zhang Yao Zhao TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equations Alexandria Engineering Journal Neural network design Ordinary differential equations Numerical approximation methods Image classifications |
| title | TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equations |
| title_full | TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equations |
| title_fullStr | TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equations |
| title_full_unstemmed | TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equations |
| title_short | TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equations |
| title_sort | trapezoidalnet a new network architecture inspired from the numerical solution of ordinary differential equations |
| topic | Neural network design Ordinary differential equations Numerical approximation methods Image classifications |
| url | http://www.sciencedirect.com/science/article/pii/S1110016823007834 |
| work_keys_str_mv | AT haoyuchu trapezoidalnetanewnetworkarchitectureinspiredfromthenumericalsolutionofordinarydifferentialequations AT shikuiwei trapezoidalnetanewnetworkarchitectureinspiredfromthenumericalsolutionofordinarydifferentialequations AT shunlizhang trapezoidalnetanewnetworkarchitectureinspiredfromthenumericalsolutionofordinarydifferentialequations AT yaozhao trapezoidalnetanewnetworkarchitectureinspiredfromthenumericalsolutionofordinarydifferentialequations |