STEER: simple temporal regularization for neural ODEs
Training Neural Ordinary Differential Equations (ODEs) is often computationally expensive. Indeed, computing the forward pass of such models involves solving an ODE which can become arbitrarily complex during training. Recent works have shown that regularizing the dynamics of the ODE can partially a...
| Main Authors: | , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
Neural Information Processing Systems Foundation, Inc.
2020
|
| _version_ | 1826260737678376960 |
|---|---|
| author | Ghosh, A Behl, HS Dupont, E Torr, PHS Namboodiri, V |
| author_facet | Ghosh, A Behl, HS Dupont, E Torr, PHS Namboodiri, V |
| author_sort | Ghosh, A |
| collection | OXFORD |
| description | Training Neural Ordinary Differential Equations (ODEs) is often computationally expensive. Indeed, computing the forward pass of such models involves solving an ODE which can become arbitrarily complex during training. Recent works have shown that regularizing the dynamics of the ODE can partially alleviate this. In this paper we propose a new regularization technique: randomly sampling the end time of the ODE during training. The proposed regularization is simple to implement, has negligible overhead and is effective across a wide variety of tasks. Further, the technique is orthogonal to several other methods proposed to regularize the dynamics of ODEs and as such can be used in conjunction with them. We show through experiments on normalizing flows, time series models and image recognition that the proposed regularization can significantly decrease training time and even improve performance over baseline models. |
| first_indexed | 2024-03-06T19:10:27Z |
| format | Conference item |
| id | oxford-uuid:1698dcce-ff68-445d-b0b8-8ea62b3e170f |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-06T19:10:27Z |
| publishDate | 2020 |
| publisher | Neural Information Processing Systems Foundation, Inc. |
| record_format | dspace |
| spelling | oxford-uuid:1698dcce-ff68-445d-b0b8-8ea62b3e170f2022-03-26T10:32:16ZSTEER: simple temporal regularization for neural ODEsConference itemhttp://purl.org/coar/resource_type/c_5794uuid:1698dcce-ff68-445d-b0b8-8ea62b3e170fEnglishSymplectic ElementsNeural Information Processing Systems Foundation, Inc.2020Ghosh, ABehl, HSDupont, ETorr, PHSNamboodiri, VTraining Neural Ordinary Differential Equations (ODEs) is often computationally expensive. Indeed, computing the forward pass of such models involves solving an ODE which can become arbitrarily complex during training. Recent works have shown that regularizing the dynamics of the ODE can partially alleviate this. In this paper we propose a new regularization technique: randomly sampling the end time of the ODE during training. The proposed regularization is simple to implement, has negligible overhead and is effective across a wide variety of tasks. Further, the technique is orthogonal to several other methods proposed to regularize the dynamics of ODEs and as such can be used in conjunction with them. We show through experiments on normalizing flows, time series models and image recognition that the proposed regularization can significantly decrease training time and even improve performance over baseline models. |
| spellingShingle | Ghosh, A Behl, HS Dupont, E Torr, PHS Namboodiri, V STEER: simple temporal regularization for neural ODEs |
| title | STEER: simple temporal regularization for neural ODEs |
| title_full | STEER: simple temporal regularization for neural ODEs |
| title_fullStr | STEER: simple temporal regularization for neural ODEs |
| title_full_unstemmed | STEER: simple temporal regularization for neural ODEs |
| title_short | STEER: simple temporal regularization for neural ODEs |
| title_sort | steer simple temporal regularization for neural odes |
| work_keys_str_mv | AT ghosha steersimpletemporalregularizationforneuralodes AT behlhs steersimpletemporalregularizationforneuralodes AT duponte steersimpletemporalregularizationforneuralodes AT torrphs steersimpletemporalregularizationforneuralodes AT namboodiriv steersimpletemporalregularizationforneuralodes |