Backprop as Functor: A compositional perspective on supervised learning
A supervised learning algorithm searches over a set of functions A→B parametrised by a space P to find the best approximation to some ideal function f:A→B. It does this by taking examples (a,f(a))∈A×B, and updating the parameter according to some rule. We define a category where these update rules m...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Association for Computing Machinery
2020
|
| Online Access: | https://hdl.handle.net/1721.1/123513 |
| _version_ | 1811074020655759360 |
|---|---|
| author | Fong, Brendan C Spivak, David I Tuyeras, Remy V |
| author2 | Massachusetts Institute of Technology. Department of Mathematics |
| author_facet | Massachusetts Institute of Technology. Department of Mathematics Fong, Brendan C Spivak, David I Tuyeras, Remy V |
| author_sort | Fong, Brendan C |
| collection | MIT |
| description | A supervised learning algorithm searches over a set of functions A→B parametrised by a space P to find the best approximation to some ideal function f:A→B. It does this by taking examples (a,f(a))∈A×B, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent---with respect to a fixed step size and an error function satisfying a certain property---defines a monoidal functor from a category of parametrised functions to this category of update rules. This provides a structural perspective on backpropagation, as well as a broad generalisation of neural networks. |
| first_indexed | 2024-09-23T09:41:49Z |
| format | Article |
| id | mit-1721.1/123513 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T09:41:49Z |
| publishDate | 2020 |
| publisher | Association for Computing Machinery |
| record_format | dspace |
| spelling | mit-1721.1/1235132022-09-26T13:12:09Z Backprop as Functor: A compositional perspective on supervised learning Fong, Brendan C Spivak, David I Tuyeras, Remy V Massachusetts Institute of Technology. Department of Mathematics Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Spivak, David I. A supervised learning algorithm searches over a set of functions A→B parametrised by a space P to find the best approximation to some ideal function f:A→B. It does this by taking examples (a,f(a))∈A×B, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent---with respect to a fixed step size and an error function satisfying a certain property---defines a monoidal functor from a category of parametrised functions to this category of update rules. This provides a structural perspective on backpropagation, as well as a broad generalisation of neural networks. Air Force Office of Scientific Research (Award FA9550-14-1-0031) Air Force Office of Scientific Research (Award FA9550-17-1-0058) 2020-01-21T21:27:51Z 2020-01-21T21:27:51Z 2019-06 Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/123513 Fong, Brendan et al. "Backprop as Functor: A compositional perspective on supervised learning." Thirty-Fourth Annual ACM/IEEE Symposium on Logic in Computer Science (LICS), June 2019, Vancouver, Canada, Association for Computing Machinery, June 2019 en_US https://lics.siglog.org/lics19/ Thirty-Fourth Annual ACM/IEEE Symposium on Logic in Computer Science (LICS) Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Association for Computing Machinery David Spivak |
| spellingShingle | Fong, Brendan C Spivak, David I Tuyeras, Remy V Backprop as Functor: A compositional perspective on supervised learning |
| title | Backprop as Functor: A compositional perspective on supervised learning |
| title_full | Backprop as Functor: A compositional perspective on supervised learning |
| title_fullStr | Backprop as Functor: A compositional perspective on supervised learning |
| title_full_unstemmed | Backprop as Functor: A compositional perspective on supervised learning |
| title_short | Backprop as Functor: A compositional perspective on supervised learning |
| title_sort | backprop as functor a compositional perspective on supervised learning |
| url | https://hdl.handle.net/1721.1/123513 |
| work_keys_str_mv | AT fongbrendanc backpropasfunctoracompositionalperspectiveonsupervisedlearning AT spivakdavidi backpropasfunctoracompositionalperspectiveonsupervisedlearning AT tuyerasremyv backpropasfunctoracompositionalperspectiveonsupervisedlearning |