Deep Convolutional Networks are Hierarchical Kernel Machines
We extend i-theory to incorporate not only pooling but also rectifying nonlinearities in an extended HW module (eHW) designed for supervised learning. The two operations roughly correspond to invariance and selectivity, respectively. Under the assumption of normalized inputs, we show that appropriat...
| Main Authors: | , , , |
|---|---|
| Format: | Technical Report |
| Language: | en_US |
| Published: |
Center for Brains, Minds and Machines (CBMM), arXiv
2015
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/100200 |
| _version_ | 1826190982961430528 |
|---|---|
| author | Anselmi, Fabio Rosasco, Lorenzo Tan, Cheston Poggio, Tomaso |
| author_facet | Anselmi, Fabio Rosasco, Lorenzo Tan, Cheston Poggio, Tomaso |
| author_sort | Anselmi, Fabio |
| collection | MIT |
| description | We extend i-theory to incorporate not only pooling but also rectifying nonlinearities in an extended HW module (eHW) designed for supervised learning. The two operations roughly correspond to invariance and selectivity, respectively. Under the assumption of normalized inputs, we show that appropriate linear combinations of rectifying nonlinearities are equivalent to radial kernels. If pooling is present an equivalent kernel also exist. Thus present-day DCNs (Deep Convolutional Networks) can be exactly equivalent to a hierarchy of kernel machines with pooling and non-pooling layers. Finally, we describe a conjecture for theoretically understanding hierarchies of such modules. A main consequence of the conjecture is that hierarchies of eHW modules minimize memory requirements while computing a selective and invariant representation. |
| first_indexed | 2024-09-23T08:48:17Z |
| format | Technical Report |
| id | mit-1721.1/100200 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T08:48:17Z |
| publishDate | 2015 |
| publisher | Center for Brains, Minds and Machines (CBMM), arXiv |
| record_format | dspace |
| spelling | mit-1721.1/1002002019-08-02T03:02:31Z Deep Convolutional Networks are Hierarchical Kernel Machines Anselmi, Fabio Rosasco, Lorenzo Tan, Cheston Poggio, Tomaso i-theory extended HW module (eHW) Invariance Selectivity Hierarchy Machine Learning We extend i-theory to incorporate not only pooling but also rectifying nonlinearities in an extended HW module (eHW) designed for supervised learning. The two operations roughly correspond to invariance and selectivity, respectively. Under the assumption of normalized inputs, we show that appropriate linear combinations of rectifying nonlinearities are equivalent to radial kernels. If pooling is present an equivalent kernel also exist. Thus present-day DCNs (Deep Convolutional Networks) can be exactly equivalent to a hierarchy of kernel machines with pooling and non-pooling layers. Finally, we describe a conjecture for theoretically understanding hierarchies of such modules. A main consequence of the conjecture is that hierarchies of eHW modules minimize memory requirements while computing a selective and invariant representation. This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. 2015-12-11T22:26:00Z 2015-12-11T22:26:00Z 2015-08-05 Technical Report Working Paper Other http://hdl.handle.net/1721.1/100200 arXiv:1508.01084 en_US CBMM Memo Series;035 Attribution-NonCommercial 3.0 United States http://creativecommons.org/licenses/by-nc/3.0/us/ application/pdf Center for Brains, Minds and Machines (CBMM), arXiv |
| spellingShingle | i-theory extended HW module (eHW) Invariance Selectivity Hierarchy Machine Learning Anselmi, Fabio Rosasco, Lorenzo Tan, Cheston Poggio, Tomaso Deep Convolutional Networks are Hierarchical Kernel Machines |
| title | Deep Convolutional Networks are Hierarchical Kernel Machines |
| title_full | Deep Convolutional Networks are Hierarchical Kernel Machines |
| title_fullStr | Deep Convolutional Networks are Hierarchical Kernel Machines |
| title_full_unstemmed | Deep Convolutional Networks are Hierarchical Kernel Machines |
| title_short | Deep Convolutional Networks are Hierarchical Kernel Machines |
| title_sort | deep convolutional networks are hierarchical kernel machines |
| topic | i-theory extended HW module (eHW) Invariance Selectivity Hierarchy Machine Learning |
| url | http://hdl.handle.net/1721.1/100200 |
| work_keys_str_mv | AT anselmifabio deepconvolutionalnetworksarehierarchicalkernelmachines AT rosascolorenzo deepconvolutionalnetworksarehierarchicalkernelmachines AT tancheston deepconvolutionalnetworksarehierarchicalkernelmachines AT poggiotomaso deepconvolutionalnetworksarehierarchicalkernelmachines |