Hierarchically Local Tasks and Deep Convolutional Networks
The main success stories of deep learning, starting with ImageNet, depend on convolutional networks, which on certain tasks perform significantly better than traditional shallow classifiers, such as support vector machines. Is there something special about deep convolutional networks that other lear...
| Main Authors: | , , , |
|---|---|
| Format: | Technical Report |
| Published: |
Center for Brains, Minds and Machines (CBMM)
2020
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/1721.1/125980 |
| _version_ | 1811075071386583040 |
|---|---|
| author | Deza, Arturo Liao, Qianli Banburski, Andrzej Poggio, Tomaso |
| author_facet | Deza, Arturo Liao, Qianli Banburski, Andrzej Poggio, Tomaso |
| author_sort | Deza, Arturo |
| collection | MIT |
| description | The main success stories of deep learning, starting with ImageNet, depend on convolutional networks, which on certain tasks perform significantly better than traditional shallow classifiers, such as support vector machines. Is there something special about deep convolutional networks that other learning machines do not possess? Recent results in approximation theory have shown that there is an exponential advantage of deep convolutional-like networks in approximating functions with hierarchical locality in their compositional structure. These mathematical results, however, do not say which tasks are expected to have input-output functions with hierarchical locality. Among all the possible hierarchically local tasks in vision, text and speech we explore a few of them experimentally by studying how they are affected by disrupting locality in the input images. We also discuss a taxonomy of tasks ranging from local, to hierarchically local, to global and make predictions about the type of networks required to perform efficiently on these different types of tasks. |
| first_indexed | 2024-09-23T10:00:14Z |
| format | Technical Report |
| id | mit-1721.1/125980 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T10:00:14Z |
| publishDate | 2020 |
| publisher | Center for Brains, Minds and Machines (CBMM) |
| record_format | dspace |
| spelling | mit-1721.1/1259802020-07-31T10:15:34Z Hierarchically Local Tasks and Deep Convolutional Networks Deza, Arturo Liao, Qianli Banburski, Andrzej Poggio, Tomaso Compositionality Inductive Bias perception Theory of Deep Learning The main success stories of deep learning, starting with ImageNet, depend on convolutional networks, which on certain tasks perform significantly better than traditional shallow classifiers, such as support vector machines. Is there something special about deep convolutional networks that other learning machines do not possess? Recent results in approximation theory have shown that there is an exponential advantage of deep convolutional-like networks in approximating functions with hierarchical locality in their compositional structure. These mathematical results, however, do not say which tasks are expected to have input-output functions with hierarchical locality. Among all the possible hierarchically local tasks in vision, text and speech we explore a few of them experimentally by studying how they are affected by disrupting locality in the input images. We also discuss a taxonomy of tasks ranging from local, to hierarchically local, to global and make predictions about the type of networks required to perform efficiently on these different types of tasks. This material is based upon work supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. 2020-06-25T14:51:06Z 2020-06-25T14:51:06Z 2020-06-24 Technical Report Working Paper Other https://hdl.handle.net/1721.1/125980 CBMM Memo;109 application/pdf Center for Brains, Minds and Machines (CBMM) |
| spellingShingle | Compositionality Inductive Bias perception Theory of Deep Learning Deza, Arturo Liao, Qianli Banburski, Andrzej Poggio, Tomaso Hierarchically Local Tasks and Deep Convolutional Networks |
| title | Hierarchically Local Tasks and Deep Convolutional Networks |
| title_full | Hierarchically Local Tasks and Deep Convolutional Networks |
| title_fullStr | Hierarchically Local Tasks and Deep Convolutional Networks |
| title_full_unstemmed | Hierarchically Local Tasks and Deep Convolutional Networks |
| title_short | Hierarchically Local Tasks and Deep Convolutional Networks |
| title_sort | hierarchically local tasks and deep convolutional networks |
| topic | Compositionality Inductive Bias perception Theory of Deep Learning |
| url | https://hdl.handle.net/1721.1/125980 |
| work_keys_str_mv | AT dezaarturo hierarchicallylocaltasksanddeepconvolutionalnetworks AT liaoqianli hierarchicallylocaltasksanddeepconvolutionalnetworks AT banburskiandrzej hierarchicallylocaltasksanddeepconvolutionalnetworks AT poggiotomaso hierarchicallylocaltasksanddeepconvolutionalnetworks |