Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
The primate visual system achieves remarkable visual object recognition performance even in brief presentations, and under changes to object exemplar, geometric transformations, and background variation (a.k.a. core visual object recognition). This remarkable performance is mediated by the represent...
| Main Authors: | , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Public Library of Science
2014
|
| Online Access: | http://hdl.handle.net/1721.1/92502 https://orcid.org/0000-0001-9910-5627 https://orcid.org/0000-0002-1592-5896 https://orcid.org/0000-0001-7779-2219 |
| _version_ | 1811078359137910784 |
|---|---|
| author | Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Ardila, Diego Solomon, Ethan A. Majaj, Najib J. DiCarlo, James |
| author2 | Massachusetts Institute of Technology. Institute for Medical Engineering & Science |
| author_facet | Massachusetts Institute of Technology. Institute for Medical Engineering & Science Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Ardila, Diego Solomon, Ethan A. Majaj, Najib J. DiCarlo, James |
| author_sort | Cadieu, Charles |
| collection | MIT |
| description | The primate visual system achieves remarkable visual object recognition performance even in brief presentations, and under changes to object exemplar, geometric transformations, and background variation (a.k.a. core visual object recognition). This remarkable performance is mediated by the representation formed in inferior temporal (IT) cortex. In parallel, recent advances in machine learning have led to ever higher performing models of object recognition using artificial deep neural networks (DNNs). It remains unclear, however, whether the representational performance of DNNs rivals that of the brain. To accurately produce such a comparison, a major difficulty has been a unifying metric that accounts for experimental limitations, such as the amount of noise, the number of neural recording sites, and the number of trials, and computational limitations, such as the complexity of the decoding classifier and the number of classifier training examples. In this work, we perform a direct comparison that corrects for these experimental limitations and computational considerations. As part of our methodology, we propose an extension of “kernel analysis” that measures the generalization accuracy as a function of representational complexity. Our evaluations show that, unlike previous bio-inspired models, the latest DNNs rival the representational performance of IT cortex on this visual object recognition task. Furthermore, we show that models that perform well on measures of representational performance also perform well on measures of representational similarity to IT, and on measures of predicting individual IT multi-unit responses. Whether these DNNs rely on computational mechanisms similar to the primate visual system is yet to be determined, but, unlike all previous bio-inspired models, that possibility cannot be ruled out merely on representational performance grounds. |
| first_indexed | 2024-09-23T10:58:16Z |
| format | Article |
| id | mit-1721.1/92502 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T10:58:16Z |
| publishDate | 2014 |
| publisher | Public Library of Science |
| record_format | dspace |
| spelling | mit-1721.1/925022022-10-01T00:17:11Z Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Ardila, Diego Solomon, Ethan A. Majaj, Najib J. DiCarlo, James Massachusetts Institute of Technology. Institute for Medical Engineering & Science Harvard University--MIT Division of Health Sciences and Technology Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences McGovern Institute for Brain Research at MIT Hong, Ha Cadieu, Charles Yamins, Daniel L. K. Pinto, Nicolas Ardila, Diego Solomon, Ethan A. Majaj, Najib J. DiCarlo, James The primate visual system achieves remarkable visual object recognition performance even in brief presentations, and under changes to object exemplar, geometric transformations, and background variation (a.k.a. core visual object recognition). This remarkable performance is mediated by the representation formed in inferior temporal (IT) cortex. In parallel, recent advances in machine learning have led to ever higher performing models of object recognition using artificial deep neural networks (DNNs). It remains unclear, however, whether the representational performance of DNNs rivals that of the brain. To accurately produce such a comparison, a major difficulty has been a unifying metric that accounts for experimental limitations, such as the amount of noise, the number of neural recording sites, and the number of trials, and computational limitations, such as the complexity of the decoding classifier and the number of classifier training examples. In this work, we perform a direct comparison that corrects for these experimental limitations and computational considerations. As part of our methodology, we propose an extension of “kernel analysis” that measures the generalization accuracy as a function of representational complexity. Our evaluations show that, unlike previous bio-inspired models, the latest DNNs rival the representational performance of IT cortex on this visual object recognition task. Furthermore, we show that models that perform well on measures of representational performance also perform well on measures of representational similarity to IT, and on measures of predicting individual IT multi-unit responses. Whether these DNNs rely on computational mechanisms similar to the primate visual system is yet to be determined, but, unlike all previous bio-inspired models, that possibility cannot be ruled out merely on representational performance grounds. National Eye Institute (NIH NEI: 5R01EY014970-09) National Science Foundation (U.S.) (NSF:0964269) United States. Defense Advanced Research Projects Agency (DARPA: HR0011-10-C-0032) National Eye Institute (NIH: F32 EY022845-01) 2014-12-24T16:48:44Z 2014-12-24T16:48:44Z 2014-12 Article http://purl.org/eprint/type/JournalArticle 1553-7358 http://hdl.handle.net/1721.1/92502 Cadieu, Charles F., Ha Hong, Daniel L. K. Yamins, Nicolas Pinto, Diego Ardila, Ethan A. Solomon, Najib J. Majaj, and James J. DiCarlo. “Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition.” Edited by Matthias Bethge. PLoS Comput Biol 10, no. 12 (December 18, 2014): e1003963. https://orcid.org/0000-0001-9910-5627 https://orcid.org/0000-0002-1592-5896 https://orcid.org/0000-0001-7779-2219 en_US http://dx.doi.org/10.1371/journal.pcbi.1003963 PLoS Computational Biology Creative Commons Attribution http://creativecommons.org/licenses/by/4.0/ application/pdf Public Library of Science Public Library of Science |
| spellingShingle | Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Ardila, Diego Solomon, Ethan A. Majaj, Najib J. DiCarlo, James Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition |
| title | Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition |
| title_full | Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition |
| title_fullStr | Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition |
| title_full_unstemmed | Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition |
| title_short | Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition |
| title_sort | deep neural networks rival the representation of primate it cortex for core visual object recognition |
| url | http://hdl.handle.net/1721.1/92502 https://orcid.org/0000-0001-9910-5627 https://orcid.org/0000-0002-1592-5896 https://orcid.org/0000-0001-7779-2219 |
| work_keys_str_mv | AT cadieucharles deepneuralnetworksrivaltherepresentationofprimateitcortexforcorevisualobjectrecognition AT hongha deepneuralnetworksrivaltherepresentationofprimateitcortexforcorevisualobjectrecognition AT yaminsdaniellk deepneuralnetworksrivaltherepresentationofprimateitcortexforcorevisualobjectrecognition AT pintonicolas deepneuralnetworksrivaltherepresentationofprimateitcortexforcorevisualobjectrecognition AT ardiladiego deepneuralnetworksrivaltherepresentationofprimateitcortexforcorevisualobjectrecognition AT solomonethana deepneuralnetworksrivaltherepresentationofprimateitcortexforcorevisualobjectrecognition AT majajnajibj deepneuralnetworksrivaltherepresentationofprimateitcortexforcorevisualobjectrecognition AT dicarlojames deepneuralnetworksrivaltherepresentationofprimateitcortexforcorevisualobjectrecognition |