The Neural Representation Benchmark and its Evaluation on Brain and Machine
A key requirement for the development of effective learning representations is their evaluation and comparison to representations we know to be effective. In natural sensory domains, the community has viewed the brain as a source of inspiration and as an implicit benchmark for success. However, it h...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
2014
|
| Online Access: | http://hdl.handle.net/1721.1/87124 https://orcid.org/0000-0001-9910-5627 https://orcid.org/0000-0002-1592-5896 https://orcid.org/0000-0001-7779-2219 |
| _version_ | 1811089396262240256 |
|---|---|
| author | Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Majaj, Najib J. DiCarlo, James |
| author2 | Harvard University--MIT Division of Health Sciences and Technology |
| author_facet | Harvard University--MIT Division of Health Sciences and Technology Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Majaj, Najib J. DiCarlo, James |
| author_sort | Cadieu, Charles |
| collection | MIT |
| description | A key requirement for the development of effective learning representations is their evaluation and comparison to representations we know to be effective. In natural sensory domains, the community has viewed the brain as a source of inspiration and as an implicit benchmark for success. However, it has not been possible to directly test representational learning algorithms directly against the representations contained in neural systems. Here, we propose a new benchmark for visual representations on which we have directly tested the neural representation in multiple visual cortical areas in macaque (utilizing data from [Majaj et al., 2012]), and on which any computer vision algorithm that produces a feature space can be tested. The benchmark measures the effectiveness of the neural or machine representation by computing the classification loss on the ordered eigendecomposition of a kernel matrix [Montavon et al., 2011]. In our analysis we find that the neural representation in visual area IT is superior to visual area V4. In our analysis of representational learning algorithms, we find that three-layer models approach the representational performance of V4 and the algorithm in [Le et al., 2012] surpasses the performance of V4. Impressively, we find that a recent supervised algorithm [Krizhevsky et al., 2012] achieves performance comparable to that of IT for an intermediate level of image variation difficulty, and surpasses IT at a higher difficulty level. We believe this result represents a major milestone: it is the first learning algorithm we have found that exceeds our current estimate of IT representation performance. We hope that this benchmark will assist the community in matching the representational performance of visual cortex and will serve as an initial rallying point for further correspondence between representations derived in brains and machines. |
| first_indexed | 2024-09-23T14:18:30Z |
| format | Article |
| id | mit-1721.1/87124 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T14:18:30Z |
| publishDate | 2014 |
| record_format | dspace |
| spelling | mit-1721.1/871242022-10-01T20:29:49Z The Neural Representation Benchmark and its Evaluation on Brain and Machine Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Majaj, Najib J. DiCarlo, James Harvard University--MIT Division of Health Sciences and Technology Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences McGovern Institute for Brain Research at MIT Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Majaj, Najib J. DiCarlo, James A key requirement for the development of effective learning representations is their evaluation and comparison to representations we know to be effective. In natural sensory domains, the community has viewed the brain as a source of inspiration and as an implicit benchmark for success. However, it has not been possible to directly test representational learning algorithms directly against the representations contained in neural systems. Here, we propose a new benchmark for visual representations on which we have directly tested the neural representation in multiple visual cortical areas in macaque (utilizing data from [Majaj et al., 2012]), and on which any computer vision algorithm that produces a feature space can be tested. The benchmark measures the effectiveness of the neural or machine representation by computing the classification loss on the ordered eigendecomposition of a kernel matrix [Montavon et al., 2011]. In our analysis we find that the neural representation in visual area IT is superior to visual area V4. In our analysis of representational learning algorithms, we find that three-layer models approach the representational performance of V4 and the algorithm in [Le et al., 2012] surpasses the performance of V4. Impressively, we find that a recent supervised algorithm [Krizhevsky et al., 2012] achieves performance comparable to that of IT for an intermediate level of image variation difficulty, and surpasses IT at a higher difficulty level. We believe this result represents a major milestone: it is the first learning algorithm we have found that exceeds our current estimate of IT representation performance. We hope that this benchmark will assist the community in matching the representational performance of visual cortex and will serve as an initial rallying point for further correspondence between representations derived in brains and machines. National Eye Institute (NIH NEI: 5R01EY014970-09) National Science Foundation (U.S.) (NSF: 0964269) United States. Defense Advanced Research Projects Agency (DARPA: HR0011-10-C-0032) National Eye Institute (NIH: F32 EY022845-01) 2014-05-23T16:37:33Z 2014-05-23T16:37:33Z 2013 2013 Article http://purl.org/eprint/type/ConferencePaper arXiv:1301.3530v2 http://hdl.handle.net/1721.1/87124 Charles F. Cadieu, Ha Hong, Dan Yamins, Nicolas Pinto, Najib J. Majaj, James J. DiCarlo. "The Neural Representation Benchmark and its Evaluation on Brain and Machine." https://orcid.org/0000-0001-9910-5627 https://orcid.org/0000-0002-1592-5896 https://orcid.org/0000-0001-7779-2219 en_US Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf arXiv |
| spellingShingle | Cadieu, Charles Hong, Ha Yamins, Daniel L. K. Pinto, Nicolas Majaj, Najib J. DiCarlo, James The Neural Representation Benchmark and its Evaluation on Brain and Machine |
| title | The Neural Representation Benchmark and its Evaluation on Brain and Machine |
| title_full | The Neural Representation Benchmark and its Evaluation on Brain and Machine |
| title_fullStr | The Neural Representation Benchmark and its Evaluation on Brain and Machine |
| title_full_unstemmed | The Neural Representation Benchmark and its Evaluation on Brain and Machine |
| title_short | The Neural Representation Benchmark and its Evaluation on Brain and Machine |
| title_sort | neural representation benchmark and its evaluation on brain and machine |
| url | http://hdl.handle.net/1721.1/87124 https://orcid.org/0000-0001-9910-5627 https://orcid.org/0000-0002-1592-5896 https://orcid.org/0000-0001-7779-2219 |
| work_keys_str_mv | AT cadieucharles theneuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT hongha theneuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT yaminsdaniellk theneuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT pintonicolas theneuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT majajnajibj theneuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT dicarlojames theneuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT cadieucharles neuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT hongha neuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT yaminsdaniellk neuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT pintonicolas neuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT majajnajibj neuralrepresentationbenchmarkanditsevaluationonbrainandmachine AT dicarlojames neuralrepresentationbenchmarkanditsevaluationonbrainandmachine |