Atoms of recognition in human and computer vision
Discovering the visual features and representations used by the brain to recognize objects is a central problem in the study of vision. Recently, neural network models of visual object recognition, including biological and deep network models, have shown remarkable progress and have begun to rival h...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
National Academy of Sciences (U.S.)
2017
|
| Online Access: | http://hdl.handle.net/1721.1/106502 https://orcid.org/0000-0003-4331-298X https://orcid.org/0000-0003-4745-9292 |
| _version_ | 1826209206497181696 |
|---|---|
| author | Ullman, Shimon Assif, Liav Fetaya, Ethan Harari, Daniel |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Ullman, Shimon Assif, Liav Fetaya, Ethan Harari, Daniel |
| author_sort | Ullman, Shimon |
| collection | MIT |
| description | Discovering the visual features and representations used by the brain to recognize objects is a central problem in the study of vision. Recently, neural network models of visual object recognition, including biological and deep network models, have shown remarkable progress and have begun to rival human performance in some challenging tasks. These models are trained on image examples and learn to extract features and representations and to use them for categorization. It remains unclear, however, whether the representations and learning processes discovered by current models are similar to those used by the human visual system. Here we show, by introducing and using minimal recognizable images, that the human visual system uses features and processes that are not used by current models and that are critical for recognition. We found by psychophysical studies that at the level of minimal recognizable images a minute change in the image can have a drastic effect on recognition, thus identifying features that are critical for the task. Simulations then showed that current models cannot explain this sensitivity to precise feature configurations and, more generally, do not learn to recognize minimal images at a human level. The role of the features shown here is revealed uniquely at the minimal level, where the contribution of each feature is essential. A full understanding of the learning and use of such features will extend our understanding of visual recognition and its cortical mechanisms and will enhance the capacity of computational models to learn from visual experience and to deal with recognition and detailed image interpretation. |
| first_indexed | 2024-09-23T14:18:52Z |
| format | Article |
| id | mit-1721.1/106502 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T14:18:52Z |
| publishDate | 2017 |
| publisher | National Academy of Sciences (U.S.) |
| record_format | dspace |
| spelling | mit-1721.1/1065022022-10-01T20:33:56Z Atoms of recognition in human and computer vision Ullman, Shimon Assif, Liav Fetaya, Ethan Harari, Daniel Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences McGovern Institute for Brain Research at MIT Ullman, Shimon Harari, Daniel Discovering the visual features and representations used by the brain to recognize objects is a central problem in the study of vision. Recently, neural network models of visual object recognition, including biological and deep network models, have shown remarkable progress and have begun to rival human performance in some challenging tasks. These models are trained on image examples and learn to extract features and representations and to use them for categorization. It remains unclear, however, whether the representations and learning processes discovered by current models are similar to those used by the human visual system. Here we show, by introducing and using minimal recognizable images, that the human visual system uses features and processes that are not used by current models and that are critical for recognition. We found by psychophysical studies that at the level of minimal recognizable images a minute change in the image can have a drastic effect on recognition, thus identifying features that are critical for the task. Simulations then showed that current models cannot explain this sensitivity to precise feature configurations and, more generally, do not learn to recognize minimal images at a human level. The role of the features shown here is revealed uniquely at the minimal level, where the contribution of each feature is essential. A full understanding of the learning and use of such features will extend our understanding of visual recognition and its cortical mechanisms and will enhance the capacity of computational models to learn from visual experience and to deal with recognition and detailed image interpretation. European Research Council (Advanced Grant “Digital Baby”) National Science Foundation (U.S.) (STC Center for Brains, Minds and Machines Award CCF-1231216) 2017-01-17T15:21:33Z 2017-01-17T15:21:33Z 2016-02 2015-01 Article http://purl.org/eprint/type/JournalArticle 0027-8424 1091-6490 http://hdl.handle.net/1721.1/106502 Ullman, Shimon et al. “Atoms of Recognition in Human and Computer Vision.” Proceedings of the National Academy of Sciences 113.10 (2016): 2744–2749. © 2016 National Academy of Sciences https://orcid.org/0000-0003-4331-298X https://orcid.org/0000-0003-4745-9292 en_US http://dx.doi.org/10.1073/pnas.1513198113 Proceedings of the National Academy of Sciences Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf National Academy of Sciences (U.S.) PNAS |
| spellingShingle | Ullman, Shimon Assif, Liav Fetaya, Ethan Harari, Daniel Atoms of recognition in human and computer vision |
| title | Atoms of recognition in human and computer vision |
| title_full | Atoms of recognition in human and computer vision |
| title_fullStr | Atoms of recognition in human and computer vision |
| title_full_unstemmed | Atoms of recognition in human and computer vision |
| title_short | Atoms of recognition in human and computer vision |
| title_sort | atoms of recognition in human and computer vision |
| url | http://hdl.handle.net/1721.1/106502 https://orcid.org/0000-0003-4331-298X https://orcid.org/0000-0003-4745-9292 |
| work_keys_str_mv | AT ullmanshimon atomsofrecognitioninhumanandcomputervision AT assifliav atomsofrecognitioninhumanandcomputervision AT fetayaethan atomsofrecognitioninhumanandcomputervision AT hararidaniel atomsofrecognitioninhumanandcomputervision |