What am I searching for?
Can we infer intentions and goals from a person's actions? As an example of this family of problems, we consider here whether it is possible to decipher what a person is searching for by decoding their eye movement behavior. We conducted two human psychophysics experiments on object arrays and...
| Main Authors: | , , , , |
|---|---|
| Format: | Technical Report |
| Language: | en_US |
| Published: |
Center for Brains, Minds and Machines (CBMM), arXiv.org
2018
|
| Online Access: | http://hdl.handle.net/1721.1/119576 |
| _version_ | 1811080062186815488 |
|---|---|
| author | Zhang, Mengmi Feng, Jiashi Lim, Joo Hwee Zhao, Qi Kreiman, Gabriel |
| author_facet | Zhang, Mengmi Feng, Jiashi Lim, Joo Hwee Zhao, Qi Kreiman, Gabriel |
| author_sort | Zhang, Mengmi |
| collection | MIT |
| description | Can we infer intentions and goals from a person's actions? As an example of this family of problems, we consider here whether it is possible to decipher what a person is searching for by decoding their eye movement behavior. We conducted two human psychophysics experiments on object arrays and natural images where we monitored subjects' eye movements while they were looking for a target object. Using as input the pattern of "error" fixations on non-target objects before the target was found, we developed a model (InferNet) whose goal was to infer what the target was. "Error" fixations share similar features with the sought target. The Infernet model uses a pre-trained 2D convolutional architecture to extract features from the error fixations and computes a 2D similarity map between the error fixation and all locations across the search image by modulating the search image via convolution across layers. InferNet consolidates the modulated response maps across layers via max pooling to keep track of the sub-patterns highly similar to features at error fixations and integrates these maps across all error fixations. InferNet successfully identifies the subject's goal and outperforms all the competitive null models, even without any object-specific training on the inference task. |
| first_indexed | 2024-09-23T11:25:09Z |
| format | Technical Report |
| id | mit-1721.1/119576 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T11:25:09Z |
| publishDate | 2018 |
| publisher | Center for Brains, Minds and Machines (CBMM), arXiv.org |
| record_format | dspace |
| spelling | mit-1721.1/1195762019-04-11T02:26:01Z What am I searching for? Zhang, Mengmi Feng, Jiashi Lim, Joo Hwee Zhao, Qi Kreiman, Gabriel Can we infer intentions and goals from a person's actions? As an example of this family of problems, we consider here whether it is possible to decipher what a person is searching for by decoding their eye movement behavior. We conducted two human psychophysics experiments on object arrays and natural images where we monitored subjects' eye movements while they were looking for a target object. Using as input the pattern of "error" fixations on non-target objects before the target was found, we developed a model (InferNet) whose goal was to infer what the target was. "Error" fixations share similar features with the sought target. The Infernet model uses a pre-trained 2D convolutional architecture to extract features from the error fixations and computes a 2D similarity map between the error fixation and all locations across the search image by modulating the search image via convolution across layers. InferNet consolidates the modulated response maps across layers via max pooling to keep track of the sub-patterns highly similar to features at error fixations and integrates these maps across all error fixations. InferNet successfully identifies the subject's goal and outperforms all the competitive null models, even without any object-specific training on the inference task. This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF – 1231216. 2018-12-11T20:40:45Z 2018-12-11T20:40:45Z 2018-07-31 Technical Report Working Paper Other http://hdl.handle.net/1721.1/119576 en_US CBMM Memo Series;096 application/pdf Center for Brains, Minds and Machines (CBMM), arXiv.org |
| spellingShingle | Zhang, Mengmi Feng, Jiashi Lim, Joo Hwee Zhao, Qi Kreiman, Gabriel What am I searching for? |
| title | What am I searching for? |
| title_full | What am I searching for? |
| title_fullStr | What am I searching for? |
| title_full_unstemmed | What am I searching for? |
| title_short | What am I searching for? |
| title_sort | what am i searching for |
| url | http://hdl.handle.net/1721.1/119576 |
| work_keys_str_mv | AT zhangmengmi whatamisearchingfor AT fengjiashi whatamisearchingfor AT limjoohwee whatamisearchingfor AT zhaoqi whatamisearchingfor AT kreimangabriel whatamisearchingfor |