Connecting Deep Learning Models to the Human Brain
In this thesis, we introduce innovative methodologies for connecting new deep learning models, particularly models that integrate vision and language with human brain processing. These models have shown remarkable advancements in tasks such as object recognition, scene classification, and language p...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2024
|
| Online Access: | https://hdl.handle.net/1721.1/156797 |
| _version_ | 1811088714284138496 |
|---|---|
| author | Subramaniam, Vighnesh |
| author2 | Katz, Boris |
| author_facet | Katz, Boris Subramaniam, Vighnesh |
| author_sort | Subramaniam, Vighnesh |
| collection | MIT |
| description | In this thesis, we introduce innovative methodologies for connecting new deep learning models, particularly models that integrate vision and language with human brain processing. These models have shown remarkable advancements in tasks such as object recognition, scene classification, and language processing, achieving near-human accuracy in some cases. This raises intriguing questions about how closely the computations and geometric structure of these models mirror that of the human brain. Our method starts with measuring brain activity in response to vision and language stimuli and then exposes these stimuli to deep learning models to collect their internal activations. We analyze the similarity between these activations and brain activity using a specific representational distance metric. We focus on introducing statistical algorithms to assess whether one model is significantly more similar with the brain than another. Through our novel methodology, we assess whether there’s a more significant correlation between brain regions and multimodal models compared to unimodal ones. Our investigation reveals brain areas associated with vision-language integration and models of vision-language integration that are potentially most similar to the brain. |
| first_indexed | 2024-09-23T14:06:23Z |
| format | Thesis |
| id | mit-1721.1/156797 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T14:06:23Z |
| publishDate | 2024 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1567972024-09-17T03:24:36Z Connecting Deep Learning Models to the Human Brain Subramaniam, Vighnesh Katz, Boris Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science In this thesis, we introduce innovative methodologies for connecting new deep learning models, particularly models that integrate vision and language with human brain processing. These models have shown remarkable advancements in tasks such as object recognition, scene classification, and language processing, achieving near-human accuracy in some cases. This raises intriguing questions about how closely the computations and geometric structure of these models mirror that of the human brain. Our method starts with measuring brain activity in response to vision and language stimuli and then exposes these stimuli to deep learning models to collect their internal activations. We analyze the similarity between these activations and brain activity using a specific representational distance metric. We focus on introducing statistical algorithms to assess whether one model is significantly more similar with the brain than another. Through our novel methodology, we assess whether there’s a more significant correlation between brain regions and multimodal models compared to unimodal ones. Our investigation reveals brain areas associated with vision-language integration and models of vision-language integration that are potentially most similar to the brain. M.Eng. 2024-09-16T13:49:50Z 2024-09-16T13:49:50Z 2024-05 2024-07-11T14:36:59.760Z Thesis https://hdl.handle.net/1721.1/156797 Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) Copyright retained by author(s) https://creativecommons.org/licenses/by-nc-nd/4.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Subramaniam, Vighnesh Connecting Deep Learning Models to the Human Brain |
| title | Connecting Deep Learning Models to the Human Brain |
| title_full | Connecting Deep Learning Models to the Human Brain |
| title_fullStr | Connecting Deep Learning Models to the Human Brain |
| title_full_unstemmed | Connecting Deep Learning Models to the Human Brain |
| title_short | Connecting Deep Learning Models to the Human Brain |
| title_sort | connecting deep learning models to the human brain |
| url | https://hdl.handle.net/1721.1/156797 |
| work_keys_str_mv | AT subramaniamvighnesh connectingdeeplearningmodelstothehumanbrain |