Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance
Object sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whethe...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-09-01
|
| Series: | Brain Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3425/11/9/1206 |
| _version_ | 1797520054469787648 |
|---|---|
| author | Erika Almadori Serena Mastroberardino Fabiano Botta Riccardo Brunetti Juan Lupiáñez Charles Spence Valerio Santangelo |
| author_facet | Erika Almadori Serena Mastroberardino Fabiano Botta Riccardo Brunetti Juan Lupiáñez Charles Spence Valerio Santangelo |
| author_sort | Erika Almadori |
| collection | DOAJ |
| description | Object sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whether this effect is modulated by the object consistency with the background visual scene. In two experiments, participants viewed everyday visual scenes for 500 ms while listening to an object sound, which could either be semantically related to the object that served as the STM target at retrieval or not. This defined crossmodal semantically cued vs. uncued targets. The target was either in- or out-of-context with respect to the background visual scene. After a maintenance period of 2000 ms, the target was presented in isolation against a neutral background, in either the same or different spatial position as in the original scene. The participants judged the same vs. different position of the object and then provided a confidence judgment concerning the certainty of their response. The results revealed greater accuracy when judging the spatial position of targets paired with a semantically congruent object sound at encoding. This crossmodal facilitatory effect was modulated by whether the target object was in- or out-of-context with respect to the background scene, with out-of-context targets reducing the facilitatory effect of object sounds. Overall, these findings suggest that the presence of the object sound at encoding facilitated the selection and processing of the semantically related visual stimuli, but this effect depends on the semantic configuration of the visual scene. |
| first_indexed | 2024-03-10T07:51:25Z |
| format | Article |
| id | doaj.art-899e85f6e1d94d1a82d70987cc8ca4a1 |
| institution | Directory Open Access Journal |
| issn | 2076-3425 |
| language | English |
| last_indexed | 2024-03-10T07:51:25Z |
| publishDate | 2021-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Brain Sciences |
| spelling | doaj.art-899e85f6e1d94d1a82d70987cc8ca4a12023-11-22T12:14:40ZengMDPI AGBrain Sciences2076-34252021-09-01119120610.3390/brainsci11091206Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory PerformanceErika Almadori0Serena Mastroberardino1Fabiano Botta2Riccardo Brunetti3Juan Lupiáñez4Charles Spence5Valerio Santangelo6Neuroimaging Laboratory, IRCCS Santa Lucia Foundation, Via Ardeatina 306, 00179 Rome, ItalyDepartment of Psychology, School of Medicine & Psychology, Sapienza University of Rome, Via dei Marsi 78, 00185 Rome, ItalyDepartment of Experimental Psychology and Mind, Brain, and Behavior Research Center (CIMCYC), University of Granada, 18071 Granada, SpainCognitive and Clinical Psychology Laboratory, Department of Human Sciences, Università Europea di Roma, 00163 Roma, ItalyDepartment of Experimental Psychology and Mind, Brain, and Behavior Research Center (CIMCYC), University of Granada, 18071 Granada, SpainDepartment of Experimental Psychology, Oxford University, Oxford OX2 6GG, UKNeuroimaging Laboratory, IRCCS Santa Lucia Foundation, Via Ardeatina 306, 00179 Rome, ItalyObject sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whether this effect is modulated by the object consistency with the background visual scene. In two experiments, participants viewed everyday visual scenes for 500 ms while listening to an object sound, which could either be semantically related to the object that served as the STM target at retrieval or not. This defined crossmodal semantically cued vs. uncued targets. The target was either in- or out-of-context with respect to the background visual scene. After a maintenance period of 2000 ms, the target was presented in isolation against a neutral background, in either the same or different spatial position as in the original scene. The participants judged the same vs. different position of the object and then provided a confidence judgment concerning the certainty of their response. The results revealed greater accuracy when judging the spatial position of targets paired with a semantically congruent object sound at encoding. This crossmodal facilitatory effect was modulated by whether the target object was in- or out-of-context with respect to the background scene, with out-of-context targets reducing the facilitatory effect of object sounds. Overall, these findings suggest that the presence of the object sound at encoding facilitated the selection and processing of the semantically related visual stimuli, but this effect depends on the semantic configuration of the visual scene.https://www.mdpi.com/2076-3425/11/9/1206short-term memorysemanticsvisual scenesauditorymultisensoryspatial orienting |
| spellingShingle | Erika Almadori Serena Mastroberardino Fabiano Botta Riccardo Brunetti Juan Lupiáñez Charles Spence Valerio Santangelo Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance Brain Sciences short-term memory semantics visual scenes auditory multisensory spatial orienting |
| title | Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance |
| title_full | Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance |
| title_fullStr | Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance |
| title_full_unstemmed | Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance |
| title_short | Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance |
| title_sort | crossmodal semantic congruence interacts with object contextual consistency in complex visual scenes to enhance short term memory performance |
| topic | short-term memory semantics visual scenes auditory multisensory spatial orienting |
| url | https://www.mdpi.com/2076-3425/11/9/1206 |
| work_keys_str_mv | AT erikaalmadori crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance AT serenamastroberardino crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance AT fabianobotta crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance AT riccardobrunetti crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance AT juanlupianez crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance AT charlesspence crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance AT valeriosantangelo crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance |