Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model
This paper provides new research perspectives in the field of multimodal comprehension (auditory crossing visual information) by using immersion and incorporating eye tracking in a virtual reality environment. The objective is to investigate the influence of a change in narrative perspective (point...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-06-01
|
| Series: | Frontiers in Virtual Reality |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frvir.2022.874054/full |
| _version_ | 1818251298170994688 |
|---|---|
| author | Véronique Drai-Zerbib Léa Bernigaud Alexandre Gaston-Bellegarde Jean-Michel Boucheix Thierry Baccino |
| author_facet | Véronique Drai-Zerbib Léa Bernigaud Alexandre Gaston-Bellegarde Jean-Michel Boucheix Thierry Baccino |
| author_sort | Véronique Drai-Zerbib |
| collection | DOAJ |
| description | This paper provides new research perspectives in the field of multimodal comprehension (auditory crossing visual information) by using immersion and incorporating eye tracking in a virtual reality environment. The objective is to investigate the influence of a change in narrative perspective (point of view) during the activation of a mental model underlying comprehension between visual and auditory modalities. Twenty-eight participants, equipped with a headset SMI HMD HTC eye-tracking 250 Hz watched 16 visual scenes in virtual reality accompanied by their corresponding auditory narration. The change in perspective may occur either in the visual scenes or in listening. Mean fixations durations on typical objects of the visual scenes (Area of Interest) that were related to the perspective shift were analyzed as well as the free recall of narratives. We split each scene into three periods according to different parts of the narration (Before, Target, After), the target was where a shift in perspective could occur. Results shown that when a visual change of perspective occurred, mean fixation duration was shorter (compared to no change) for both Target and After. However, when auditory change of perspective occurred, no difference was found on Target, although during After, mean fixation duration was longer (compared to no change). In the context of 3D video visualization, it seems that auditory processing prevails over visual processing of verbal information: The visual change of perspective induces less visual processing of the Area of Interest (AOIs) included in the visual scene, but the auditory change in perspective leads to increased visual processing of the visual scene. Moreover, the analysis showed higher recall of information (verbatim and paraphrase) when an auditory change in perspective was coupled with no visual change of perspective. Thus, our results indicate a more effective integration of information when there is an inconsistency between the narration heard and viewed. A change in perspective, instead of creating comprehension and integration difficulties, seems to effectively raise the attention and induce a shorter visual inspection. These results are discussed in the context of cross-modal comprehension. |
| first_indexed | 2024-12-12T16:06:03Z |
| format | Article |
| id | doaj.art-58f6ae7fa71d474e9b07e2c1d15eca57 |
| institution | Directory Open Access Journal |
| issn | 2673-4192 |
| language | English |
| last_indexed | 2024-12-12T16:06:03Z |
| publishDate | 2022-06-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Virtual Reality |
| spelling | doaj.art-58f6ae7fa71d474e9b07e2c1d15eca572022-12-22T00:19:17ZengFrontiers Media S.A.Frontiers in Virtual Reality2673-41922022-06-01310.3389/frvir.2022.874054874054Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental ModelVéronique Drai-Zerbib0Léa Bernigaud1Alexandre Gaston-Bellegarde2Jean-Michel Boucheix3Thierry Baccino4Laboratory LEAD-CNRS, UMR5022, Université Bourgogne, Dijon, FranceLaboratory LEAD-CNRS, UMR5022, Université Bourgogne, Dijon, FranceLaboratory LEAD-CNRS, UMR5022, Université Bourgogne, Dijon, FranceLaboratory LEAD-CNRS, UMR5022, Université Bourgogne, Dijon, FranceLUTIN, University Paris 8, Paris, FranceThis paper provides new research perspectives in the field of multimodal comprehension (auditory crossing visual information) by using immersion and incorporating eye tracking in a virtual reality environment. The objective is to investigate the influence of a change in narrative perspective (point of view) during the activation of a mental model underlying comprehension between visual and auditory modalities. Twenty-eight participants, equipped with a headset SMI HMD HTC eye-tracking 250 Hz watched 16 visual scenes in virtual reality accompanied by their corresponding auditory narration. The change in perspective may occur either in the visual scenes or in listening. Mean fixations durations on typical objects of the visual scenes (Area of Interest) that were related to the perspective shift were analyzed as well as the free recall of narratives. We split each scene into three periods according to different parts of the narration (Before, Target, After), the target was where a shift in perspective could occur. Results shown that when a visual change of perspective occurred, mean fixation duration was shorter (compared to no change) for both Target and After. However, when auditory change of perspective occurred, no difference was found on Target, although during After, mean fixation duration was longer (compared to no change). In the context of 3D video visualization, it seems that auditory processing prevails over visual processing of verbal information: The visual change of perspective induces less visual processing of the Area of Interest (AOIs) included in the visual scene, but the auditory change in perspective leads to increased visual processing of the visual scene. Moreover, the analysis showed higher recall of information (verbatim and paraphrase) when an auditory change in perspective was coupled with no visual change of perspective. Thus, our results indicate a more effective integration of information when there is an inconsistency between the narration heard and viewed. A change in perspective, instead of creating comprehension and integration difficulties, seems to effectively raise the attention and induce a shorter visual inspection. These results are discussed in the context of cross-modal comprehension.https://www.frontiersin.org/articles/10.3389/frvir.2022.874054/fullvirtual realityeye trackingcomprehensionmultimodalityshift in narrative perspective |
| spellingShingle | Véronique Drai-Zerbib Léa Bernigaud Alexandre Gaston-Bellegarde Jean-Michel Boucheix Thierry Baccino Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model Frontiers in Virtual Reality virtual reality eye tracking comprehension multimodality shift in narrative perspective |
| title | Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model |
| title_full | Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model |
| title_fullStr | Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model |
| title_full_unstemmed | Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model |
| title_short | Eye Movements During Comprehension in Virtual Reality: The Influence of a Change in Point of View Between Auditory and Visual Information in the Activation of a Mental Model |
| title_sort | eye movements during comprehension in virtual reality the influence of a change in point of view between auditory and visual information in the activation of a mental model |
| topic | virtual reality eye tracking comprehension multimodality shift in narrative perspective |
| url | https://www.frontiersin.org/articles/10.3389/frvir.2022.874054/full |
| work_keys_str_mv | AT veroniquedraizerbib eyemovementsduringcomprehensioninvirtualrealitytheinfluenceofachangeinpointofviewbetweenauditoryandvisualinformationintheactivationofamentalmodel AT leabernigaud eyemovementsduringcomprehensioninvirtualrealitytheinfluenceofachangeinpointofviewbetweenauditoryandvisualinformationintheactivationofamentalmodel AT alexandregastonbellegarde eyemovementsduringcomprehensioninvirtualrealitytheinfluenceofachangeinpointofviewbetweenauditoryandvisualinformationintheactivationofamentalmodel AT jeanmichelboucheix eyemovementsduringcomprehensioninvirtualrealitytheinfluenceofachangeinpointofviewbetweenauditoryandvisualinformationintheactivationofamentalmodel AT thierrybaccino eyemovementsduringcomprehensioninvirtualrealitytheinfluenceofachangeinpointofviewbetweenauditoryandvisualinformationintheactivationofamentalmodel |