Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device
Current advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device (SSD), t...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-03-01
|
| Series: | Frontiers in Human Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnhum.2023.1058617/full |
| _version_ | 1828000606677106688 |
|---|---|
| author | Shira Shvadron Shira Shvadron Adi Snir Adi Snir Amber Maimon Amber Maimon Or Yizhar Or Yizhar Or Yizhar Or Yizhar Sapir Harel Sapir Harel Keinan Poradosu Keinan Poradosu Keinan Poradosu Amir Amedi Amir Amedi |
| author_facet | Shira Shvadron Shira Shvadron Adi Snir Adi Snir Amber Maimon Amber Maimon Or Yizhar Or Yizhar Or Yizhar Or Yizhar Sapir Harel Sapir Harel Keinan Poradosu Keinan Poradosu Keinan Poradosu Amir Amedi Amir Amedi |
| author_sort | Shira Shvadron |
| collection | DOAJ |
| description | Current advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device (SSD), the EyeMusic, an algorithm that converts images to sound. The EyeMusic was initially developed to allow blind individuals to create a spatial representation of information arriving from a video feed at a slow sampling rate. In this study, we aimed to use the EyeMusic for the blind areas of sighted individuals. We use it in this initial proof-of-concept study to test the ability of sighted subjects to combine visual information with surrounding auditory sonification representing visual information. Participants in this study were tasked with recognizing and adequately placing the stimuli, using sound to represent the areas outside the standard human visual field. As such, the participants were asked to report shapes’ identities as well as their spatial orientation (front/right/back/left), requiring combined visual (90° frontal) and auditory input (the remaining 270°) for the successful performance of the task (content in both vision and audition was presented in a sweeping clockwise motion around the participant). We found that participants were successful at a highly above chance level after a brief 1-h-long session of online training and one on-site training session of an average of 20 min. They could even draw a 2D representation of this image in some cases. Participants could also generalize, recognizing new shapes they were not explicitly trained on. Our findings provide an initial proof of concept indicating that sensory augmentation devices and techniques can potentially be used in combination with natural sensory information in order to expand the natural fields of sensory perception. |
| first_indexed | 2024-04-10T06:18:03Z |
| format | Article |
| id | doaj.art-0cf1ccf45c0d4626a992beef3b92425e |
| institution | Directory Open Access Journal |
| issn | 1662-5161 |
| language | English |
| last_indexed | 2024-04-10T06:18:03Z |
| publishDate | 2023-03-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Human Neuroscience |
| spelling | doaj.art-0cf1ccf45c0d4626a992beef3b92425e2023-03-02T05:36:39ZengFrontiers Media S.A.Frontiers in Human Neuroscience1662-51612023-03-011710.3389/fnhum.2023.10586171058617Shape detection beyond the visual field using a visual-to-auditory sensory augmentation deviceShira Shvadron0Shira Shvadron1Adi Snir2Adi Snir3Amber Maimon4Amber Maimon5Or Yizhar6Or Yizhar7Or Yizhar8Or Yizhar9Sapir Harel10Sapir Harel11Keinan Poradosu12Keinan Poradosu13Keinan Poradosu14Amir Amedi15Amir Amedi16Baruch Ivcher School of Psychology, The Baruch Ivcher Institute for Brain, Cognition, and Technology, Reichman University, Herzliya, IsraelThe Ruth and Meir Rosenthal, Brain Imaging Center, Reichman University, Herzliya, IsraelBaruch Ivcher School of Psychology, The Baruch Ivcher Institute for Brain, Cognition, and Technology, Reichman University, Herzliya, IsraelThe Ruth and Meir Rosenthal, Brain Imaging Center, Reichman University, Herzliya, IsraelBaruch Ivcher School of Psychology, The Baruch Ivcher Institute for Brain, Cognition, and Technology, Reichman University, Herzliya, IsraelThe Ruth and Meir Rosenthal, Brain Imaging Center, Reichman University, Herzliya, IsraelBaruch Ivcher School of Psychology, The Baruch Ivcher Institute for Brain, Cognition, and Technology, Reichman University, Herzliya, IsraelThe Ruth and Meir Rosenthal, Brain Imaging Center, Reichman University, Herzliya, IsraelResearch Group Adaptive Memory and Decision Making, Max Planck Institute for Human Development, Berlin, GermanyMax Planck Dahlem Campus of Cognition (MPDCC), Max Planck Institute for Human Development, Berlin, GermanyBaruch Ivcher School of Psychology, The Baruch Ivcher Institute for Brain, Cognition, and Technology, Reichman University, Herzliya, IsraelThe Ruth and Meir Rosenthal, Brain Imaging Center, Reichman University, Herzliya, IsraelBaruch Ivcher School of Psychology, The Baruch Ivcher Institute for Brain, Cognition, and Technology, Reichman University, Herzliya, IsraelThe Ruth and Meir Rosenthal, Brain Imaging Center, Reichman University, Herzliya, IsraelWeizmann Institute of Science, Rehovot, IsraelBaruch Ivcher School of Psychology, The Baruch Ivcher Institute for Brain, Cognition, and Technology, Reichman University, Herzliya, IsraelThe Ruth and Meir Rosenthal, Brain Imaging Center, Reichman University, Herzliya, IsraelCurrent advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device (SSD), the EyeMusic, an algorithm that converts images to sound. The EyeMusic was initially developed to allow blind individuals to create a spatial representation of information arriving from a video feed at a slow sampling rate. In this study, we aimed to use the EyeMusic for the blind areas of sighted individuals. We use it in this initial proof-of-concept study to test the ability of sighted subjects to combine visual information with surrounding auditory sonification representing visual information. Participants in this study were tasked with recognizing and adequately placing the stimuli, using sound to represent the areas outside the standard human visual field. As such, the participants were asked to report shapes’ identities as well as their spatial orientation (front/right/back/left), requiring combined visual (90° frontal) and auditory input (the remaining 270°) for the successful performance of the task (content in both vision and audition was presented in a sweeping clockwise motion around the participant). We found that participants were successful at a highly above chance level after a brief 1-h-long session of online training and one on-site training session of an average of 20 min. They could even draw a 2D representation of this image in some cases. Participants could also generalize, recognizing new shapes they were not explicitly trained on. Our findings provide an initial proof of concept indicating that sensory augmentation devices and techniques can potentially be used in combination with natural sensory information in order to expand the natural fields of sensory perception.https://www.frontiersin.org/articles/10.3389/fnhum.2023.1058617/fullspatial perceptionvisual-auditorysensory substitutionsensory substitution device (SSD)visual-spatial perceptionauditory spatial perception |
| spellingShingle | Shira Shvadron Shira Shvadron Adi Snir Adi Snir Amber Maimon Amber Maimon Or Yizhar Or Yizhar Or Yizhar Or Yizhar Sapir Harel Sapir Harel Keinan Poradosu Keinan Poradosu Keinan Poradosu Amir Amedi Amir Amedi Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device Frontiers in Human Neuroscience spatial perception visual-auditory sensory substitution sensory substitution device (SSD) visual-spatial perception auditory spatial perception |
| title | Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device |
| title_full | Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device |
| title_fullStr | Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device |
| title_full_unstemmed | Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device |
| title_short | Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device |
| title_sort | shape detection beyond the visual field using a visual to auditory sensory augmentation device |
| topic | spatial perception visual-auditory sensory substitution sensory substitution device (SSD) visual-spatial perception auditory spatial perception |
| url | https://www.frontiersin.org/articles/10.3389/fnhum.2023.1058617/full |
| work_keys_str_mv | AT shirashvadron shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT shirashvadron shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT adisnir shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT adisnir shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT ambermaimon shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT ambermaimon shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT oryizhar shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT oryizhar shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT oryizhar shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT oryizhar shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT sapirharel shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT sapirharel shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT keinanporadosu shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT keinanporadosu shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT keinanporadosu shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT amiramedi shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice AT amiramedi shapedetectionbeyondthevisualfieldusingavisualtoauditorysensoryaugmentationdevice |