A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach
In the context of assisted human, identifying and enhancing non-stationary speech targets speech in various noise environments, such as a cocktail party, is an important issue for real-time speech separation. Previous studies mostly used microphone signal processing to perform target speech separati...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-06-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/20/12/3527 |
| _version_ | 1797564549071634432 |
|---|---|
| author | Ching-Feng Liu Wei-Siang Ciou Peng-Ting Chen Yi-Chun Du |
| author_facet | Ching-Feng Liu Wei-Siang Ciou Peng-Ting Chen Yi-Chun Du |
| author_sort | Ching-Feng Liu |
| collection | DOAJ |
| description | In the context of assisted human, identifying and enhancing non-stationary speech targets speech in various noise environments, such as a cocktail party, is an important issue for real-time speech separation. Previous studies mostly used microphone signal processing to perform target speech separation and analysis, such as feature recognition through a large amount of training data and supervised machine learning. The method was suitable for stationary noise suppression, but relatively limited for non-stationary noise and difficult to meet the real-time processing requirement. In this study, we propose a real-time speech separation method based on an approach that combines an optical camera and a microphone array. The method was divided into two stages. Stage 1 used computer vision technology with the camera to detect and identify interest targets and evaluate source angles and distance. Stage 2 used beamforming technology with microphone array to enhance and separate the target speech sound. The asynchronous update function was utilized to integrate the beamforming control and speech processing to reduce the effect of the processing delay. The experimental results show that the noise reduction in various stationary and non-stationary noise environments were 6.1 dB and 5.2 dB respectively. The response time of speech processing was less than 10ms, which meets the requirements of a real-time system. The proposed method has high potential to be applied in auxiliary listening systems or machine language processing like intelligent personal assistant. |
| first_indexed | 2024-03-10T18:58:44Z |
| format | Article |
| id | doaj.art-dc2dc2d89672401293cd7f65574efbbb |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T18:58:44Z |
| publishDate | 2020-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-dc2dc2d89672401293cd7f65574efbbb2023-11-20T04:34:29ZengMDPI AGSensors1424-82202020-06-012012352710.3390/s20123527A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion ApproachChing-Feng Liu0Wei-Siang Ciou1Peng-Ting Chen2Yi-Chun Du3Department of Electrical Engineering, Southern Taiwan University of Science and Technology, Tainan 71005, TaiwanDepartment of Electrical Engineering, Southern Taiwan University of Science and Technology, Tainan 71005, TaiwanDepartment of Biomedical Engineering & Medical Device Innovation Center, National Cheng Kung University, Tainan 70105, TaiwanDepartment of Electrical Engineering, Southern Taiwan University of Science and Technology, Tainan 71005, TaiwanIn the context of assisted human, identifying and enhancing non-stationary speech targets speech in various noise environments, such as a cocktail party, is an important issue for real-time speech separation. Previous studies mostly used microphone signal processing to perform target speech separation and analysis, such as feature recognition through a large amount of training data and supervised machine learning. The method was suitable for stationary noise suppression, but relatively limited for non-stationary noise and difficult to meet the real-time processing requirement. In this study, we propose a real-time speech separation method based on an approach that combines an optical camera and a microphone array. The method was divided into two stages. Stage 1 used computer vision technology with the camera to detect and identify interest targets and evaluate source angles and distance. Stage 2 used beamforming technology with microphone array to enhance and separate the target speech sound. The asynchronous update function was utilized to integrate the beamforming control and speech processing to reduce the effect of the processing delay. The experimental results show that the noise reduction in various stationary and non-stationary noise environments were 6.1 dB and 5.2 dB respectively. The response time of speech processing was less than 10ms, which meets the requirements of a real-time system. The proposed method has high potential to be applied in auxiliary listening systems or machine language processing like intelligent personal assistant.https://www.mdpi.com/1424-8220/20/12/3527non-stationaryreal-time speech separation methodmicrophone arraybeamformingasynchronous update |
| spellingShingle | Ching-Feng Liu Wei-Siang Ciou Peng-Ting Chen Yi-Chun Du A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach Sensors non-stationary real-time speech separation method microphone array beamforming asynchronous update |
| title | A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach |
| title_full | A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach |
| title_fullStr | A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach |
| title_full_unstemmed | A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach |
| title_short | A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach |
| title_sort | real time speech separation method based on camera and microphone array sensors fusion approach |
| topic | non-stationary real-time speech separation method microphone array beamforming asynchronous update |
| url | https://www.mdpi.com/1424-8220/20/12/3527 |
| work_keys_str_mv | AT chingfengliu arealtimespeechseparationmethodbasedoncameraandmicrophonearraysensorsfusionapproach AT weisiangciou arealtimespeechseparationmethodbasedoncameraandmicrophonearraysensorsfusionapproach AT pengtingchen arealtimespeechseparationmethodbasedoncameraandmicrophonearraysensorsfusionapproach AT yichundu arealtimespeechseparationmethodbasedoncameraandmicrophonearraysensorsfusionapproach AT chingfengliu realtimespeechseparationmethodbasedoncameraandmicrophonearraysensorsfusionapproach AT weisiangciou realtimespeechseparationmethodbasedoncameraandmicrophonearraysensorsfusionapproach AT pengtingchen realtimespeechseparationmethodbasedoncameraandmicrophonearraysensorsfusionapproach AT yichundu realtimespeechseparationmethodbasedoncameraandmicrophonearraysensorsfusionapproach |