A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event Videos
As bio-inspired vision devices, dynamic vision sensors (DVS) are being applied in more and more applications. Unlike normal cameras, pixels in DVS independently respond to the luminance change with asynchronous output spikes. Therefore, removing raindrops and streaks from DVS event videos is a new b...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-08-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnbot.2022.928707/full |
| _version_ | 1818172440499453952 |
|---|---|
| author | Long Cheng Ni Liu Xusen Guo Yuhao Shen Zijun Meng Kai Huang Kai Huang Xiaoqin Zhang |
| author_facet | Long Cheng Ni Liu Xusen Guo Yuhao Shen Zijun Meng Kai Huang Kai Huang Xiaoqin Zhang |
| author_sort | Long Cheng |
| collection | DOAJ |
| description | As bio-inspired vision devices, dynamic vision sensors (DVS) are being applied in more and more applications. Unlike normal cameras, pixels in DVS independently respond to the luminance change with asynchronous output spikes. Therefore, removing raindrops and streaks from DVS event videos is a new but challenging task as the conventional deraining methods are no longer applicable. In this article, we propose to perform the deraining process in the width and time (W-T) space. This is motivated by the observation that rain steaks exhibits discontinuity in the width and time directions while background moving objects are usually piecewise smooth along with both directions. The W-T space can fuse the discontinuity in both directions and thus transforms raindrops and streaks to approximately uniform noise that are easy to remove. The non-local means filter is adopted as background object motion has periodic patterns in the W-T space. A repairing method is also designed to restore edge details erased during the deraining process. Experimental results demonstrate that our approach can better remove rain noise than the four existing methods for traditional camera videos. We also study how the event buffer depth and event frame time affect the performance investigate the potential implementation of our approach to classic RGB images. A new real-world database for DVS deraining is also created and shared for public use. |
| first_indexed | 2024-12-11T19:12:39Z |
| format | Article |
| id | doaj.art-a262e446abf44009a7a2fe001b645664 |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-12-11T19:12:39Z |
| publishDate | 2022-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-a262e446abf44009a7a2fe001b6456642022-12-22T00:53:43ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182022-08-011610.3389/fnbot.2022.928707928707A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event VideosLong Cheng0Ni Liu1Xusen Guo2Yuhao Shen3Zijun Meng4Kai Huang5Kai Huang6Xiaoqin Zhang7College of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, ChinaSchool of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, ChinaSchool of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, ChinaCollege of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, ChinaCollege of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, ChinaSchool of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, ChinaKey Laboratory of Machine Intelligence and Advanced Computing, Guangzhou, ChinaCollege of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, ChinaAs bio-inspired vision devices, dynamic vision sensors (DVS) are being applied in more and more applications. Unlike normal cameras, pixels in DVS independently respond to the luminance change with asynchronous output spikes. Therefore, removing raindrops and streaks from DVS event videos is a new but challenging task as the conventional deraining methods are no longer applicable. In this article, we propose to perform the deraining process in the width and time (W-T) space. This is motivated by the observation that rain steaks exhibits discontinuity in the width and time directions while background moving objects are usually piecewise smooth along with both directions. The W-T space can fuse the discontinuity in both directions and thus transforms raindrops and streaks to approximately uniform noise that are easy to remove. The non-local means filter is adopted as background object motion has periodic patterns in the W-T space. A repairing method is also designed to restore edge details erased during the deraining process. Experimental results demonstrate that our approach can better remove rain noise than the four existing methods for traditional camera videos. We also study how the event buffer depth and event frame time affect the performance investigate the potential implementation of our approach to classic RGB images. A new real-world database for DVS deraining is also created and shared for public use.https://www.frontiersin.org/articles/10.3389/fnbot.2022.928707/fulldynamic vision sensorsrain removalintelligent drivingoutdoor vision systemsderaining |
| spellingShingle | Long Cheng Ni Liu Xusen Guo Yuhao Shen Zijun Meng Kai Huang Kai Huang Xiaoqin Zhang A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event Videos Frontiers in Neurorobotics dynamic vision sensors rain removal intelligent driving outdoor vision systems deraining |
| title | A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event Videos |
| title_full | A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event Videos |
| title_fullStr | A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event Videos |
| title_full_unstemmed | A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event Videos |
| title_short | A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event Videos |
| title_sort | novel rain removal approach for outdoor dynamic vision sensor event videos |
| topic | dynamic vision sensors rain removal intelligent driving outdoor vision systems deraining |
| url | https://www.frontiersin.org/articles/10.3389/fnbot.2022.928707/full |
| work_keys_str_mv | AT longcheng anovelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT niliu anovelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT xusenguo anovelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT yuhaoshen anovelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT zijunmeng anovelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT kaihuang anovelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT kaihuang anovelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT xiaoqinzhang anovelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT longcheng novelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT niliu novelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT xusenguo novelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT yuhaoshen novelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT zijunmeng novelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT kaihuang novelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT kaihuang novelrainremovalapproachforoutdoordynamicvisionsensoreventvideos AT xiaoqinzhang novelrainremovalapproachforoutdoordynamicvisionsensoreventvideos |