E2VIDX: improved bridge between conventional vision and bionic vision
Common RGBD, CMOS, and CCD-based cameras produce motion blur and incorrect exposure under high-speed and improper lighting conditions. According to the bionic principle, the event camera developed has the advantages of low delay, high dynamic range, and no motion blur. However, due to its unique dat...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-10-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnbot.2023.1277160/full |
| _version_ | 1797648828960079872 |
|---|---|
| author | Xujia Hou Feihu Zhang Dhiraj Gulati Tingfeng Tan Wei Zhang |
| author_facet | Xujia Hou Feihu Zhang Dhiraj Gulati Tingfeng Tan Wei Zhang |
| author_sort | Xujia Hou |
| collection | DOAJ |
| description | Common RGBD, CMOS, and CCD-based cameras produce motion blur and incorrect exposure under high-speed and improper lighting conditions. According to the bionic principle, the event camera developed has the advantages of low delay, high dynamic range, and no motion blur. However, due to its unique data representation, it encounters significant obstacles in practical applications. The image reconstruction algorithm based on an event camera solves the problem by converting a series of “events” into common frames to apply existing vision algorithms. Due to the rapid development of neural networks, this field has made significant breakthroughs in past few years. Based on the most popular Events-to-Video (E2VID) method, this study designs a new network called E2VIDX. The proposed network includes group convolution and sub-pixel convolution, which not only achieves better feature fusion but also the network model size is reduced by 25%. Futhermore, we propose a new loss function. The loss function is divided into two parts, first part calculates the high level features and the second part calculates the low level features of the reconstructed image. The experimental results clearly outperform against the state-of-the-art method. Compared with the original method, Structural Similarity (SSIM) increases by 1.3%, Learned Perceptual Image Patch Similarity (LPIPS) decreases by 1.7%, Mean Squared Error (MSE) decreases by 2.5%, and it runs faster on GPU and CPU. Additionally, we evaluate the results of E2VIDX with application to image classification, object detection, and instance segmentation. The experiments show that conversions using our method can help event cameras directly apply existing vision algorithms in most scenarios. |
| first_indexed | 2024-03-11T15:37:36Z |
| format | Article |
| id | doaj.art-ce1f443503674f20900928a11974f4a2 |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-03-11T15:37:36Z |
| publishDate | 2023-10-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-ce1f443503674f20900928a11974f4a22023-10-26T14:45:30ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182023-10-011710.3389/fnbot.2023.12771601277160E2VIDX: improved bridge between conventional vision and bionic visionXujia Hou0Feihu Zhang1Dhiraj Gulati2Tingfeng Tan3Wei Zhang4School of Marine Science and Technology, Northwestern Polytechnical University, Xi'An, ChinaSchool of Marine Science and Technology, Northwestern Polytechnical University, Xi'An, ChinaSiemens EDA, Munich, GermanySchool of Marine Science and Technology, Northwestern Polytechnical University, Xi'An, ChinaSchool of Marine Science and Technology, Northwestern Polytechnical University, Xi'An, ChinaCommon RGBD, CMOS, and CCD-based cameras produce motion blur and incorrect exposure under high-speed and improper lighting conditions. According to the bionic principle, the event camera developed has the advantages of low delay, high dynamic range, and no motion blur. However, due to its unique data representation, it encounters significant obstacles in practical applications. The image reconstruction algorithm based on an event camera solves the problem by converting a series of “events” into common frames to apply existing vision algorithms. Due to the rapid development of neural networks, this field has made significant breakthroughs in past few years. Based on the most popular Events-to-Video (E2VID) method, this study designs a new network called E2VIDX. The proposed network includes group convolution and sub-pixel convolution, which not only achieves better feature fusion but also the network model size is reduced by 25%. Futhermore, we propose a new loss function. The loss function is divided into two parts, first part calculates the high level features and the second part calculates the low level features of the reconstructed image. The experimental results clearly outperform against the state-of-the-art method. Compared with the original method, Structural Similarity (SSIM) increases by 1.3%, Learned Perceptual Image Patch Similarity (LPIPS) decreases by 1.7%, Mean Squared Error (MSE) decreases by 2.5%, and it runs faster on GPU and CPU. Additionally, we evaluate the results of E2VIDX with application to image classification, object detection, and instance segmentation. The experiments show that conversions using our method can help event cameras directly apply existing vision algorithms in most scenarios.https://www.frontiersin.org/articles/10.3389/fnbot.2023.1277160/fullimage reconstructiondeep learningdynamic vision sensorevent cameraimage classificationobject detection |
| spellingShingle | Xujia Hou Feihu Zhang Dhiraj Gulati Tingfeng Tan Wei Zhang E2VIDX: improved bridge between conventional vision and bionic vision Frontiers in Neurorobotics image reconstruction deep learning dynamic vision sensor event camera image classification object detection |
| title | E2VIDX: improved bridge between conventional vision and bionic vision |
| title_full | E2VIDX: improved bridge between conventional vision and bionic vision |
| title_fullStr | E2VIDX: improved bridge between conventional vision and bionic vision |
| title_full_unstemmed | E2VIDX: improved bridge between conventional vision and bionic vision |
| title_short | E2VIDX: improved bridge between conventional vision and bionic vision |
| title_sort | e2vidx improved bridge between conventional vision and bionic vision |
| topic | image reconstruction deep learning dynamic vision sensor event camera image classification object detection |
| url | https://www.frontiersin.org/articles/10.3389/fnbot.2023.1277160/full |
| work_keys_str_mv | AT xujiahou e2vidximprovedbridgebetweenconventionalvisionandbionicvision AT feihuzhang e2vidximprovedbridgebetweenconventionalvisionandbionicvision AT dhirajgulati e2vidximprovedbridgebetweenconventionalvisionandbionicvision AT tingfengtan e2vidximprovedbridgebetweenconventionalvisionandbionicvision AT weizhang e2vidximprovedbridgebetweenconventionalvisionandbionicvision |