Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching
Vision-based underwater autonomous systems play a significant role in marine exploration. Stereo matching is one of the most popular applications for vision-based underwater autonomous systems, which recovers the geometric information of underwater scenes via stereo disparity estimation. While stere...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-04-01
|
| Series: | Journal of Marine Science and Engineering |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2077-1312/11/5/930 |
| _version_ | 1797599572346798080 |
|---|---|
| author | Jiaqi Leng Qingxuan Lv Shu Zhang Yuan Rao Yimei Liu Hao Fan |
| author_facet | Jiaqi Leng Qingxuan Lv Shu Zhang Yuan Rao Yimei Liu Hao Fan |
| author_sort | Jiaqi Leng |
| collection | DOAJ |
| description | Vision-based underwater autonomous systems play a significant role in marine exploration. Stereo matching is one of the most popular applications for vision-based underwater autonomous systems, which recovers the geometric information of underwater scenes via stereo disparity estimation. While stereo matching in the air has achieved great progress with the development of neural networks, it generalizes poorly to the underwater scenario due to the challenging underwater degradation. In this paper, we propose a novel Multilevel Inverse Patchmatch Network (MIPNet) to iteratively model pair-wise correlations under underwater degradation and estimate stereo disparity with both local and global refinements. Specifically, we first utilized the inverse Patchmatch module in a novel multilevel pyramid structure to recover the detailed stereo disparity from the input stereo images. Secondly, we introduced a powerful Attentional Feature Fusion module to model pair-wise correlations with global context, ensuring high-quality stereo disparity estimation for both in-air and underwater scenarios. We evaluate the proposed method on the popular real-world ETH3D benchmark, and the highly competitive performance against the popular baselines demonstrates the effectiveness of the proposed method. Moreover, with its superior performance on our real-world underwater dataset, e.g., our method outperforms the popular baseline RAFT-Stereo by 27.1%, we show the good generalization ability of our method to underwater scenarios. We finally discuss the potential challenges for underwater stereo matching via our experiments on the impact of water. |
| first_indexed | 2024-03-11T03:36:13Z |
| format | Article |
| id | doaj.art-21f4a43f1b964b9fa5760c046137e0a8 |
| institution | Directory Open Access Journal |
| issn | 2077-1312 |
| language | English |
| last_indexed | 2024-03-11T03:36:13Z |
| publishDate | 2023-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Journal of Marine Science and Engineering |
| spelling | doaj.art-21f4a43f1b964b9fa5760c046137e0a82023-11-18T01:58:39ZengMDPI AGJournal of Marine Science and Engineering2077-13122023-04-0111593010.3390/jmse11050930Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo MatchingJiaqi Leng0Qingxuan Lv1Shu Zhang2Yuan Rao3Yimei Liu4Hao Fan5Haide College, Ocean University of China, Qingdao 266100, ChinaCollege of Computer Science and Technology, Ocean University of China, Qingdao 266100, ChinaCollege of Computer Science and Technology, Ocean University of China, Qingdao 266100, ChinaCollege of Computer Science and Technology, Ocean University of China, Qingdao 266100, ChinaCollege of Computer Science and Technology, Ocean University of China, Qingdao 266100, ChinaCollege of Computer Science and Technology, Ocean University of China, Qingdao 266100, ChinaVision-based underwater autonomous systems play a significant role in marine exploration. Stereo matching is one of the most popular applications for vision-based underwater autonomous systems, which recovers the geometric information of underwater scenes via stereo disparity estimation. While stereo matching in the air has achieved great progress with the development of neural networks, it generalizes poorly to the underwater scenario due to the challenging underwater degradation. In this paper, we propose a novel Multilevel Inverse Patchmatch Network (MIPNet) to iteratively model pair-wise correlations under underwater degradation and estimate stereo disparity with both local and global refinements. Specifically, we first utilized the inverse Patchmatch module in a novel multilevel pyramid structure to recover the detailed stereo disparity from the input stereo images. Secondly, we introduced a powerful Attentional Feature Fusion module to model pair-wise correlations with global context, ensuring high-quality stereo disparity estimation for both in-air and underwater scenarios. We evaluate the proposed method on the popular real-world ETH3D benchmark, and the highly competitive performance against the popular baselines demonstrates the effectiveness of the proposed method. Moreover, with its superior performance on our real-world underwater dataset, e.g., our method outperforms the popular baseline RAFT-Stereo by 27.1%, we show the good generalization ability of our method to underwater scenarios. We finally discuss the potential challenges for underwater stereo matching via our experiments on the impact of water.https://www.mdpi.com/2077-1312/11/5/930underwater stereo matchingDeep Inverse Patchmatch Networkmultilevel recurrent neural networkAttentional Feature Fusionreal-world underwater scenarios |
| spellingShingle | Jiaqi Leng Qingxuan Lv Shu Zhang Yuan Rao Yimei Liu Hao Fan Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching Journal of Marine Science and Engineering underwater stereo matching Deep Inverse Patchmatch Network multilevel recurrent neural network Attentional Feature Fusion real-world underwater scenarios |
| title | Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching |
| title_full | Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching |
| title_fullStr | Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching |
| title_full_unstemmed | Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching |
| title_short | Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching |
| title_sort | multilevel inverse patchmatch network with local and global refinement for underwater stereo matching |
| topic | underwater stereo matching Deep Inverse Patchmatch Network multilevel recurrent neural network Attentional Feature Fusion real-world underwater scenarios |
| url | https://www.mdpi.com/2077-1312/11/5/930 |
| work_keys_str_mv | AT jiaqileng multilevelinversepatchmatchnetworkwithlocalandglobalrefinementforunderwaterstereomatching AT qingxuanlv multilevelinversepatchmatchnetworkwithlocalandglobalrefinementforunderwaterstereomatching AT shuzhang multilevelinversepatchmatchnetworkwithlocalandglobalrefinementforunderwaterstereomatching AT yuanrao multilevelinversepatchmatchnetworkwithlocalandglobalrefinementforunderwaterstereomatching AT yimeiliu multilevelinversepatchmatchnetworkwithlocalandglobalrefinementforunderwaterstereomatching AT haofan multilevelinversepatchmatchnetworkwithlocalandglobalrefinementforunderwaterstereomatching |