SDF-MAN: Semi-Supervised Disparity Fusion with Multi-Scale Adversarial Networks
Refining raw disparity maps from different algorithms to exploit their complementary advantages is still challenging. Uncertainty estimation and complex disparity relationships among pixels limit the accuracy and robustness of existing methods and there is no standard method for fusion of different...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2019-02-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/11/5/487 |
| _version_ | 1818974187292721152 |
|---|---|
| author | Can Pu Runzi Song Radim Tylecek Nanbo Li Robert B. Fisher |
| author_facet | Can Pu Runzi Song Radim Tylecek Nanbo Li Robert B. Fisher |
| author_sort | Can Pu |
| collection | DOAJ |
| description | Refining raw disparity maps from different algorithms to exploit their complementary advantages is still challenging. Uncertainty estimation and complex disparity relationships among pixels limit the accuracy and robustness of existing methods and there is no standard method for fusion of different kinds of depth data. In this paper, we introduce a new method to fuse disparity maps from different sources, while incorporating supplementary information (intensity, gradient, etc.) into a refiner network to better refine raw disparity inputs. A discriminator network classifies disparities at different receptive fields and scales. Assuming a Markov Random Field for the refined disparity map produces better estimates of the true disparity distribution. Both fully supervised and semi-supervised versions of the algorithm are proposed. The approach includes a more robust loss function to inpaint invalid disparity values and requires much less labeled data to train in the semi-supervised learning mode. The algorithm can be generalized to fuse depths from different kinds of depth sources. Experiments explored different fusion opportunities: stereo-monocular fusion, stereo-ToF fusion and stereo-stereo fusion. The experiments show the superiority of the proposed algorithm compared with the most recent algorithms on public synthetic datasets (Scene Flow, SYNTH3, our synthetic garden dataset) and real datasets (Kitti2015 dataset and Trimbot2020 Garden dataset). |
| first_indexed | 2024-12-20T15:36:04Z |
| format | Article |
| id | doaj.art-1873836c1b4549a1bcec458c3440f245 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-12-20T15:36:04Z |
| publishDate | 2019-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-1873836c1b4549a1bcec458c3440f2452022-12-21T19:35:23ZengMDPI AGRemote Sensing2072-42922019-02-0111548710.3390/rs11050487rs11050487SDF-MAN: Semi-Supervised Disparity Fusion with Multi-Scale Adversarial NetworksCan Pu0Runzi Song1Radim Tylecek2Nanbo Li3Robert B. Fisher4School of Informatics, University of Edinburgh, Edinburgh EH8 9BT, UKDepartment of Mechanical Engineering, Tsinghua University, Beijing 100084, ChinaSchool of Informatics, University of Edinburgh, Edinburgh EH8 9BT, UKSchool of Informatics, University of Edinburgh, Edinburgh EH8 9BT, UKSchool of Informatics, University of Edinburgh, Edinburgh EH8 9BT, UKRefining raw disparity maps from different algorithms to exploit their complementary advantages is still challenging. Uncertainty estimation and complex disparity relationships among pixels limit the accuracy and robustness of existing methods and there is no standard method for fusion of different kinds of depth data. In this paper, we introduce a new method to fuse disparity maps from different sources, while incorporating supplementary information (intensity, gradient, etc.) into a refiner network to better refine raw disparity inputs. A discriminator network classifies disparities at different receptive fields and scales. Assuming a Markov Random Field for the refined disparity map produces better estimates of the true disparity distribution. Both fully supervised and semi-supervised versions of the algorithm are proposed. The approach includes a more robust loss function to inpaint invalid disparity values and requires much less labeled data to train in the semi-supervised learning mode. The algorithm can be generalized to fuse depths from different kinds of depth sources. Experiments explored different fusion opportunities: stereo-monocular fusion, stereo-ToF fusion and stereo-stereo fusion. The experiments show the superiority of the proposed algorithm compared with the most recent algorithms on public synthetic datasets (Scene Flow, SYNTH3, our synthetic garden dataset) and real datasets (Kitti2015 dataset and Trimbot2020 Garden dataset).https://www.mdpi.com/2072-4292/11/5/487depth fusiondisparity fusionstereo vision, monocular visiontime of flight |
| spellingShingle | Can Pu Runzi Song Radim Tylecek Nanbo Li Robert B. Fisher SDF-MAN: Semi-Supervised Disparity Fusion with Multi-Scale Adversarial Networks Remote Sensing depth fusion disparity fusion stereo vision, monocular vision time of flight |
| title | SDF-MAN: Semi-Supervised Disparity Fusion with Multi-Scale Adversarial Networks |
| title_full | SDF-MAN: Semi-Supervised Disparity Fusion with Multi-Scale Adversarial Networks |
| title_fullStr | SDF-MAN: Semi-Supervised Disparity Fusion with Multi-Scale Adversarial Networks |
| title_full_unstemmed | SDF-MAN: Semi-Supervised Disparity Fusion with Multi-Scale Adversarial Networks |
| title_short | SDF-MAN: Semi-Supervised Disparity Fusion with Multi-Scale Adversarial Networks |
| title_sort | sdf man semi supervised disparity fusion with multi scale adversarial networks |
| topic | depth fusion disparity fusion stereo vision, monocular vision time of flight |
| url | https://www.mdpi.com/2072-4292/11/5/487 |
| work_keys_str_mv | AT canpu sdfmansemisuperviseddisparityfusionwithmultiscaleadversarialnetworks AT runzisong sdfmansemisuperviseddisparityfusionwithmultiscaleadversarialnetworks AT radimtylecek sdfmansemisuperviseddisparityfusionwithmultiscaleadversarialnetworks AT nanboli sdfmansemisuperviseddisparityfusionwithmultiscaleadversarialnetworks AT robertbfisher sdfmansemisuperviseddisparityfusionwithmultiscaleadversarialnetworks |