Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning
Abstract Widespread adaptation of autonomous, robotic systems relies greatly on safe and reliable operation, which in many cases is derived from the ability to maintain accurate and robust perception capabilities. Environmental and operational conditions as well as improper maintenance can produce c...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2024-01-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-024-53009-z |
| _version_ | 1797275021274513408 |
|---|---|
| author | Shmuel Y. Hayoun Meir Halachmi Doron Serebro Kfir Twizer Elinor Medezinski Liron Korkidi Moshik Cohen Itai Orr |
| author_facet | Shmuel Y. Hayoun Meir Halachmi Doron Serebro Kfir Twizer Elinor Medezinski Liron Korkidi Moshik Cohen Itai Orr |
| author_sort | Shmuel Y. Hayoun |
| collection | DOAJ |
| description | Abstract Widespread adaptation of autonomous, robotic systems relies greatly on safe and reliable operation, which in many cases is derived from the ability to maintain accurate and robust perception capabilities. Environmental and operational conditions as well as improper maintenance can produce calibration errors inhibiting sensor fusion and, consequently, degrading the perception performance and overall system usability. Traditionally, sensor calibration is performed in a controlled environment with one or more known targets. Such a procedure can only be carried out in between operations and is done manually; a tedious task if it must be conducted on a regular basis. This creates an acute need for online targetless methods, capable of yielding a set of geometric transformations based on perceived environmental features. However, the often-required redundancy in sensing modalities poses further challenges, as the features captured by each sensor and their distinctiveness may vary. We present a holistic approach to performing joint calibration of a camera–lidar–radar trio in a representative autonomous driving application. Leveraging prior knowledge and physical properties of these sensing modalities together with semantic information, we propose two targetless calibration methods within a cost minimization framework: the first via direct online optimization, and the second through self-supervised learning (SSL). |
| first_indexed | 2024-03-07T15:07:24Z |
| format | Article |
| id | doaj.art-97b38f0ba87143808b3f604e14ae7934 |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-03-07T15:07:24Z |
| publishDate | 2024-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-97b38f0ba87143808b3f604e14ae79342024-03-05T18:50:15ZengNature PortfolioScientific Reports2045-23222024-01-0114111610.1038/s41598-024-53009-zPhysics and semantic informed multi-sensor calibration via optimization theory and self-supervised learningShmuel Y. Hayoun0Meir Halachmi1Doron Serebro2Kfir Twizer3Elinor Medezinski4Liron Korkidi5Moshik Cohen6Itai Orr7Wisense Technologies Ltd.Wisense Technologies Ltd.Wisense Technologies Ltd.Wisense Technologies Ltd.Wisense Technologies Ltd.Wisense Technologies Ltd.Wisense Technologies Ltd.Wisense Technologies Ltd.Abstract Widespread adaptation of autonomous, robotic systems relies greatly on safe and reliable operation, which in many cases is derived from the ability to maintain accurate and robust perception capabilities. Environmental and operational conditions as well as improper maintenance can produce calibration errors inhibiting sensor fusion and, consequently, degrading the perception performance and overall system usability. Traditionally, sensor calibration is performed in a controlled environment with one or more known targets. Such a procedure can only be carried out in between operations and is done manually; a tedious task if it must be conducted on a regular basis. This creates an acute need for online targetless methods, capable of yielding a set of geometric transformations based on perceived environmental features. However, the often-required redundancy in sensing modalities poses further challenges, as the features captured by each sensor and their distinctiveness may vary. We present a holistic approach to performing joint calibration of a camera–lidar–radar trio in a representative autonomous driving application. Leveraging prior knowledge and physical properties of these sensing modalities together with semantic information, we propose two targetless calibration methods within a cost minimization framework: the first via direct online optimization, and the second through self-supervised learning (SSL).https://doi.org/10.1038/s41598-024-53009-z |
| spellingShingle | Shmuel Y. Hayoun Meir Halachmi Doron Serebro Kfir Twizer Elinor Medezinski Liron Korkidi Moshik Cohen Itai Orr Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning Scientific Reports |
| title | Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning |
| title_full | Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning |
| title_fullStr | Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning |
| title_full_unstemmed | Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning |
| title_short | Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning |
| title_sort | physics and semantic informed multi sensor calibration via optimization theory and self supervised learning |
| url | https://doi.org/10.1038/s41598-024-53009-z |
| work_keys_str_mv | AT shmuelyhayoun physicsandsemanticinformedmultisensorcalibrationviaoptimizationtheoryandselfsupervisedlearning AT meirhalachmi physicsandsemanticinformedmultisensorcalibrationviaoptimizationtheoryandselfsupervisedlearning AT doronserebro physicsandsemanticinformedmultisensorcalibrationviaoptimizationtheoryandselfsupervisedlearning AT kfirtwizer physicsandsemanticinformedmultisensorcalibrationviaoptimizationtheoryandselfsupervisedlearning AT elinormedezinski physicsandsemanticinformedmultisensorcalibrationviaoptimizationtheoryandselfsupervisedlearning AT lironkorkidi physicsandsemanticinformedmultisensorcalibrationviaoptimizationtheoryandselfsupervisedlearning AT moshikcohen physicsandsemanticinformedmultisensorcalibrationviaoptimizationtheoryandselfsupervisedlearning AT itaiorr physicsandsemanticinformedmultisensorcalibrationviaoptimizationtheoryandselfsupervisedlearning |