StereoSpike: Depth Learning With a Spiking Neural Network
Depth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here, we propose to solve it using StereoSpike, an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Netwo...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9969606/ |
| _version_ | 1811201144715739136 |
|---|---|
| author | Ulysse Rancon Javier Cuadrado-Anibarro Benoit R. Cottereau Timothee Masquelier |
| author_facet | Ulysse Rancon Javier Cuadrado-Anibarro Benoit R. Cottereau Timothee Masquelier |
| author_sort | Ulysse Rancon |
| collection | DOAJ |
| description | Depth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here, we propose to solve it using StereoSpike, an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Network (SNN) with a modified U-Net-like encoder-decoder architecture. More specifically, we used the Multi Vehicle Stereo Event Camera Dataset (MVSEC). It provides a depth ground-truth, which was used to train StereoSpike in a supervised manner, using surrogate gradient descent. We propose a novel readout paradigm to obtain a dense analog prediction–the depth of each pixel– from the spikes of the decoder. We demonstrate that this architecture generalizes very well, even better than its non-spiking counterparts, leading to near state-of-the-art test accuracy. To the best of our knowledge, it is the first time that such a large-scale regression problem is solved by a fully spiking neural network. Finally, we show that very low firing rates (< 5%) can be obtained via regularization, with a minimal cost in accuracy. This means that StereoSpike could be efficiently implemented on neuromorphic chips, opening the door for low power and real time embedded systems. |
| first_indexed | 2024-04-12T02:16:53Z |
| format | Article |
| id | doaj.art-970bc469366749b1928af3cb1253531a |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-12T02:16:53Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-970bc469366749b1928af3cb1253531a2022-12-22T03:52:14ZengIEEEIEEE Access2169-35362022-01-011012742812743910.1109/ACCESS.2022.32264849969606StereoSpike: Depth Learning With a Spiking Neural NetworkUlysse Rancon0https://orcid.org/0000-0002-3149-4870Javier Cuadrado-Anibarro1Benoit R. Cottereau2https://orcid.org/0000-0002-2624-7680Timothee Masquelier3https://orcid.org/0000-0001-8629-9506CerCo, CNRS UM 5549, Université Toulouse III, Toulouse, FranceCerCo, CNRS UM 5549, Université Toulouse III, Toulouse, FranceCerCo, CNRS UM 5549, Université Toulouse III, Toulouse, FranceCerCo, CNRS UM 5549, Université Toulouse III, Toulouse, FranceDepth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here, we propose to solve it using StereoSpike, an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Network (SNN) with a modified U-Net-like encoder-decoder architecture. More specifically, we used the Multi Vehicle Stereo Event Camera Dataset (MVSEC). It provides a depth ground-truth, which was used to train StereoSpike in a supervised manner, using surrogate gradient descent. We propose a novel readout paradigm to obtain a dense analog prediction–the depth of each pixel– from the spikes of the decoder. We demonstrate that this architecture generalizes very well, even better than its non-spiking counterparts, leading to near state-of-the-art test accuracy. To the best of our knowledge, it is the first time that such a large-scale regression problem is solved by a fully spiking neural network. Finally, we show that very low firing rates (< 5%) can be obtained via regularization, with a minimal cost in accuracy. This means that StereoSpike could be efficiently implemented on neuromorphic chips, opening the door for low power and real time embedded systems.https://ieeexplore.ieee.org/document/9969606/Computer visionbio-inspired learningdeep neural architecturesneuromorphic computingspiking neural networksstereo depth regression |
| spellingShingle | Ulysse Rancon Javier Cuadrado-Anibarro Benoit R. Cottereau Timothee Masquelier StereoSpike: Depth Learning With a Spiking Neural Network IEEE Access Computer vision bio-inspired learning deep neural architectures neuromorphic computing spiking neural networks stereo depth regression |
| title | StereoSpike: Depth Learning With a Spiking Neural Network |
| title_full | StereoSpike: Depth Learning With a Spiking Neural Network |
| title_fullStr | StereoSpike: Depth Learning With a Spiking Neural Network |
| title_full_unstemmed | StereoSpike: Depth Learning With a Spiking Neural Network |
| title_short | StereoSpike: Depth Learning With a Spiking Neural Network |
| title_sort | stereospike depth learning with a spiking neural network |
| topic | Computer vision bio-inspired learning deep neural architectures neuromorphic computing spiking neural networks stereo depth regression |
| url | https://ieeexplore.ieee.org/document/9969606/ |
| work_keys_str_mv | AT ulysserancon stereospikedepthlearningwithaspikingneuralnetwork AT javiercuadradoanibarro stereospikedepthlearningwithaspikingneuralnetwork AT benoitrcottereau stereospikedepthlearningwithaspikingneuralnetwork AT timotheemasquelier stereospikedepthlearningwithaspikingneuralnetwork |