Optical flow estimation from event-based cameras and spiking neural networks
Event-based cameras are raising interest within the computer vision community. These sensors operate with asynchronous pixels, emitting events, or “spikes”, when the luminance change at a given pixel since the last event surpasses a certain threshold. Thanks to their inherent qualities, such as thei...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-05-01
|
| Series: | Frontiers in Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnins.2023.1160034/full |
| _version_ | 1797829281020116992 |
|---|---|
| author | Javier Cuadrado Ulysse Rançon Benoit R. Cottereau Benoit R. Cottereau Francisco Barranco Timothée Masquelier |
| author_facet | Javier Cuadrado Ulysse Rançon Benoit R. Cottereau Benoit R. Cottereau Francisco Barranco Timothée Masquelier |
| author_sort | Javier Cuadrado |
| collection | DOAJ |
| description | Event-based cameras are raising interest within the computer vision community. These sensors operate with asynchronous pixels, emitting events, or “spikes”, when the luminance change at a given pixel since the last event surpasses a certain threshold. Thanks to their inherent qualities, such as their low power consumption, low latency, and high dynamic range, they seem particularly tailored to applications with challenging temporal constraints and safety requirements. Event-based sensors are an excellent fit for Spiking Neural Networks (SNNs), since the coupling of an asynchronous sensor with neuromorphic hardware can yield real-time systems with minimal power requirements. In this work, we seek to develop one such system, using both event sensor data from the DSEC dataset and spiking neural networks to estimate optical flow for driving scenarios. We propose a U-Net-like SNN which, after supervised training, is able to make dense optical flow estimations. To do so, we encourage both minimal norm for the error vector and minimal angle between ground-truth and predicted flow, training our model with back-propagation using a surrogate gradient. In addition, the use of 3d convolutions allows us to capture the dynamic nature of the data by increasing the temporal receptive fields. Upsampling after each decoding stage ensures that each decoder's output contributes to the final estimation. Thanks to separable convolutions, we have been able to develop a light model (when compared to competitors) that can nonetheless yield reasonably accurate optical flow estimates. |
| first_indexed | 2024-04-09T13:18:59Z |
| format | Article |
| id | doaj.art-8436b69982584f8e80ba185b1d5fd307 |
| institution | Directory Open Access Journal |
| issn | 1662-453X |
| language | English |
| last_indexed | 2024-04-09T13:18:59Z |
| publishDate | 2023-05-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroscience |
| spelling | doaj.art-8436b69982584f8e80ba185b1d5fd3072023-05-11T11:22:18ZengFrontiers Media S.A.Frontiers in Neuroscience1662-453X2023-05-011710.3389/fnins.2023.11600341160034Optical flow estimation from event-based cameras and spiking neural networksJavier Cuadrado0Ulysse Rançon1Benoit R. Cottereau2Benoit R. Cottereau3Francisco Barranco4Timothée Masquelier5CerCo UMR 5549, CNRS – Université Toulouse III, Toulouse, FranceCerCo UMR 5549, CNRS – Université Toulouse III, Toulouse, FranceCerCo UMR 5549, CNRS – Université Toulouse III, Toulouse, FranceIPAL, CNRS IRL 2955, Singapore, SingaporeDepartment of Computer Engineering, Automatics and Robotics, CITIC, University of Granada, Granada, SpainCerCo UMR 5549, CNRS – Université Toulouse III, Toulouse, FranceEvent-based cameras are raising interest within the computer vision community. These sensors operate with asynchronous pixels, emitting events, or “spikes”, when the luminance change at a given pixel since the last event surpasses a certain threshold. Thanks to their inherent qualities, such as their low power consumption, low latency, and high dynamic range, they seem particularly tailored to applications with challenging temporal constraints and safety requirements. Event-based sensors are an excellent fit for Spiking Neural Networks (SNNs), since the coupling of an asynchronous sensor with neuromorphic hardware can yield real-time systems with minimal power requirements. In this work, we seek to develop one such system, using both event sensor data from the DSEC dataset and spiking neural networks to estimate optical flow for driving scenarios. We propose a U-Net-like SNN which, after supervised training, is able to make dense optical flow estimations. To do so, we encourage both minimal norm for the error vector and minimal angle between ground-truth and predicted flow, training our model with back-propagation using a surrogate gradient. In addition, the use of 3d convolutions allows us to capture the dynamic nature of the data by increasing the temporal receptive fields. Upsampling after each decoding stage ensures that each decoder's output contributes to the final estimation. Thanks to separable convolutions, we have been able to develop a light model (when compared to competitors) that can nonetheless yield reasonably accurate optical flow estimates.https://www.frontiersin.org/articles/10.3389/fnins.2023.1160034/fulloptical flowevent visionspiking neural networksneuromorphic computingedge AI |
| spellingShingle | Javier Cuadrado Ulysse Rançon Benoit R. Cottereau Benoit R. Cottereau Francisco Barranco Timothée Masquelier Optical flow estimation from event-based cameras and spiking neural networks Frontiers in Neuroscience optical flow event vision spiking neural networks neuromorphic computing edge AI |
| title | Optical flow estimation from event-based cameras and spiking neural networks |
| title_full | Optical flow estimation from event-based cameras and spiking neural networks |
| title_fullStr | Optical flow estimation from event-based cameras and spiking neural networks |
| title_full_unstemmed | Optical flow estimation from event-based cameras and spiking neural networks |
| title_short | Optical flow estimation from event-based cameras and spiking neural networks |
| title_sort | optical flow estimation from event based cameras and spiking neural networks |
| topic | optical flow event vision spiking neural networks neuromorphic computing edge AI |
| url | https://www.frontiersin.org/articles/10.3389/fnins.2023.1160034/full |
| work_keys_str_mv | AT javiercuadrado opticalflowestimationfromeventbasedcamerasandspikingneuralnetworks AT ulysserancon opticalflowestimationfromeventbasedcamerasandspikingneuralnetworks AT benoitrcottereau opticalflowestimationfromeventbasedcamerasandspikingneuralnetworks AT benoitrcottereau opticalflowestimationfromeventbasedcamerasandspikingneuralnetworks AT franciscobarranco opticalflowestimationfromeventbasedcamerasandspikingneuralnetworks AT timotheemasquelier opticalflowestimationfromeventbasedcamerasandspikingneuralnetworks |