LLM Multimodal Traffic Accident Forecasting
With the rise in traffic congestion in urban centers, predicting accidents has become paramount for city planning and public safety. This work comprehensively studied the efficacy of modern deep learning (DL) methods in forecasting traffic accidents and enhancing Level-4 and Level-5 (L-4 and L-5) dr...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-11-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/22/9225 |
| _version_ | 1797457771202871296 |
|---|---|
| author | I. de Zarzà J. de Curtò Gemma Roig Carlos T. Calafate |
| author_facet | I. de Zarzà J. de Curtò Gemma Roig Carlos T. Calafate |
| author_sort | I. de Zarzà |
| collection | DOAJ |
| description | With the rise in traffic congestion in urban centers, predicting accidents has become paramount for city planning and public safety. This work comprehensively studied the efficacy of modern deep learning (DL) methods in forecasting traffic accidents and enhancing Level-4 and Level-5 (L-4 and L-5) driving assistants with actionable visual and language cues. Using a rich dataset detailing accident occurrences, we juxtaposed the Transformer model against traditional time series models like ARIMA and the more recent Prophet model. Additionally, through detailed analysis, we delved deep into feature importance using principal component analysis (PCA) loadings, uncovering key factors contributing to accidents. We introduce the idea of using real-time interventions with large language models (LLMs) in autonomous driving with the use of lightweight compact LLMs like LLaMA-2 and Zephyr-7b-<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>α</mi></semantics></math></inline-formula>. Our exploration extends to the realm of multimodality, through the use of Large Language-and-Vision Assistant (LLaVA)—a bridge between visual and linguistic cues by means of a Visual Language Model (VLM)—in conjunction with deep probabilistic reasoning, enhancing the real-time responsiveness of autonomous driving systems. In this study, we elucidate the advantages of employing large multimodal models within DL and deep probabilistic programming for enhancing the performance and usability of time series forecasting and feature weight importance, particularly in a self-driving scenario. This work paves the way for safer, smarter cities, underpinned by data-driven decision making. |
| first_indexed | 2024-03-09T16:27:45Z |
| format | Article |
| id | doaj.art-b7bc8b2aae044142a8f81ad55a64b58c |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T16:27:45Z |
| publishDate | 2023-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-b7bc8b2aae044142a8f81ad55a64b58c2023-11-24T15:05:49ZengMDPI AGSensors1424-82202023-11-012322922510.3390/s23229225LLM Multimodal Traffic Accident ForecastingI. de Zarzà0J. de Curtò1Gemma Roig2Carlos T. Calafate3Informatik und Mathematik, GOETHE-University Frankfurt am Main, 60323 Frankfurt am Main, GermanyInformatik und Mathematik, GOETHE-University Frankfurt am Main, 60323 Frankfurt am Main, GermanyInformatik und Mathematik, GOETHE-University Frankfurt am Main, 60323 Frankfurt am Main, GermanyDepartamento de Informática de Sistemas y Computadores, Universitat Politècnica de València, 46022 València, SpainWith the rise in traffic congestion in urban centers, predicting accidents has become paramount for city planning and public safety. This work comprehensively studied the efficacy of modern deep learning (DL) methods in forecasting traffic accidents and enhancing Level-4 and Level-5 (L-4 and L-5) driving assistants with actionable visual and language cues. Using a rich dataset detailing accident occurrences, we juxtaposed the Transformer model against traditional time series models like ARIMA and the more recent Prophet model. Additionally, through detailed analysis, we delved deep into feature importance using principal component analysis (PCA) loadings, uncovering key factors contributing to accidents. We introduce the idea of using real-time interventions with large language models (LLMs) in autonomous driving with the use of lightweight compact LLMs like LLaMA-2 and Zephyr-7b-<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>α</mi></semantics></math></inline-formula>. Our exploration extends to the realm of multimodality, through the use of Large Language-and-Vision Assistant (LLaVA)—a bridge between visual and linguistic cues by means of a Visual Language Model (VLM)—in conjunction with deep probabilistic reasoning, enhancing the real-time responsiveness of autonomous driving systems. In this study, we elucidate the advantages of employing large multimodal models within DL and deep probabilistic programming for enhancing the performance and usability of time series forecasting and feature weight importance, particularly in a self-driving scenario. This work paves the way for safer, smarter cities, underpinned by data-driven decision making.https://www.mdpi.com/1424-8220/23/22/9225LLMVLMLLaVAaccident forecastingtransformerstime series analysis |
| spellingShingle | I. de Zarzà J. de Curtò Gemma Roig Carlos T. Calafate LLM Multimodal Traffic Accident Forecasting Sensors LLM VLM LLaVA accident forecasting transformers time series analysis |
| title | LLM Multimodal Traffic Accident Forecasting |
| title_full | LLM Multimodal Traffic Accident Forecasting |
| title_fullStr | LLM Multimodal Traffic Accident Forecasting |
| title_full_unstemmed | LLM Multimodal Traffic Accident Forecasting |
| title_short | LLM Multimodal Traffic Accident Forecasting |
| title_sort | llm multimodal traffic accident forecasting |
| topic | LLM VLM LLaVA accident forecasting transformers time series analysis |
| url | https://www.mdpi.com/1424-8220/23/22/9225 |
| work_keys_str_mv | AT idezarza llmmultimodaltrafficaccidentforecasting AT jdecurto llmmultimodaltrafficaccidentforecasting AT gemmaroig llmmultimodaltrafficaccidentforecasting AT carlostcalafate llmmultimodaltrafficaccidentforecasting |