Trajectory Prediction with Linguistic Representations
Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajectories, and is trained using trajectory sam-...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Published: |
Center for Brains, Minds and Machines (CBMM), International Conference on Robotics and Automation (ICRA)
2022
|
| Online Access: | https://hdl.handle.net/1721.1/141362 |
| _version_ | 1826207070654824448 |
|---|---|
| author | Kuo, Yen-Ling Huang, Xin Barbu, Andrei McGill, Stephen G. Katz, Boris Leonard, John J. Rosman, Guy |
| author_facet | Kuo, Yen-Ling Huang, Xin Barbu, Andrei McGill, Stephen G. Katz, Boris Leonard, John J. Rosman, Guy |
| author_sort | Kuo, Yen-Ling |
| collection | MIT |
| description | Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajectories, and is trained using trajectory sam- ples with partially-annotated captions. The model learns the meaning of each of the words without direct per-word supervision. At inference time, it generates a linguistic description of trajectories which captures maneuvers and interactions over an extended time interval. This generated description is used to refine predictions of the trajectories of multiple agents. We train and validate our model on the Argoverse dataset, and demonstrate improved accuracy results in trajectory prediction. In addition, our model is more interpretable: it presents part of its reasoning in plain language as captions, which can aid model development and can aid in building confidence in the model before deploying it. |
| first_indexed | 2024-09-23T13:43:02Z |
| format | Article |
| id | mit-1721.1/141362 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T13:43:02Z |
| publishDate | 2022 |
| publisher | Center for Brains, Minds and Machines (CBMM), International Conference on Robotics and Automation (ICRA) |
| record_format | dspace |
| spelling | mit-1721.1/1413622022-03-25T03:17:15Z Trajectory Prediction with Linguistic Representations Kuo, Yen-Ling Huang, Xin Barbu, Andrei McGill, Stephen G. Katz, Boris Leonard, John J. Rosman, Guy Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajectories, and is trained using trajectory sam- ples with partially-annotated captions. The model learns the meaning of each of the words without direct per-word supervision. At inference time, it generates a linguistic description of trajectories which captures maneuvers and interactions over an extended time interval. This generated description is used to refine predictions of the trajectories of multiple agents. We train and validate our model on the Argoverse dataset, and demonstrate improved accuracy results in trajectory prediction. In addition, our model is more interpretable: it presents part of its reasoning in plain language as captions, which can aid model development and can aid in building confidence in the model before deploying it. This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF – 1231216. 2022-03-24T17:24:15Z 2022-03-24T17:24:15Z 2022-03-09 Article Software Working Paper https://hdl.handle.net/1721.1/141362 CBMM Memo;132 application/pdf Center for Brains, Minds and Machines (CBMM), International Conference on Robotics and Automation (ICRA) |
| spellingShingle | Kuo, Yen-Ling Huang, Xin Barbu, Andrei McGill, Stephen G. Katz, Boris Leonard, John J. Rosman, Guy Trajectory Prediction with Linguistic Representations |
| title | Trajectory Prediction with Linguistic Representations |
| title_full | Trajectory Prediction with Linguistic Representations |
| title_fullStr | Trajectory Prediction with Linguistic Representations |
| title_full_unstemmed | Trajectory Prediction with Linguistic Representations |
| title_short | Trajectory Prediction with Linguistic Representations |
| title_sort | trajectory prediction with linguistic representations |
| url | https://hdl.handle.net/1721.1/141362 |
| work_keys_str_mv | AT kuoyenling trajectorypredictionwithlinguisticrepresentations AT huangxin trajectorypredictionwithlinguisticrepresentations AT barbuandrei trajectorypredictionwithlinguisticrepresentations AT mcgillstepheng trajectorypredictionwithlinguisticrepresentations AT katzboris trajectorypredictionwithlinguisticrepresentations AT leonardjohnj trajectorypredictionwithlinguisticrepresentations AT rosmanguy trajectorypredictionwithlinguisticrepresentations |