Scalable end-to-end autonomous vehicle testing via rare-event simulation
© 2018 Curran Associates Inc.All rights reserved. While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rar...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
2022
|
| Online Access: | https://hdl.handle.net/1721.1/143975 |
| _version_ | 1826213706581671936 |
|---|---|
| author | O'Kelly, M Duchi, J Sinha, A Namkoong, H Tedrake, R |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science O'Kelly, M Duchi, J Sinha, A Namkoong, H Tedrake, R |
| author_sort | O'Kelly, M |
| collection | MIT |
| description | © 2018 Curran Associates Inc.All rights reserved. While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by 2-20 times over naive Monte Carlo sampling methods and 10-300P times (where P is the number of processors) over real-world testing. |
| first_indexed | 2024-09-23T15:53:32Z |
| format | Article |
| id | mit-1721.1/143975 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T15:53:32Z |
| publishDate | 2022 |
| record_format | dspace |
| spelling | mit-1721.1/1439752023-01-27T21:45:51Z Scalable end-to-end autonomous vehicle testing via rare-event simulation O'Kelly, M Duchi, J Sinha, A Namkoong, H Tedrake, R Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory © 2018 Curran Associates Inc.All rights reserved. While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by 2-20 times over naive Monte Carlo sampling methods and 10-300P times (where P is the number of processors) over real-world testing. 2022-07-22T15:46:31Z 2022-07-22T15:46:31Z 2018-01-01 2022-07-22T15:40:53Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/143975 O'Kelly, M, Duchi, J, Sinha, A, Namkoong, H and Tedrake, R. 2018. "Scalable end-to-end autonomous vehicle testing via rare-event simulation." Advances in Neural Information Processing Systems, 2018-December. en https://papers.nips.cc/paper/2018/hash/653c579e3f9ba5c03f2f2f8cf4512b39-Abstract.html Advances in Neural Information Processing Systems Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Neural Information Processing Systems (NIPS) |
| spellingShingle | O'Kelly, M Duchi, J Sinha, A Namkoong, H Tedrake, R Scalable end-to-end autonomous vehicle testing via rare-event simulation |
| title | Scalable end-to-end autonomous vehicle testing via rare-event simulation |
| title_full | Scalable end-to-end autonomous vehicle testing via rare-event simulation |
| title_fullStr | Scalable end-to-end autonomous vehicle testing via rare-event simulation |
| title_full_unstemmed | Scalable end-to-end autonomous vehicle testing via rare-event simulation |
| title_short | Scalable end-to-end autonomous vehicle testing via rare-event simulation |
| title_sort | scalable end to end autonomous vehicle testing via rare event simulation |
| url | https://hdl.handle.net/1721.1/143975 |
| work_keys_str_mv | AT okellym scalableendtoendautonomousvehicletestingviarareeventsimulation AT duchij scalableendtoendautonomousvehicletestingviarareeventsimulation AT sinhaa scalableendtoendautonomousvehicletestingviarareeventsimulation AT namkoongh scalableendtoendautonomousvehicletestingviarareeventsimulation AT tedraker scalableendtoendautonomousvehicletestingviarareeventsimulation |