FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality
© 2019 IEEE. FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it pro...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021
|
| Online Access: | https://hdl.handle.net/1721.1/136978 |
| _version_ | 1826212931664084992 |
|---|---|
| author | Guerra, Winter Tal, Ezra Murali, Varun Ryou, Gilhyun Karaman, Sertac |
| author2 | Massachusetts Institute of Technology. Laboratory for Information and Decision Systems |
| author_facet | Massachusetts Institute of Technology. Laboratory for Information and Decision Systems Guerra, Winter Tal, Ezra Murali, Varun Ryou, Gilhyun Karaman, Sertac |
| author_sort | Guerra, Winter |
| collection | MIT |
| description | © 2019 IEEE. FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it provides the ability to combine (i) synthetic exteroceptive measurements generated in silico in real time and (ii) vehicle dynamics and proprioceptive measurements generated in motio by vehicle(s) in flight in a motion-capture facility. FlightGoggles is capable of simulating a virtual-reality environment around autonomous vehicle(s) in flight. While a vehicle is in flight in the Flight-Goggles virtual reality environment, exteroceptive sensors are rendered synthetically in real time while all complex dynamics are generated organically through natural interactions of the vehicle. The FlightGoggles framework allows for researchers to accelerate development by circumventing the need to estimate complex and hard-to-model interactions such as aerodynamics, motor mechanics, battery electrochemistry, and behavior of other agents. The ability to perform vehicle-in-the-loop experiments with photorealistic exteroceptive sensor simulation facilitates novel research directions involving, e.g., fast and agile autonomous flight in obstacle-rich environments, safe human interaction, and flexible sensor selection. FlightGoggles has been utilized as the main test for selecting nine teams that will advance in the AlphaPilot autonomous drone racing challenge. We survey approaches and results from the top AlphaPilot teams, which may be of independent interest. FlightGoggles is distributed as open-source software along with the photorealistic graphics assets for several simulation environments, under the MIT license at http://flightgoggles.mit.edu. |
| first_indexed | 2024-09-23T15:40:45Z |
| format | Article |
| id | mit-1721.1/136978 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T15:40:45Z |
| publishDate | 2021 |
| publisher | IEEE |
| record_format | dspace |
| spelling | mit-1721.1/1369782023-02-10T19:23:58Z FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality Guerra, Winter Tal, Ezra Murali, Varun Ryou, Gilhyun Karaman, Sertac Massachusetts Institute of Technology. Laboratory for Information and Decision Systems © 2019 IEEE. FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it provides the ability to combine (i) synthetic exteroceptive measurements generated in silico in real time and (ii) vehicle dynamics and proprioceptive measurements generated in motio by vehicle(s) in flight in a motion-capture facility. FlightGoggles is capable of simulating a virtual-reality environment around autonomous vehicle(s) in flight. While a vehicle is in flight in the Flight-Goggles virtual reality environment, exteroceptive sensors are rendered synthetically in real time while all complex dynamics are generated organically through natural interactions of the vehicle. The FlightGoggles framework allows for researchers to accelerate development by circumventing the need to estimate complex and hard-to-model interactions such as aerodynamics, motor mechanics, battery electrochemistry, and behavior of other agents. The ability to perform vehicle-in-the-loop experiments with photorealistic exteroceptive sensor simulation facilitates novel research directions involving, e.g., fast and agile autonomous flight in obstacle-rich environments, safe human interaction, and flexible sensor selection. FlightGoggles has been utilized as the main test for selecting nine teams that will advance in the AlphaPilot autonomous drone racing challenge. We survey approaches and results from the top AlphaPilot teams, which may be of independent interest. FlightGoggles is distributed as open-source software along with the photorealistic graphics assets for several simulation environments, under the MIT license at http://flightgoggles.mit.edu. 2021-11-01T16:47:32Z 2021-11-01T16:47:32Z 2019-11 2021-04-30T17:24:14Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/136978 Guerra, Winter, Tal, Ezra, Murali, Varun, Ryou, Gilhyun and Karaman, Sertac. 2019. "FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality." IEEE International Conference on Intelligent Robots and Systems. en 10.1109/iros40897.2019.8968116 IEEE International Conference on Intelligent Robots and Systems Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf IEEE arXiv |
| spellingShingle | Guerra, Winter Tal, Ezra Murali, Varun Ryou, Gilhyun Karaman, Sertac FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality |
| title | FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality |
| title_full | FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality |
| title_fullStr | FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality |
| title_full_unstemmed | FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality |
| title_short | FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality |
| title_sort | flightgoggles photorealistic sensor simulation for perception driven robotics using photogrammetry and virtual reality |
| url | https://hdl.handle.net/1721.1/136978 |
| work_keys_str_mv | AT guerrawinter flightgogglesphotorealisticsensorsimulationforperceptiondrivenroboticsusingphotogrammetryandvirtualreality AT talezra flightgogglesphotorealisticsensorsimulationforperceptiondrivenroboticsusingphotogrammetryandvirtualreality AT muralivarun flightgogglesphotorealisticsensorsimulationforperceptiondrivenroboticsusingphotogrammetryandvirtualreality AT ryougilhyun flightgogglesphotorealisticsensorsimulationforperceptiondrivenroboticsusingphotogrammetryandvirtualreality AT karamansertac flightgogglesphotorealisticsensorsimulationforperceptiondrivenroboticsusingphotogrammetryandvirtualreality |