Towards Automated Design of Machine Perception Systems
Animal's visual perception systems have evolved to their environment over billions of years, enabling them to navigate, avoid predators, and hunt prey. In contrast, machine perception systems designed by humans require significant engineering and often use standard cameras that may not be well...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/151981 |
| _version_ | 1811086986563289088 |
|---|---|
| author | Klinghoffer, Tzofi |
| author2 | Raskar, Ramesh |
| author_facet | Raskar, Ramesh Klinghoffer, Tzofi |
| author_sort | Klinghoffer, Tzofi |
| collection | MIT |
| description | Animal's visual perception systems have evolved to their environment over billions of years, enabling them to navigate, avoid predators, and hunt prey. In contrast, machine perception systems designed by humans require significant engineering and often use standard cameras that may not be well suited to their task or environment. Consider building a robot to pick up trash. The choice of robot sensors impacts which type of trash it can detect, e.g. perhaps an infrared sensor is needed to detect plastic bottles. In addition, animals are able to understand their environment from different viewpoints and under variable lighting, while machine perception systems often fail to generalize beyond the distribution of training data. Inspired by the evolution of animal's visual perception systems, this thesis explores two distinct but related problems: (1) automated design of machine perception systems, and (2) robustness of machine perception systems to physical phenomena, such as lighting and camera viewpoint. Machine perception systems -- also referred to as imaging systems in this thesis -- consist of cameras and perception models. Cameras are used to sense the environment and capture observations, while perception models are used to analyze captured observations. Cameras contain (1) illumination sources, (2) optical elements, and (3) sensors, while perception models use (4) algorithms. Directly searching over all combinations of these four building blocks to design a machine perception system is challenging due to the size of the search space. In Part I of this thesis, we introduce DISeR: Designing Imaging Systems with Reinforcement Learning, a method that allows task-specific imaging systems to be created and optimized in simulation. In Part II of this thesis, we study the robustness of machine perception systems to physical phenomena. We introduce two methods to mitigate the susceptibility of deep learning models to failure when exposed to out of distribution lighting and camera viewpoints. The first method uses disentanglement of features to improve robustness, while the second method modifies pixels to improve robustness. We evaluate our work using standard benchmarks and peer-reviewed publication. |
| first_indexed | 2024-09-23T13:37:59Z |
| format | Thesis |
| id | mit-1721.1/151981 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T13:37:59Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1519812023-09-01T03:25:58Z Towards Automated Design of Machine Perception Systems Klinghoffer, Tzofi Raskar, Ramesh Program in Media Arts and Sciences (Massachusetts Institute of Technology) Animal's visual perception systems have evolved to their environment over billions of years, enabling them to navigate, avoid predators, and hunt prey. In contrast, machine perception systems designed by humans require significant engineering and often use standard cameras that may not be well suited to their task or environment. Consider building a robot to pick up trash. The choice of robot sensors impacts which type of trash it can detect, e.g. perhaps an infrared sensor is needed to detect plastic bottles. In addition, animals are able to understand their environment from different viewpoints and under variable lighting, while machine perception systems often fail to generalize beyond the distribution of training data. Inspired by the evolution of animal's visual perception systems, this thesis explores two distinct but related problems: (1) automated design of machine perception systems, and (2) robustness of machine perception systems to physical phenomena, such as lighting and camera viewpoint. Machine perception systems -- also referred to as imaging systems in this thesis -- consist of cameras and perception models. Cameras are used to sense the environment and capture observations, while perception models are used to analyze captured observations. Cameras contain (1) illumination sources, (2) optical elements, and (3) sensors, while perception models use (4) algorithms. Directly searching over all combinations of these four building blocks to design a machine perception system is challenging due to the size of the search space. In Part I of this thesis, we introduce DISeR: Designing Imaging Systems with Reinforcement Learning, a method that allows task-specific imaging systems to be created and optimized in simulation. In Part II of this thesis, we study the robustness of machine perception systems to physical phenomena. We introduce two methods to mitigate the susceptibility of deep learning models to failure when exposed to out of distribution lighting and camera viewpoints. The first method uses disentanglement of features to improve robustness, while the second method modifies pixels to improve robustness. We evaluate our work using standard benchmarks and peer-reviewed publication. S.M. 2023-08-30T15:56:12Z 2023-08-30T15:56:12Z 2023-06 2023-08-16T20:34:16.356Z Thesis https://hdl.handle.net/1721.1/151981 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Klinghoffer, Tzofi Towards Automated Design of Machine Perception Systems |
| title | Towards Automated Design of Machine Perception Systems |
| title_full | Towards Automated Design of Machine Perception Systems |
| title_fullStr | Towards Automated Design of Machine Perception Systems |
| title_full_unstemmed | Towards Automated Design of Machine Perception Systems |
| title_short | Towards Automated Design of Machine Perception Systems |
| title_sort | towards automated design of machine perception systems |
| url | https://hdl.handle.net/1721.1/151981 |
| work_keys_str_mv | AT klinghoffertzofi towardsautomateddesignofmachineperceptionsystems |