Computational Discovery of Hidden Cues in Photographs
Images of everyday scenes often contain hidden information that can be extracted to localize objects outside the view of the camera and to see around corners. For example, we show that it is possible to look at shadows cast by an object on a table, such as a teapot, and reconstruct an image of the s...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/151975 |
| _version_ | 1826198013854351360 |
|---|---|
| author | Swedish, Tristan |
| author2 | Raskar, Ramesh |
| author_facet | Raskar, Ramesh Swedish, Tristan |
| author_sort | Swedish, Tristan |
| collection | MIT |
| description | Images of everyday scenes often contain hidden information that can be extracted to localize objects outside the view of the camera and to see around corners. For example, we show that it is possible to look at shadows cast by an object on a table, such as a teapot, and reconstruct an image of the surrounding room. We describe how to identify and make use of these hidden cues such as shadows, reflections, and other subtle changes in an image caused by the interaction of light with objects in a scene that are not in the direct-line-of-sight. We use the term computational discovery to describe techniques that can be used to uncover these cues and reveal hidden information.
Despite incredible advances in computer vision in recent years, cameras are limited to a single viewpoint of a scene, requiring invasive multi-camera setups or active imaging modalities to solve many perception tasks today. Prior work has identified hidden cues that are present in photographs of certain environments, but these methods often require human insight to identify cues, and extensive calibration to make use of them. In order to address the limitations found in prior work, we propose an end-to-end machine learning framework to identify hidden cues. More generally, we show that object localization is approximately equal to localizing a point light source, and describe how this insight can be used to identify situations when object localization is possible. Furthermore, we show that physically-based "inverse rendering" can be used to estimate how light travels within a scene, turning objects, like coffee cups or picture frames, into "object cameras". Physical models are quite fragile to small errors in estimated scene parameters. As such, we suggest reconstruction methods that make use of the uncertainty in scene parameters to improve robustness.
The thesis suggests a number of other interesting ways hidden cues may be used in combination with imaging systems. This work could inspire future cameras that incorporate the environment itself as part of the imaging system, blurring the line between observer and subject. |
| first_indexed | 2024-09-23T10:57:31Z |
| format | Thesis |
| id | mit-1721.1/151975 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T10:57:31Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1519752023-09-01T03:39:06Z Computational Discovery of Hidden Cues in Photographs Swedish, Tristan Raskar, Ramesh Raskar, Ramesh Program in Media Arts and Sciences (Massachusetts Institute of Technology) Images of everyday scenes often contain hidden information that can be extracted to localize objects outside the view of the camera and to see around corners. For example, we show that it is possible to look at shadows cast by an object on a table, such as a teapot, and reconstruct an image of the surrounding room. We describe how to identify and make use of these hidden cues such as shadows, reflections, and other subtle changes in an image caused by the interaction of light with objects in a scene that are not in the direct-line-of-sight. We use the term computational discovery to describe techniques that can be used to uncover these cues and reveal hidden information. Despite incredible advances in computer vision in recent years, cameras are limited to a single viewpoint of a scene, requiring invasive multi-camera setups or active imaging modalities to solve many perception tasks today. Prior work has identified hidden cues that are present in photographs of certain environments, but these methods often require human insight to identify cues, and extensive calibration to make use of them. In order to address the limitations found in prior work, we propose an end-to-end machine learning framework to identify hidden cues. More generally, we show that object localization is approximately equal to localizing a point light source, and describe how this insight can be used to identify situations when object localization is possible. Furthermore, we show that physically-based "inverse rendering" can be used to estimate how light travels within a scene, turning objects, like coffee cups or picture frames, into "object cameras". Physical models are quite fragile to small errors in estimated scene parameters. As such, we suggest reconstruction methods that make use of the uncertainty in scene parameters to improve robustness. The thesis suggests a number of other interesting ways hidden cues may be used in combination with imaging systems. This work could inspire future cameras that incorporate the environment itself as part of the imaging system, blurring the line between observer and subject. Ph.D. 2023-08-30T15:55:23Z 2023-08-30T15:55:23Z 2022-09 2023-08-16T20:45:52.440Z Thesis https://hdl.handle.net/1721.1/151975 0000-0002-5453-6121 In Copyright - Educational Use Permitted Copyright MIT http://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Swedish, Tristan Computational Discovery of Hidden Cues in Photographs |
| title | Computational Discovery of Hidden Cues in Photographs |
| title_full | Computational Discovery of Hidden Cues in Photographs |
| title_fullStr | Computational Discovery of Hidden Cues in Photographs |
| title_full_unstemmed | Computational Discovery of Hidden Cues in Photographs |
| title_short | Computational Discovery of Hidden Cues in Photographs |
| title_sort | computational discovery of hidden cues in photographs |
| url | https://hdl.handle.net/1721.1/151975 |
| work_keys_str_mv | AT swedishtristan computationaldiscoveryofhiddencuesinphotographs |