Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions
We introduce visual deprojection: The task of recovering an image or video that has been collapsed along a dimension. Projections arise in various contexts, such as long-exposure photography, where a dynamic scene is collapsed in time to produce a motion-blurred image, and corner cameras, where refl...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021
|
| Online Access: | https://hdl.handle.net/1721.1/129833 |
| _version_ | 1811079944726380544 |
|---|---|
| author | Balakrishnan, Guha Dalca, Adrian Vasile Zhao, Amy (Xiaoyu Amy) Guttag, John V Durand, Frédo Freeman, William T |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Balakrishnan, Guha Dalca, Adrian Vasile Zhao, Amy (Xiaoyu Amy) Guttag, John V Durand, Frédo Freeman, William T |
| author_sort | Balakrishnan, Guha |
| collection | MIT |
| description | We introduce visual deprojection: The task of recovering an image or video that has been collapsed along a dimension. Projections arise in various contexts, such as long-exposure photography, where a dynamic scene is collapsed in time to produce a motion-blurred image, and corner cameras, where reflected light from a scene is collapsed along a spatial dimension because of an edge occluder to yield a 1D video. Deprojection is ill-posed - often there are many plausible solutions for a given input. We first propose a probabilistic model capturing the ambiguity of the task. We then present a variational inference strategy using convolutional neural networks as functional approximators. Sampling from the inference network at test time yields plausible candidates from the distribution of original signals that are consistent with a given input projection. We evaluate the method on several datasets for both spatial and temporal deprojection tasks. We first demonstrate the method can recover human gait videos and face images from spatial projections, and then show that it can recover videos of moving digits from dramatically motion-blurred images obtained via temporal projection. |
| first_indexed | 2024-09-23T11:22:58Z |
| format | Article |
| id | mit-1721.1/129833 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T11:22:58Z |
| publishDate | 2021 |
| publisher | IEEE |
| record_format | dspace |
| spelling | mit-1721.1/1298332022-10-01T03:15:46Z Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions Balakrishnan, Guha Dalca, Adrian Vasile Zhao, Amy (Xiaoyu Amy) Guttag, John V Durand, Frédo Freeman, William T Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science We introduce visual deprojection: The task of recovering an image or video that has been collapsed along a dimension. Projections arise in various contexts, such as long-exposure photography, where a dynamic scene is collapsed in time to produce a motion-blurred image, and corner cameras, where reflected light from a scene is collapsed along a spatial dimension because of an edge occluder to yield a 1D video. Deprojection is ill-posed - often there are many plausible solutions for a given input. We first propose a probabilistic model capturing the ambiguity of the task. We then present a variational inference strategy using convolutional neural networks as functional approximators. Sampling from the inference network at test time yields plausible candidates from the distribution of original signals that are consistent with a given input projection. We evaluate the method on several datasets for both spatial and temporal deprojection tasks. We first demonstrate the method can recover human gait videos and face images from spatial projections, and then show that it can recover videos of moving digits from dramatically motion-blurred images obtained via temporal projection. United States. Defense Advanced Research Projects Agency. Revolutionary Enhancement of Visibility by Exploiting Active Light-fields Program (Contract HR0011-16-C-0030) National Institutes of Health (U.S.) (Grant 1R21AG050122) 2021-02-19T15:02:36Z 2021-02-19T15:02:36Z 2019-10 2020-12-11T17:36:04Z Article http://purl.org/eprint/type/ConferencePaper 9781728148038 1550-5499 https://hdl.handle.net/1721.1/129833 Balakrishnan, Guha et al. “Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions.” Paper in the Proceedings of the IEEE International Conference on Computer Vision, 2019-October, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 27 October-2 November 2019, IEEE © 2019 The Author(s) en 10.1109/ICCV.2019.00026 Proceedings of the IEEE International Conference on Computer Vision Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf IEEE arXiv |
| spellingShingle | Balakrishnan, Guha Dalca, Adrian Vasile Zhao, Amy (Xiaoyu Amy) Guttag, John V Durand, Frédo Freeman, William T Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions |
| title | Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions |
| title_full | Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions |
| title_fullStr | Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions |
| title_full_unstemmed | Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions |
| title_short | Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions |
| title_sort | visual deprojection probabilistic recovery of collapsed dimensions |
| url | https://hdl.handle.net/1721.1/129833 |
| work_keys_str_mv | AT balakrishnanguha visualdeprojectionprobabilisticrecoveryofcollapseddimensions AT dalcaadrianvasile visualdeprojectionprobabilisticrecoveryofcollapseddimensions AT zhaoamyxiaoyuamy visualdeprojectionprobabilisticrecoveryofcollapseddimensions AT guttagjohnv visualdeprojectionprobabilisticrecoveryofcollapseddimensions AT durandfredo visualdeprojectionprobabilisticrecoveryofcollapseddimensions AT freemanwilliamt visualdeprojectionprobabilisticrecoveryofcollapseddimensions |