Motion-Compensated Viewpoint Shift
Eye contact is an essential social cue that conveys our attention to others but is difficult to maintain during video calls. Many existing methods to synthesize a gaze-corrected view involve estimating a 3D face model and projecting it into the desired camera view, which is too computationally expen...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2024
|
| Online Access: | https://hdl.handle.net/1721.1/156638 |
| _version_ | 1811074436222156800 |
|---|---|
| author | Tao, Julius L. |
| author2 | Lee, Chong U. |
| author_facet | Lee, Chong U. Tao, Julius L. |
| author_sort | Tao, Julius L. |
| collection | MIT |
| description | Eye contact is an essential social cue that conveys our attention to others but is difficult to maintain during video calls. Many existing methods to synthesize a gaze-corrected view involve estimating a 3D face model and projecting it into the desired camera view, which is too computationally expensive for most personal computers. By drawing inspiration from 2D methods of video frame interpolation, we wish to not only correct eye gaze but also better align the face towards the camera without this expensive 3D modeling. Our findings suggest that adding a second webcam opposite the first and interpolating between the two outer camera views can give realistic, gaze-aligned center views. We conclude that the prevailing approach of 3D modeling is surprisingly not necessary for gaze correction. Not only do 2D techniques suffice, but their synthesized frames can appear more natural than prior results. We believe that this work is a crucial step towards true-to-life viewpoint shift for live video conferences. |
| first_indexed | 2024-09-23T09:49:26Z |
| format | Thesis |
| id | mit-1721.1/156638 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T09:49:26Z |
| publishDate | 2024 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1566382024-09-04T03:51:45Z Motion-Compensated Viewpoint Shift Tao, Julius L. Lee, Chong U. Lim, Jae S. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Eye contact is an essential social cue that conveys our attention to others but is difficult to maintain during video calls. Many existing methods to synthesize a gaze-corrected view involve estimating a 3D face model and projecting it into the desired camera view, which is too computationally expensive for most personal computers. By drawing inspiration from 2D methods of video frame interpolation, we wish to not only correct eye gaze but also better align the face towards the camera without this expensive 3D modeling. Our findings suggest that adding a second webcam opposite the first and interpolating between the two outer camera views can give realistic, gaze-aligned center views. We conclude that the prevailing approach of 3D modeling is surprisingly not necessary for gaze correction. Not only do 2D techniques suffice, but their synthesized frames can appear more natural than prior results. We believe that this work is a crucial step towards true-to-life viewpoint shift for live video conferences. M.Eng. 2024-09-03T21:13:49Z 2024-09-03T21:13:49Z 2024-05 2024-07-11T14:36:17.388Z Thesis https://hdl.handle.net/1721.1/156638 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Tao, Julius L. Motion-Compensated Viewpoint Shift |
| title | Motion-Compensated Viewpoint Shift |
| title_full | Motion-Compensated Viewpoint Shift |
| title_fullStr | Motion-Compensated Viewpoint Shift |
| title_full_unstemmed | Motion-Compensated Viewpoint Shift |
| title_short | Motion-Compensated Viewpoint Shift |
| title_sort | motion compensated viewpoint shift |
| url | https://hdl.handle.net/1721.1/156638 |
| work_keys_str_mv | AT taojuliusl motioncompensatedviewpointshift |