Joint Fourier Disparity Layers Unrolling With Learned View Synthesis for Light Field Reconstruction From Few-Shots Focal Stacks
This paper addresses the problem of capturing a light field using a single traditional camera, by solving the inverse problem of dense light field reconstruction from a focal stack containing only very few images captured at different focus distances. An end-to-end joint optimization framework is pr...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10304112/ |
| _version_ | 1797632785714774016 |
|---|---|
| author | Brandon Le Bon Mikael Le Pendu Christine Guillemot |
| author_facet | Brandon Le Bon Mikael Le Pendu Christine Guillemot |
| author_sort | Brandon Le Bon |
| collection | DOAJ |
| description | This paper addresses the problem of capturing a light field using a single traditional camera, by solving the inverse problem of dense light field reconstruction from a focal stack containing only very few images captured at different focus distances. An end-to-end joint optimization framework is presented, where a novel unrolled optimization method is jointly optimized with a view synthesis deep neural network. The proposed unrolled optimization method constructs Fourier Disparity Layers (FDL), a compact representation of light fields which samples Lambertian non-occluded scenes in the depth dimension and from which all the light field viewpoints can be computed. Solving the optimization problem in the FDL domain allows us to derive a closed-form expression of the data-fit term of the inverse problem. Furthermore, unrolling the FDL optimization allows to learn a prior directly in the FDL domain. In order to widen the FDL representation to more complex scenes, a Deep Convolutional Neural Network (DCNN) is trained to synthesize novel views from the optimized FDL. We show that this joint optimization framework reduces occlusion issues of the FDL model, and outperforms recent state-of-the-art methods for light field reconstruction from focal stack measurements. |
| first_indexed | 2024-03-11T11:42:42Z |
| format | Article |
| id | doaj.art-9e3dd2998014495e867f5bb07d73322a |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T11:42:42Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-9e3dd2998014495e867f5bb07d73322a2023-11-10T00:00:39ZengIEEEIEEE Access2169-35362023-01-011112335012336010.1109/ACCESS.2023.332932810304112Joint Fourier Disparity Layers Unrolling With Learned View Synthesis for Light Field Reconstruction From Few-Shots Focal StacksBrandon Le Bon0https://orcid.org/0000-0003-0252-7335Mikael Le Pendu1Christine Guillemot2https://orcid.org/0000-0003-1604-967XINRIA Rennes—Bretagne Atlantique, Rennes, FranceINTERDIGITAL, Cesson-Sévigné, FranceINRIA Rennes—Bretagne Atlantique, Rennes, FranceThis paper addresses the problem of capturing a light field using a single traditional camera, by solving the inverse problem of dense light field reconstruction from a focal stack containing only very few images captured at different focus distances. An end-to-end joint optimization framework is presented, where a novel unrolled optimization method is jointly optimized with a view synthesis deep neural network. The proposed unrolled optimization method constructs Fourier Disparity Layers (FDL), a compact representation of light fields which samples Lambertian non-occluded scenes in the depth dimension and from which all the light field viewpoints can be computed. Solving the optimization problem in the FDL domain allows us to derive a closed-form expression of the data-fit term of the inverse problem. Furthermore, unrolling the FDL optimization allows to learn a prior directly in the FDL domain. In order to widen the FDL representation to more complex scenes, a Deep Convolutional Neural Network (DCNN) is trained to synthesize novel views from the optimized FDL. We show that this joint optimization framework reduces occlusion issues of the FDL model, and outperforms recent state-of-the-art methods for light field reconstruction from focal stack measurements.https://ieeexplore.ieee.org/document/10304112/Unrolled optimizationview synthesisjoint optimizationFourier disparity layerlight field reconstructionfocal stack |
| spellingShingle | Brandon Le Bon Mikael Le Pendu Christine Guillemot Joint Fourier Disparity Layers Unrolling With Learned View Synthesis for Light Field Reconstruction From Few-Shots Focal Stacks IEEE Access Unrolled optimization view synthesis joint optimization Fourier disparity layer light field reconstruction focal stack |
| title | Joint Fourier Disparity Layers Unrolling With Learned View Synthesis for Light Field Reconstruction From Few-Shots Focal Stacks |
| title_full | Joint Fourier Disparity Layers Unrolling With Learned View Synthesis for Light Field Reconstruction From Few-Shots Focal Stacks |
| title_fullStr | Joint Fourier Disparity Layers Unrolling With Learned View Synthesis for Light Field Reconstruction From Few-Shots Focal Stacks |
| title_full_unstemmed | Joint Fourier Disparity Layers Unrolling With Learned View Synthesis for Light Field Reconstruction From Few-Shots Focal Stacks |
| title_short | Joint Fourier Disparity Layers Unrolling With Learned View Synthesis for Light Field Reconstruction From Few-Shots Focal Stacks |
| title_sort | joint fourier disparity layers unrolling with learned view synthesis for light field reconstruction from few shots focal stacks |
| topic | Unrolled optimization view synthesis joint optimization Fourier disparity layer light field reconstruction focal stack |
| url | https://ieeexplore.ieee.org/document/10304112/ |
| work_keys_str_mv | AT brandonlebon jointfourierdisparitylayersunrollingwithlearnedviewsynthesisforlightfieldreconstructionfromfewshotsfocalstacks AT mikaellependu jointfourierdisparitylayersunrollingwithlearnedviewsynthesisforlightfieldreconstructionfromfewshotsfocalstacks AT christineguillemot jointfourierdisparitylayersunrollingwithlearnedviewsynthesisforlightfieldreconstructionfromfewshotsfocalstacks |