Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing
Recently, text-driven video editing has received increasing attention due to the surprising success of the text-to-image model in improving video quality. However, video editing based on the text prompt is facing huge challenges in achieving precise and controllable editing. Herein, we propose Laten...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-12-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/13/1/7 |
| _version_ | 1797358910812717056 |
|---|---|
| author | Liangbing Zhao Zicheng Zhang Xuecheng Nie Luoqi Liu Si Liu |
| author_facet | Liangbing Zhao Zicheng Zhang Xuecheng Nie Luoqi Liu Si Liu |
| author_sort | Liangbing Zhao |
| collection | DOAJ |
| description | Recently, text-driven video editing has received increasing attention due to the surprising success of the text-to-image model in improving video quality. However, video editing based on the text prompt is facing huge challenges in achieving precise and controllable editing. Herein, we propose Latent prompt Image-driven Video Editing (LIVE) with a precise and controllable video editing function. The important innovation of LIVE is to utilize the latent codes from reference images as latent prompts to rapidly enrich visual details. The novel latent prompt mechanism endows two powerful capabilities for LIVE: one is a comprehensively interactive ability between video frame and latent prompt in the spatial and temporal dimensions, achieved by revisiting and enhancing cross-attention, and the other is the efficient expression ability of training continuous input videos and images within the diffusion space by fine-tuning various components such as latent prompts, textual embeddings, and LDM parameters. Therefore, LIVE can efficiently generate various edited videos with visual consistency by seamlessly replacing the objects in each frame with user-specified targets. The high-definition experimental results from real-world videos not only confirmed the effectiveness of LIVE but also demonstrated important potential application prospects of LIVE in image-driven video editing. |
| first_indexed | 2024-03-08T15:09:02Z |
| format | Article |
| id | doaj.art-0b21446784e04f579bf2573ae15feab8 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-08T15:09:02Z |
| publishDate | 2023-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-0b21446784e04f579bf2573ae15feab82024-01-10T14:54:06ZengMDPI AGElectronics2079-92922023-12-01131710.3390/electronics13010007Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video EditingLiangbing Zhao0Zicheng Zhang1Xuecheng Nie2Luoqi Liu3Si Liu4School of Computer Science and Engineering, Beihang University, Beijing 100191, ChinaSchool of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, ChinaMeitu Inc., 7th Floor, Block B/C, Yousheng Building, 28 Chengfu Road, Haidian District, Beijing 100083, ChinaMeitu Inc., 7th Floor, Block B/C, Yousheng Building, 28 Chengfu Road, Haidian District, Beijing 100083, ChinaInstitute of Artificial Intelligence, Beihang University, Beijing 100191, ChinaRecently, text-driven video editing has received increasing attention due to the surprising success of the text-to-image model in improving video quality. However, video editing based on the text prompt is facing huge challenges in achieving precise and controllable editing. Herein, we propose Latent prompt Image-driven Video Editing (LIVE) with a precise and controllable video editing function. The important innovation of LIVE is to utilize the latent codes from reference images as latent prompts to rapidly enrich visual details. The novel latent prompt mechanism endows two powerful capabilities for LIVE: one is a comprehensively interactive ability between video frame and latent prompt in the spatial and temporal dimensions, achieved by revisiting and enhancing cross-attention, and the other is the efficient expression ability of training continuous input videos and images within the diffusion space by fine-tuning various components such as latent prompts, textual embeddings, and LDM parameters. Therefore, LIVE can efficiently generate various edited videos with visual consistency by seamlessly replacing the objects in each frame with user-specified targets. The high-definition experimental results from real-world videos not only confirmed the effectiveness of LIVE but also demonstrated important potential application prospects of LIVE in image-driven video editing.https://www.mdpi.com/2079-9292/13/1/7video editingdiffusion modelattention control |
| spellingShingle | Liangbing Zhao Zicheng Zhang Xuecheng Nie Luoqi Liu Si Liu Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing Electronics video editing diffusion model attention control |
| title | Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing |
| title_full | Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing |
| title_fullStr | Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing |
| title_full_unstemmed | Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing |
| title_short | Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing |
| title_sort | cross attention and seamless replacement of latent prompts for high definition image driven video editing |
| topic | video editing diffusion model attention control |
| url | https://www.mdpi.com/2079-9292/13/1/7 |
| work_keys_str_mv | AT liangbingzhao crossattentionandseamlessreplacementoflatentpromptsforhighdefinitionimagedrivenvideoediting AT zichengzhang crossattentionandseamlessreplacementoflatentpromptsforhighdefinitionimagedrivenvideoediting AT xuechengnie crossattentionandseamlessreplacementoflatentpromptsforhighdefinitionimagedrivenvideoediting AT luoqiliu crossattentionandseamlessreplacementoflatentpromptsforhighdefinitionimagedrivenvideoediting AT siliu crossattentionandseamlessreplacementoflatentpromptsforhighdefinitionimagedrivenvideoediting |