Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing
AbstractWe investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a num...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The MIT Press
2023-01-01
|
| Series: | Transactions of the Association for Computational Linguistics |
| Online Access: | https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00551/115968/Dubbing-in-Practice-A-Large-Scale-Study-of-Human |
| _version_ | 1797796601561874432 |
|---|---|
| author | William Brannon Yogesh Virkar Brian Thompson |
| author_facet | William Brannon Yogesh Virkar Brian Thompson |
| author_sort | William Brannon |
| collection | DOAJ |
| description |
AbstractWe investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (character length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as transfer of semantic properties such as emphasis and emotion, in automatic dubbing systems. |
| first_indexed | 2024-03-13T03:36:30Z |
| format | Article |
| id | doaj.art-7cd7c64a7ab346c5a7ac3360753a9343 |
| institution | Directory Open Access Journal |
| issn | 2307-387X |
| language | English |
| last_indexed | 2024-03-13T03:36:30Z |
| publishDate | 2023-01-01 |
| publisher | The MIT Press |
| record_format | Article |
| series | Transactions of the Association for Computational Linguistics |
| spelling | doaj.art-7cd7c64a7ab346c5a7ac3360753a93432023-06-23T18:58:09ZengThe MIT PressTransactions of the Association for Computational Linguistics2307-387X2023-01-011141943510.1162/tacl_a_00551Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic DubbingWilliam Brannon0Yogesh Virkar1Brian Thompson2MIT Media Lab, USA. wbrannon@mit.eduAWS AI Labs, USA. yvvirkar@amazon.comAWS AI Labs, USA. brianjt@amazon.com AbstractWe investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (character length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as transfer of semantic properties such as emphasis and emotion, in automatic dubbing systems.https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00551/115968/Dubbing-in-Practice-A-Large-Scale-Study-of-Human |
| spellingShingle | William Brannon Yogesh Virkar Brian Thompson Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing Transactions of the Association for Computational Linguistics |
| title | Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing |
| title_full | Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing |
| title_fullStr | Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing |
| title_full_unstemmed | Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing |
| title_short | Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing |
| title_sort | dubbing in practice a large scale study of human localization with insights for automatic dubbing |
| url | https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00551/115968/Dubbing-in-Practice-A-Large-Scale-Study-of-Human |
| work_keys_str_mv | AT williambrannon dubbinginpracticealargescalestudyofhumanlocalizationwithinsightsforautomaticdubbing AT yogeshvirkar dubbinginpracticealargescalestudyofhumanlocalizationwithinsightsforautomaticdubbing AT brianthompson dubbinginpracticealargescalestudyofhumanlocalizationwithinsightsforautomaticdubbing |