Evaluating Heuristics for Audio-Visual Translation
- Dubbing, i.e., the lip-synchronous translation and revoicing of audio-visual media into a target language from a different source language, is essential for the full-fledged reception of foreign audio-visual media, be it movies, instructional videos or short social media clips. In this paper, we objectify influences on the ‘dubbability’ of translations, i.e., how well a translation would be synchronously revoiceable to the lips on screen. We explore the value of traditional heuristics used in evaluating the qualitative aspects, in particular matching bilabial consonants and the jaw opening while producing vowels, and control for quantity, i.e., that translations are similar to the source in length. We perform an ablation study using an adversarial neural classifier which is trained to differentiate “true” dubbing translations from machine translations. While we are able to confirm the value of matching lip closure in dubbing, we find that the opening angle of the jaw as determined by the realized vowel may be less relevant than frequently considered in audio-visual translation.
Author: | Timo BaumannORCiDGND, Ashutosh Saboo |
---|---|
URN: | urn:nbn:de:gbv:18-228-7-2592 |
URL / DOI: | http://ceur-ws.org/Vol-2989/short_paper46.pdf |
Parent Title (English): | Proceedings of the Conference of Computational Humanities Research, (CHR2021): November 17-19, 2021, Amsterdam, The Netherlands |
Publisher: | CEUR Workshops |
Place of publication: | Amsterdam, the Netherlands |
Editor: | Maud Ehrmann |
Document Type: | conference proceeding (article) |
Language: | English |
Year of first Publication: | 2021 |
Release Date: | 2022/04/11 |
Tag: | ablation study; audiovisual translation; dubbing; lip synchrony; machine translation |
First Page: | 171 |
Last Page: | 180 |
Andere Schriftenreihe: | CEUR Workshop Proceedings ; 2989 |
Institutes: | Fakultät Informatik und Mathematik |
Begutachtungsstatus: | peer-reviewed |
Publication: | Externe Publikationen |
research focus: | Information und Kommunikation |
Licence (German): | ![]() |