Evaluating Heuristics for Audio-Visual Translation

  • Dubbing, i.e., the lip-synchronous translation and revoicing of audio-visual media into a target language from a different source language, is essential for the full-fledged reception of foreign audio-visual media, be it movies, instructional videos or short social media clips. In this paper, we objectify influences on the ‘dubbability’ of translations, i.e., how well a translation would be synchronously revoiceable to the lips on screen. We explore the value of traditional heuristics used in evaluating the qualitative aspects, in particular matching bilabial consonants and the jaw opening while producing vowels, and control for quantity, i.e., that translations are similar to the source in length. We perform an ablation study using an adversarial neural classifier which is trained to differentiate “true” dubbing translations from machine translations. While we are able to confirm the value of matching lip closure in dubbing, we find that the opening angle of the jaw as determined by the realized vowel may be less relevant than frequently considered in audio-visual translation.

Export metadata

Additional Services

Share in Twitter Search Google Scholar Statistics
Author:Timo BaumannORCiDGND, Ashutosh Saboo
URL / DOI:http://ceur-ws.org/Vol-2989/short_paper46.pdf
Parent Title (English):Proceedings of the Conference of Computational Humanities Research, (CHR2021): November 17-19, 2021, Amsterdam, The Netherlands
Publisher:CEUR Workshops
Place of publication:Amsterdam, the Netherlands
Editor:Maud Ehrmann
Document Type:conference proceeding (article)
Year of first Publication:2021
Release Date:2022/04/11
Tag:ablation study; audiovisual translation; dubbing; lip synchrony; machine translation
First Page:171
Last Page:180
Andere Schriftenreihe:CEUR Workshop Proceedings ; 2989
Institutes:Fakultät Informatik und Mathematik
Publication:Externe Publikationen
research focus:Information und Kommunikation
Licence (German):Creative Commons - CC BY - Namensnennung 4.0 International