TY - CHAP A1 - Nayak, Shravan A1 - Baumann, Timo A1 - Bhattacharya, Supratik A1 - Karakanta, Alina A1 - Negri, Matteo A1 - Turchi, Marco ED - Truong, Khiet T1 - See me Speaking? Differentiating on Whether Words are Spoken On Screen or Off to Optimize Machine Dubbing T2 - ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction, 25.10.2020 - 29.10.2020; Virtual Event Netherlands N2 - Dubbing is the art of finding a translation from a source into a target language that can be lip-synchronously revoiced, i. e., that makes the target language speech appear as if it was spoken by the very actors all along. Lip synchrony is essential for the full-fledged reception of foreign audiovisual media, such as movies and series, as violated constraints of synchrony between video (lips) and audio (speech) lead to cognitive dissonance and reduce the perceptual quality. Of course, synchrony constraints only apply to the translation when the speaker's lips are visible on screen. Therefore, deciding whether to apply synchrony constraints requires an automatic method for detecting whether an actor's lips are visible on screen for a given stretch of speech or not. In this paper, we attempt, for the first time, to classify on- from off-screen speech based on a corpus of real-world television material that has been annotated word-by-word for the visibility of talking lips on screen. We present classification experiments in which we classify Y1 - 2020 SN - 9781450380027 U6 - https://doi.org/10.1145/3395035.3425640 SP - 130 EP - 134 PB - Association for Computing Machinery CY - New York,NY,United States ER - TY - CHAP A1 - Karakanta, Alina A1 - Bhattacharya, Supratik A1 - Nayak, Shravan A1 - Baumann, Timo A1 - Negri, Matteo A1 - Turchi, Marco ED - Scott, Donia ED - Bel, Nuria ED - Zong, Chengqing T1 - The Two Shades of Dubbing in Neural Machine Translation T2 - Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020): Barcelona, Spain (Online) N2 - Dubbing has two shades; synchronisation constraints are applied only when the actor’s mouth is visible on screen, while the translation is unconstrained for off-screen dubbing. Consequently, different synchronisation requirements, and therefore translation strategies, are applied depending on the type of dubbing. In this work, we manually annotate an existing dubbing corpus (Heroes) for this dichotomy. We show that, even though we did not observe distinctive features between on- and off-screen dubbing at the textual level, on-screen dubbing is more difficult for MT (-4 BLEU points). Moreover, synchronisation constraints dramatically decrease translation quality for off-screen dubbing. We conclude that, distinguishing between on-screen and off-screen dubbing is necessary for determining successful strategies for dubbing-customised Machine Translation. Y1 - 2020 U6 - https://doi.org/10.18653/v1/2020.coling-main.382 SP - 4327 EP - 4333 PB - International Committee on Computational Linguistics CY - Stroudsburg, PA, USA ER -