Dubbing, i.e., the lip-synchronous translation and revoicing of audio-visual media into a target language from a diferent source language, is essential for the full-fledged reception of foreign audio-visual media, be it movies, instructional videos or short social media clips. In this paper, we objectify inlfuences on the 'dubbability' of translations, i.e., how well a translation would be synchronously revoiceable to the lips on screen. We explore the value of traditional heuristics used in evaluating the qualitative aspects, in particular matching bilabial consonants and the jaw opening while producing vowels, and control for quantity, i.e., that translations are similar to the source in length. We perform an ablation study using an adversarial neural classifier which is trained to diferentiate “true” dubbing translations from machine translations. While we are able to confirm the value of matching lip closure in dubbing, we find that the opening angle of the jaw as determined by the realized vowel may be less relevant than frequently considered in audio-visual translation.
Dubbing is studied in audio-visual translation [
Quality is important once quantity is established, and is concerned with matching visemic characteristics (i.e., what speech sounds looks like when pronounced) of source and target speech, such as opening angle of the jaw for vowels and lip closure for consonants (e.g., when there is a /b/ in SL, prefer a translation that features one of /m b p/ at that time over one that features /g/, to match lip closure). Quality is often characterized by the heuristic of ifnding a translation that ‘best matches phonetically’ the source language as it is visible on screen, as estimated by the human audio-visual translator. Although the idea of ‘best matching phonetically’ is intuitively plausible, there is a research gap on objective and computational source (en): No, no. Each individual's blood chemistry is unique, like fingerprints.
dubbed (es): No, no. La sangre de cada individuo es única, como una huella.
ideal MT (Google): No, no. La química de la sangre de cada individuo es única, como las huellas dactilares. measures for the dubbing quality of a given translation, which we aim to fill with this paper. Our long-term goal is to automatically generate a translated script which can be revoiced easily to yield dubbed film that transparently appears as if it had been recorded in the target language all along.
There is some limited recent work [19] on establishing quantitative similarity for dubbing in machine translation (MT). Here, we specifically explore the qualitative factors of speech sounds that may be important beyond matching syllable counts while controlling for quantity.
The need for objective measures of dubbing optimality of a given translation arises from the fact that most MT systems are trained on textual material that does not regard dubbing optimality, of which corpora are available that exceed the size of dubbed material by several orders of magnitude. Even subtitles do not fully cover dubbing characteristics. As a result, high-performance MT does not have an implicit notion of dubbing optimality and yields results that are not directly suitable for dubbing, although optimal for textual translations. It is our goal to estimate the importance of qualitative matching between SL and TL and to later add these as constraints to the translation process.
A way of enriching MT with external constraints is described in the following section and builds on heuristics that can be evaluated on partial or full translations of utterances. We use this method to balance MT for quantitative similarity as a basis for our analysis of factors that influence qualitative similarity using an ablation study that employs an adversarial classifier. Our empirical analysis confirms the importance of qualitative similarity and matching lip closures in dubbing. We find that the opening angle of the jaw is comparatively less relevant for dubbing.
Translation from one language to another aims to be a meaning-preserving conversion (typically of text but also of speech) from a source to a target language (and, to a lesser extent, from one socio-cultural context to another). Audio-visual translation adds the constraint that the target language material shall closely match the visemic characteristics of the source to give the impression that a video of the source speaker actually shows the speaker speaking TL when machine translation
speech synthesis most most faithful translation natural speech → not synchronous → not synchronous video adaptation no adaptation → no artifacts
human evaluation overall quality perception revoiced by a dubbing artist.
A perfect dubbing is not always possible given that the same meaning in two languages is expressed with diferent syntactic structures, diferent words, and resulting in diferent speech sounds (and accentuation patterns) that yield diferent articulatory characteristics (visemes such as the opening or closing of the lips and jaw). Thus, a tradeof must be found between meaning preservation and dubbability. Figure 1 shows, as an example, one original and dubbed utterance in a TV show, as well as the machine translation of the source to the target language via Google translate. We find that MT performs quite well and yields a meaning-preserving translation, which however is substantially longer. In contrast, the dubbed version changes the syntactic structure and uses synonymy to leave out material (ignoring the ‘chemistry’ aspect of blood and the ‘finger’ aspect of the print), yielding a more dubbable text.
The translation for dubbing is clearly geared towards more easily ‘dubbable’ text and it is then the dubbing artist’s task to speak the material in such a way that it appears as natural as possible given the video of the original speaker.
A full dubbing system that were to cover both translation and speech synthesis as well as potential video adaptation should yield a solution that is globally optimized towards the user perception as sketched in Figure 2: it can be wise to choose a sub-optimal translation to yield a better overall synchrony of the system.
Neural machine translation (NMT) has become a popular approach for MT, originally
proposed by [
In previous work [19], we enriched a translation system with an external dubbing optimality scorer to yield controllable and dubbing-optimal translations, however, only for the quantity of TL material produced. We here explore the influence and relative importance of qualitative aspects in human dubbing.
Dubbing optimality is majorly governed by lip synchrony and opening angle of the jaw and, of course, quantity of speech (which is often taken as granted in the research literature on dubbing). We first describe previous results in literature on quantitative measures [ 19] and then how we use these to establish and analyze qualitative measures based on an adversarial approach in which we aim to train a classifier that attempts to diferentiate human goldstandard dubbing from quantitatively re-balanced MT. If this classifier performs poorly, then MT is more difficult to distinguish from gold-standard translation. We then validate various qualitative factors, such as importance of the opening angle of the jaws, closure of the lips, prosody and word boundaries by performing ablation studies with this classifier.
In order to allow for an even approximate lip synchrony, the duration of the revoicing should match that of the original speech, in order to avoid audio-visual phantom efects. These can be seemingly ‘stray’ movements of the mouth in the dubbed version if there is too little to speak, or audible speech while the articulators are not moving if there is too much material to speak. As in [19], we use the number of syllables as the primary indicator of visemic similarity via the standard hyphenation library Pyphen1 for counting the number of syllables in SL as well as candidates in TL and take the relative diference of the two as the similarity metric. 2 We then rescore the NMT’s output by the similarity metric using some weight α. For the experiments below, we report results across the range for all 0 ≤ α < 1; [19] found an α ∼ 0.3 to yield the best balance between BLEU score (a measure for translation quality, [18]) and quantitative similarity. We will therefore highlight the results in the 0.2 ≤ α ≤ 0.5-range.
Qualitative similarity, i.e., the dubbing artist’s voice closely matching the articulatory
movements visible on screen, is also highly desirable, beyond quantitative matching. Phonetic
aspects of consonants and vowels, such as lip closure and opening angle of the jaw have been
reported as being relevant for translations that can be lip-synchronously dubbed, as well as
supra-segmental aspects such as prosodic phrasing [
We explore the relative importance of these aspects using an ablation experiment on a classifier that is trained to diferentiate human dubbing translations from NMT translations (rescored to yield quantitative similarity). For MT that is ideal for dubbing, this classifier performs poorly, whereas it performs better, the more easily gold-standard dubbing and MT can be diferentiated. In essence, if the features that the classifier is deprived of in an ablation setting are not relevant, the performance of the classifier should not reduce (or even improve); if however the classifier is deprived of relevant features, we expect a performance degradation. 1Pyphen: https://pyphen.org.
2This works well for English-Spanish translation; other language pairs may require other quantitative measures, e. g. mora-driven languages.
We here explore the importance of phonetic/visemic characteristics via diferent simplifications of the textual material that we feed to the classifier. For example, when we leave out all whitespace and punctuation, the classifier is deprived of morphological and prosodic structure features. If its performance drops (relative to the full input) this reflects the influence on dubbability. Note that Spanish, TL in our experiments, has highly regular grapheme-phoneme correspondences which allows to base our experiment directly on ablations of the graphemic representations.
We use the following simplifications in addition to passing the full text to the classifier ( full): no punctuation tests the influence of phrasing as far as expressed by punctuation in text, no whitespace (in addition to no punctuation): tests the importance of word boundaries; we hypothesize that word boundaries are of little relevance when dubbing as they are not clearly observable in continuous speech.
In addition to whitepace and punctuation removal: vowels vs. C we replace all consonants by “C ” but not the vowels, to test how opening angle of the jaw alone (which, to a large extent depends on the vowel produced) helps the model, consonants vs. V we replace all vowels by “V ”; as a result, the opening angle of the jaw is not observable to the model, bilabials vs. C vs. V we replace all vowels (“V ”) and consonants (“C ”) except for bilabials (“b, p, m”) which are not replaced; thus, lip closure is the only consonant characteristic observable to the model, C vs. V to test if syllable structure alone is valuable for dubbing optimality.
For our method, we use an encoder-encoder architecture with siamese parameters for the two
TL candidates to be compared based on the SL sentence as depicted in Figure 3. We first encode
the SL sentence bidirectionally character-by-character using an RNN based on GRUs [
We use the HEROes corpus [
softmax decision layer
subtraction Dubbed Spanish text
Machine-Translated Spanish text RNN
RNN ...
RNN
RNN
RNN ...
RNN ... characters from text
... characters from text
s see tree isam raam p Source English text
attention RNN RNN
RNN ...
RNN ...
RNN
RNN ...
characters from text hours of speech that are based on forced alignment of video subtitles to the audio tracks. The results have been manually checked and re-aligned to each other.
We trained an NMT system on the OpenSubtitles corpus [
Our classifier is trained on the triples of SL, TL dubbing, and TL NMT candidate for all
text simplifications and all values of α, using 10-fold cross validation on the corpus. We report
the overall accuracy for each classification setting as well as the standard deviation across folds.
The classifier is implemented in DyNet [ 13].3 We use 20-dimensional character encodings,
20dimensional RNN states, and 20-dimensional attention. We train with the Adam method [
The results of the experiments are presented in Figure 4, where the x-axis denotes the control over quantitative similarity (via the rescoring factor α) and the y-axis denotes the classifier 3The code and full experimental data are available at https://github.com/timobaumann/duboptimal. accuracy percentage (an accuracy around 50 % indicates that the classifier is unable to diferentiate true dubbing). The standard deviation across folds for each of the values is in the range of 1-4 percentage points. Thus, while we have not performed significance tests across folds, we feel confident that diferences reported below are likely ‘real’.
The figure shows that the classifier performs best with full input. Translations in the relevant α-range that yields good quantity (but still reasonable translations) are more difficult, indicating that the adversarial task is particularly difficult under these circumstances. Only retaining the syllabic structure (C vs. V) yields the worst performance (only marginally above chance) and can be considered as hardly helpful – this debunks the common misunderstanding that all that needs to be kept in dubbing is the right number of syllables.
Leaving out punctuation and whitespace has some but not radical efects (probably within the margin of error) indicating that both prosodic phrasing and lexicomorphology do not need to be strictly retained while translating for dubbing; instead, these allow for some degree of freedom to better match other aspects.
Regarding lip closure and jaw movements, we find that (a) removing vowel information ( all consonants vs. V) only hurts a little, whereas retaining only vowel information (all vowels vs. C) leads to a considerable performance drop. From this we conclude that matching the opening angle of the jaw is at least not done through vowel choice, and may be less critical than described in the literature.
(b) In contrast, removing vowel information and even reducing the consonant information to whether it’s bilabials or not (bilabials vs. C vs. V) yields surprisingly high performance (even better than retaining all consonants, possibly because the model learns more easily with fewer input symbols) which indicates that lip closure is indeed observed tightly in the dubbing corpus.
We have studied the importance of aspects of qualitative similarity in dubbing, in particular when quantitative similarity is controlled for. The literature in translatology for dubbing posits that jaw movement and lip closure are critical aspects to be observed in dubbing. However we found no study prior to ours to investigate the relative importance of these aspects, measure the importance in an objective way, or investigate the importance of further potential influences such as lexicomorphology and prosodic phrasing. We have presented an ablation study to try to find those features that are particularly relevant to discern qualitatively ignorant NMT from true dubbing using a neural siamese classifier.
We find that we can confirm the importance of matching lip closures in dubbing. We therefore conclude that good dubbing requires a good matching of lip closures. By comparison, the opening angle of the jaw (which intrinsically varies between diferent vowel types) appears to be far less important. Our quantification of dubbing constraints leads the way towards a further optimization of machine translation for dubbing as it enables the training or adaptation procedure to take into account these constraints. Additionally, our classifier could directly be included into NMT via an adversarial learning procedure.
Our experiments yield objective evidence about the importance of qualitative aspects for dubbing. However, we acknowledge that further research is needed. In particular, our study is restricted to the textual form and does not include the speech signal in the corpus, which would allow for a better temporal alignment analysis. Furthermore, our analysis uses the full corpus rather than only those parts where the face is visible on-screen (and hence qualitative aspects matter).4 Finally, the ultimate evaluation gold standard for dubbing would be a user study that compares diferent dubbing alternatives. This could be used to directly optimize towards human judgements of dubbing alternatives (which might even difer with user preferences), or information retention for educational material to estimate distraction of less-than-ideal dubbing.
More broadly, we believe that ablation studies are a suitable tool in computational humanities research as they can help to objectively analyse and quantify the various aspects of existing humanistic theories for complex phenomena such as in this study.
The second author’s work was performed during an internship at Universität Hamburg which
was partially supported by Volkswagen Foundation under the funding codes 91926 and 93255.
We thank the anonymous reviewers for their insightful remarks.
4A tool for on- vs. of-screen detection has become available only very recently [