Growing needs in translating multimedia content have resulted in Neural Machine Translation (NMT) gradually becoming an established practice in the field of subtitling. Contrary to text translation, subtitling is subject to spatial and temporal constraints, which greatly increase the postprocessing effort required to restore the NMT output to a proper subtitle format. In this work, we explore whether existing subtitling corpora conform to the constraints of: 1) length and reading speed; and 2) proper line breaks. We show that the process of creating parallel sentence alignments removes important time and line break information and propose practices for creating resources for subtitlingoriented NMT faithful to the subtitle format.
Machine Translation (MT) of subtitles is a growing need for various applications, given the amounts of online multimedia content becoming available daily. Subtitling translation is a complex process consisting of several stages (transcription, translation, timing), and manual approaches to the task are laborious and costly. Subtitling has to conform to spatial constraints such as length, and temporal constraints such as reading speed. While length and reading speed can be modelled as a post-processing step in an MT workflow using simple rules, subtitle segmentation, i.e. where and if to insert a line break, depends on semantic and syntactic properties. Subtitle segmentation is particularly important, since it has been shown that a
Copyright c 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
proper segmentation by phrase or sentence
significantly reduces reading time and improves
comprehension
Hence, there is ample room for developing fully or at least partially automated solutions for subtitle-oriented NMT, which would contribute in reducing post-processing effort and speedingup turn-around times. Automated approaches though, especially NMT, are data-hungry. Performance greatly depends on the availability of large amounts of high-quality data (up to tens of millions of parallel sentences), specifically tailored for the task. In the case of subtitle-oriented NMT, this implies having access to large subtitle training corpora. This leads to the following question: What should data specifically tailored for subtitling-oriented NMT look like?
There are large amounts of available parallel
data extracted from subtitles
E´ un grande onore venire 2 00:00:18,820 > 00:00:22,820 su questo palco due volte.
Vi sono estremamente grato.
Grazie mille, Chris.
E´ un grande onore venire su questo palco due volte.
Vi sono estremamente grato. titles on preserving the subtitling constraints.
In this work, we explore whether the large,
publicly available parallel data compiled from
subtitles conform to the temporal and spatial
constraints necessary for achieving quality subtitles.
We compare the existing resources to an
adaptation of MuST-C
Building an end-to-end subtitle-oriented translation system poses several challenges, mainly related to the fact that NMT training needs large amounts of high-quality data representative of the target application scenario (subtitling in our case). Human subtitlers translate either directly from the audio/video or they are provided with a template with the source text already in the format of subtitles containing time codes and line breaks, which they have to adhere to when translating.
Several projects have attempted to collect
parallel subtitling corpora. The most well-known one is
the OpenSubtitles1 corpus
The Japanese-English Subtitle Corpus
JESC
A corpus preserving both subtitle segmentation
and order of lines is SubCo
Slightly deviating from the domain of films and
TV series, corpora for Spoken Language
Translation (SLT) have been created based on TED talks.
The Web Inventory of Transcribed and Translated
Talks
Subtitle segmentation techniques have so far focused on monolingual subtitle data. A´ lvarez et al. (2014) trained Support Vector Machine and Logistic Regression classifiers on correctly/incorrectly segmented subtitles to predict line breaks. Extending this work, A´ lvarez et al. (2017) used a Conditional Random Field (CRF) classifier for the same task, also differentiating between line breaks (next subtitle line) and subtitle breaks (next subtitle block). Recently, Song et al. (2019) employed a Long-Short Term Memory Network (LSTM) to predict the position of the period in order to improve the readability of automatically generated Youtube captions. To our knowledge to date, there is no approach attempting to learn bilingual subtitle segmentation or incorporating subtitle segmentation in an end-to-end NMT system.
3.1
The quality of the translated subtitles is not
evaluated only in terms of fluency and adequacy, but
also based on their format. We assess whether
the available subtitle corpora conform to the
constraints of length, reading speed (for the corpora
where time information is available) and proper
line breaks on the basis of the criteria for
subtitle segmentation mentioned in the literature of
Audiovisual Translation (AVT)
2https://www.ted.com/participate/translate/guidelines 2. Lines per subtitle. Subtitles should not take up too much space on screen. The space allowed for a subtitle is about 20% of screen space. Therefore, a subtitle block should not exceed 2 lines. 3. Reading speed. The on-air time of a subtitle should be sufficient for the audience to read and process its content. The subtitle should match as much as possible the start and the end of an utterance. The duration of the utterance (measured either in seconds or in feet/frames) is directly equivalent to the space a subtitle should occupy. As a general rule, we consider max. 21 chars/second. 4. Preserve ‘linguistic wholes’. This criterion is related to subtitle segmentation. Subtitle segmentation does not rely only on the allowed length, but should respect linguistic norms. To facilitate readability, subtitle splits should not “break” semantic and syntactic units. In an ideal case, every subtitle line (or at least subtitle block) should represent a coherent linguistic chunk (i.e. a sentence or a phrase). For example, a noun should not be separated from its article. Lastly, subtitles should respect natural pauses. 5. Equal length of lines. Another criterion for splitting subtitles relates to aesthetics. There is no consensus about whether the top line should be longer or shorter, however, it has been shown that subtitle lines of equal length are easier to read, because the viewer’s eyes return to the same point on the screen when reading the second line.
While subtitle length and reading speed are factors that can be controlled directly by the subtitle software used by translators, subtitle segmentation is left to the decision of the translator. Translators often have to either compromise the aesthetics in favour of the linguistic wholes or resort to omissions and substitutions. Therefore, modelling the segmentation decisions based on the large available corpora is of great importance for a highquality subtitle-oriented NMT system. 3.2
In order to assess the conformity of the existing subtitle corpora to the constraints mentioned above, we implement the following filters. Characters per line (CPL): As mentioned above, the information about line breaks inside subtitle blocks is discarded in the process of creating parallel data. Therefore, we can only assume that a subtitle fulfils the criteria 1 and 2 above by calculating the maximum possible length for a subtitle block; 2 * 42 = 84 characters for Latin scripts and 2 * 14 = 28 for Japanese. If CP L > max length then the subtitle doesn’t conform to the length constraints.
lates to reading speed. For the corpora where time codes and duration are preserved, we calculate CPS as follows: CP S = d#urchatairosn Chink-Chunk: Chink-Chunk is a low-level parsing algorithm which can be used as a rulebased method to insert line breaks between subtitles. It is a simple but efficient way to detect syntactic boundaries. It relates to preserving linguistic wholes, since it uses POS information to split units only at punctuation marks (logical completion) or when an open-class or content word (chunk) is followed by a closed-class or function word (chink). Here, we use this algorithm to compute statistics about the type of subtitle block breaks in the data (punctuation break, content-function break or other). The algorithm is described in Algorithm 1.
Algorithm 1: Chink-Chunk algorithm 1 if POS last in [’PUNCT’, ’SYM’, ’X’] then 2 punc break +=1; 3 else 4 if POS last in content words and POS next in function words then
cf break +=1; 5 6 else 7 other split +=1; 8 end 9 end 10 return punc break, cf break, other split 4
For our experiments we consider the corpora which are large enough to train NMT systems; OpenSubtitles, JESC and MuST-C. We focus on 3 language pairs, Japanese, Italian and German, paired with English, as languages coming from different families and having a large portion of sentences in all corpora. We tokenise and then tag the data with Universal Dependencies3 to obtain POS tags for the Chink-Chunk algorithm.
To observe the effect of merging processes on preserving the subtitling constraints, we create a version of MuST-C at a subtitle level. We obtain
EN-IT EN-DE EN-JA
253K / 4.8M 229K / 4.2M the same .srt files used to create MuST-C. We extract only the subtitles with matching timestamps from the common talks in the language pair without any merging operations. Table 2 shows the statistics of the extracted corpus. We randomly sample 1,000 sentence pairs and manually inspect their alignments. 94% were correctly aligned, 3% partially aligned and 3% misaligned.
We apply each of the criteria filters in Section 3.2 to the corpora both on the source and the target side independently. Then, we take the intersection of the outputs of all the filters to obtain the lines/sentences which conform to all the criteria. 5
Length: The analysis of Characters per line filter shows that both OpenSubtitles and JESC conform to the quality criterion of length in at least 94% of the cases. Despite the merging operations to obtain sentence alignments, OpenSubtitles still preserves a short length of lines, possibly because of the nature of the text of subtitles. A manual inspection shows that the text is mainly short dialogues and the long sentences are parts of descriptions or monologues, which are more rare. On the other hand, the merging operations in MuSTC create long sentences that do not resemble the subtitling format. This could be attributed to the format of TED talks. TED talks mostly contain text written to be spoken, prepared talks usually delivered by one speaker with few dialogue turns. Among all corpora, MuST-C subs shows the highest conformity to the criterion of length, since indeed no merging operations were performed. Reading speed: Conformity to the criterion of reading speed is achieved to a lesser degree, as EN-IT EN-DE EN-JA
Corpus MuST-C OpenSubtitles MuST-C subs MuST-C segment X OpenSubtitles segment MuST-C subs subtitle X OpenSubtitles segment JESC subtitle MuST-C subs subtitle X segment X segment subtitle X shown by the Characters per second filter. Except for Japanese, where the allowed number of characters per line is lower, all other languages range between 66%-86%. In general, MuST-C subs, being in subtitling format, seems to conform better to reading speed. Unfortunately, time information is not present in corpora other than the two versions of MuST-C, therefore a full comparison is not possible.
Linguistic wholes: The Chink-Chunk algorithm shows interesting properties of the subtitle breaks for all the corpora. MuST-C and OpenSubtitles conform to the criterion of preserving linguistic wholes in 99% of the sentences, which does not occur in the corpora in subtitle format; JESC and MuST-C subs. Since these two corpora are compiled by removing captions based on unmatched time codes, the integrity of the documents is possibly broken. Subtitles are removed arbitrarily, so consecutive subtitles are often not kept in order. This shows the importance of preserving the order of subtitles when creating subtitling corpora.
This observation might lead to the assumption that JESC and MuST-C subs are less subtitle-like. However, a close inspection of the breaks shows that OpenSubtitles and MuST-C end in a punctuation mark in 99.9% of the cases. Even though they preserve logical completion, these corpora do not contain sufficient examples of line breaks preserving linguistic wholes. On the other hand, the subtitle-level corpora contain between 5%-11% subtitle breaks in the form of content-function word. In a realistic subtitling scenario, an NMT system at inference time will often receive unfinished sentences, either from an audio stream or a subtitling template. Therefore, line break information might be valuable for training NMT systems that learn to translate and segment.
The total retained material shows that OpenSubtitles is the most suitable corpus for producing quality subtitles in all investigated languages, as more than 90% of the sentences passed the filters. However, this is not a fair comparison, given that the data was filtered with only 2 out of the 3 filters. One serious limitation of OpenSubtitles is the lack of time information, which does not allow for modelling reading speed. We showed that corpora in subtitling format (JESC, MuST-C subs) contain useful information about line breaks not ending in punctuation marks, which are mostly absent from OpenSubtitles. Since no information about subtitle line breaks (inside a subtitle block) is preserved in any of the corpora, the criterion of equal length of lines cannot be explored in this study. 6
We explored whether the existing parallel subtitling resources conform to the subtitling constraints. We found that subtitling corpora generally conform to length and proper line breaks, despite the merging operations for aligning parallel sentences. We isolated some missing elements: the lack of time information (duration of utterance) and the insufficient representation of line breaks other than at punctuation marks.
This raises several open issues for creating corpora for subtitling-oriented NMT; i) subtitling constraints: a subtitling corpus, in order to be representative of the task, should respect the subtitling constraints; ii) duration of utterance: since the translation of a subtitle depends on the duration of the utterance, time information is highly relevant; iii) integrity of documents: a subtitle often occupies several lines, therefore the order of subtitles should be preserved whenever possible; iv) line break information: while parallel sentence alignments are indispensable, they should not compromise line break and subtitle block information. Break information could be preserved by inserting special symbols.
We intend to use these observations for an adaptation of MuST-C, containing triplets (audio, source language subtitle, target language subtitle), preserving line break information and taking advantage of natural pauses in the audio. In the long run, we would like to train NMT systems which predict line breaks while translating, possibly extending the input context using methods from document level translation.
This work is part of a project financially supported by an Amazon AWS ML Grant.