We present MonoTrans, a statistical machine translation system which only uses monolingual source language and target language data, without using any parallel corpora or language-specific rules. It translates each source word by the most similar target word, according to a combination of a string similarity measure and a word frequency similarity measure. It is designed for translation between very close languages, such as Czech and Slovak or Danish and Norwegian. It provides a lowquality translation in resource-poor scenarios where parallel data, required for training a high-quality translation system, may be scarce or unavailable. This is useful e.g. for cross-lingual NLP, where a trained model may be transferred from a resource-rich source language to a resourcepoor target language via machine translation. We evaluate MonoTrans both intrinsically, using BLEU, and extrinsically, applying it to cross-lingual tagger and parser transfer. Although it achieves low scores, it does surpass the baselines by respectable margins.
In machine translation (MT), the most common and most
successful approach is to train a translation model from
parallel text corpora, i.e. from a set of bilingual sentence
pairs with corresponding meanings. This approach was
pioneered by the IBM models [
Fortunately, parallel text corpora are a naturally
occurring resource. They can be mined from film subtitles, book
translations, documents published by international
institutions, software localization data, etc. Probably the largest
freely available collection of parallel data is Opus [
A common feature of language that is not usually
exploited in main-stream MT, is interlingual word
similarity. Typically, the systems treat the source language words
and the target language words completely independently,
usually by representing each word with a unique
identifier, with source word identifiers and target word
identifiers belonging to different domains. Only in case of
outof-vocabulary source words (OOVs), which are not part
of the available source language vocabulary and therefore
cannot be translated by the system, approaches that do
acknowledge potential interlingual word similarity are
sometimes applied, such as transliteration by a character-based
translation model [
In our work, we create a data-driven MT system for very close languages, based on utilizing only monolingual corpora and a set of heuristics with a high level of language independence. In this way, we target languages which are very low on available resources: the only resource we require for both the source and the target language is a plaintext monolingual corpus, i.e. any text written in that language (even a short one). Arguably, this is the lowest possible requirement to perform any text-based processing of a language: at least a textual input must be available, otherwise there is nothing to process. The key assumption behind our approach is that corresponding words often have the following two properties: • They are similar on the character level, i.e. the string similarity of the source word and the corresponding target word is often high. • They appear in the language with a similar frequency, i.e. the frequency of the source word in a source language corpus and the frequency of the target word in a target language corpus is usually similar.
While these assumptions obviously do not hold in general, we believe that they are mostly valid in case of very close languages (such as Czech and Slovak or Danish and Norwegian, which we use in our evaluation).
Our general approach to translating a given source language word is to look through all of the target words present in our corpus, and to return the most similar one 2This can also be understood as a very rough way of acknowledging interlingual word similarity, in the sense that it is implicitly assumed that the unknown source word may happen to be also used in the target language in an identical form. This assumption may often be true, especially in case of named entities, which constitute a major share of OOVs. Still, even in such cases, transliteration or similar transformations may be necessary to obtain the correct target word form. as the most probable translation, based on the two dimensions of similarity noted above and described in detail in Section 3.
In practice, an exhaustive search over the full target vocabulary is not viable. Therefore, we introduce a number of heuristics to speed up the search, and describe the whole translation process, in Section 4.
In Section 5.1, we evaluate MonoTrans instrinsically
with BLEU [
The quality of MonoTrans translation is too low to be useful when targeting humans; moreover, for a speaker of the target language, a similar close language is typically partially intelligible even without translation. In fact, it is exactly this partial cross-lingual intelligibility of similar languages, common with humans but generally inaccessible to machines, that we want to simulate with MonoTrans. We focus on the task of cross-lingual transfer of trained NLP tools, namely part-of-speech (POS) taggers and dependency parsers, where even a low-quality translation can provide the tools with a partial understanding of an unknown language and allow them to be applied to that language, even if their performance will inevitably be low. We evaluate MonoTrans extrinsically in this setup in Section 5.2. 2
While being a stranger in data-driven machine translation,
interlingual word similarity has often been utilized in
rulebased MT, in particular when translating between similar
languages. While rule-based MT has generally been
superseded by statistical MT, a number of fully rule-based
or hybrid3 machine translation systems do exist, such as
Apertium [
The somewhat solitary work of Irvine and
CallisonBurch [
There is also a handful of older work attempting to
construct a bilingual lexicon and/or to perform machine
translation without parallel corpora, most notably by Koehn and
Knight [
The key component of MonoTrans is a word similarity measure, composed of a string similarity simstr and a frequency similarity sim f ; the string similarity is itself composed of a Jaro-Winkler-based similarity sim jw∗ and a length-based similarity siml :
sim(wsrc, wtgt ) = simstr(wsrc, wtgt ) · sim f (wsrc, wtgt ) simstr(wsrc, wtgt ) = sim jw∗(wsrc, wtgt ) · siml (wsrc, wtgt ) (1) where wsrc and wtgt are the source and target word, respectively. The following subsections provide detailed descriptions of each of these components.
Our string similarity measure is based on the
JaroWinkler (JW) similarity [
However, JW similarity does not account for a number phenomena that are common in interlingual word similarity. We believe the following two to be of the highest importance:
4Interestingly, the authors seem to use a measure of frequency similarity very close to ours, although the provided formula (4) seems to be inverted by mistake, measuring frequency dissimilarity instead.
5From another perspective, we could say that JW similarity implicitly performs a simple soft stemming of its arguments. simJ(wsrc, wtgt )
sim0f (wsrc, wtgt ) = pair of words is often too high. Therefore, we introduce an additional penalty for words that differ in length: sim0l (wsrc, wtgt ) = where len(w) is the length of word w. The length importance L is used to put less weight on length similarity than on the other string similarity components; we use L = 0.2.
The length similarity is computed both on the original words as well as on their devowelled variants: siml = sim0l (wsrc, wtgt ) · sim0l (D(wsrc), D(wtgt )) (6) 3.3
We expect corresponding words to appear with a similar frequency in the monolingual corpora. Of course, one of them may be several times more frequent than the other, but their frequencies should be similar at least in orders of magnitude. Thus, we compare the logarithms of the frequencies to calculate the similarity: 1 1 + |log( fwsrc ) − log( fwtgt )| • diacritical marks tend to be cross-lingually inconsistent, and languages are usually intelligible even when diacritics are stripped from the text, • consonants tend to carry more meaning than vowels and tend to be more cross-lingually consistent.
Therefore, we introduce two preprocessing steps that can be employed to simplify the word forms before the computation of the JW similarity: transliteration to ASCII, and devowelling.
unidecode Python module,6 maps all characters into ASCII, trying to replace each non-ASCII character by one “near what a human with a US keyboard would choose”. However, it does not handle non-alphabetic scripts, such as Chinese or Japanese. We denote transliteration of a word w by T (w).
The devowelling strips all vowel characters, i.e. all characters that, after transliteration to ASCII, belong to the following group: a, e, i, o, u, y. We denote devowelling of a word w by D(w).
The JW-based similarity sim jw∗ is then computed as a multiplication of several components: sim jw∗(wsrc, wtgt ) =
∏ J∈{ jw, jwT, jwD, jwDT } (2)
The first component,sim jw, is the JW similarity without any preprocessing. However, as it is undefined for empty words (ε), we modify it slightly: sim jw(wsrc, wtgt ) = 1+len1(wsrc) if wtgt = ε 1
1+len(wtgt ) sim0jw(wsrc, wtgt ) otherwise if wsrc = ε (3) where len(w) is the number of characters in word w, and sim0jw is the Jaro-Winkler similarity provided by the pyjarowinkler Python module.78
The following components are the JW similarity of transliterated words (sim jwT ), the JW similarity of devowelled words (sim jwD), and the JW similarity of transliterated and devowelled words (sim jwDT ): sim jwT (wsrc, wtgt ) = sim jw(T (wsrc), T (wtgt )) sim jwD(wsrc, wtgt ) = sim jw(D(wsrc), D(wtgt )) sim jwDT (wsrc, wtgt ) = sim jw(D(T (wsrc)), D(T (wtgt ))) (4) 3.2
A target word that is significantly shorter or longer than a given source word is unlikely to be its translation; however, we found that the Jaro-Winkler similarity for such a
7https://pypi.python.org/pypi/pyjarowinkler 8Interestingly, the module method get_jaro_distance does not provide the Jaro-Winkler distance d jw, but the Jaro-Winkler similarity 1 − d jw; i.e., a value of 1 corresponds to identical strings, and the value of 0 to completely dissimilar strings.
The occurrence frequencies are computed from the monolingual corpora: fwcorpus = countcorpus(w) + S sizecorpus using a smoothing factor S that allows us to output, with a low probability, even unknown words; we use S = 0.1.
Such a measure seems to be appropriate for corpora of similar sizes. However, if one of the corpora is significantly smaller than the other, the frequencies of words in the smaller corpus are somewhat boosted due to the smaller number of word types appearing in the corpus among which the total mass is distributed. Therefore, we downscale the frequencies computed on the smaller corpus by upscaling its size used in (8): sizeA = (|A| · q ||BA|| if |A| < |B| |A| where |X | denotes the number of words in the corpus X .
We found that the definition of frequency similarity in (7) does a good job in removing many bad target language candidates; usually these are very infrequent words that are by chance string-wise similar to the source word. However, we also found that it inappropriately boosts target words with a low similarity to the source word that by chance have an extremely similar frequency. Thus, we need to keep the similarity harsh for low values, but soften it for high values. Therefore, if the value of sim0f is higher than a threshold Tf , we push the part of it which is above (7) (8) (9) the threshold down, by multiplying it by a decay factor D f : sim f =
sim0f We use Tf = 0.5 and D f = 0.1. 3.4
(Tf + D f · (sim0f − Tf ) if sim0f > Tf While the similarity measure we use is intended to be language-independent, we do acknowledge that it was hand-tuned particularly on the Czech-Slovak language pair, as the authors have a strong knowledge of both of these languages, and may not be fully adequate for all language pairs. In particular, we expect it to work best for flective languages with a preference for word-final inflection. For an optimal performance, it should be hand-tuned or machine-tuned on a more diverse set of languages.
The transliteration component is useless for languages that only use ASCII characters. Moreover, due to its implementation, it cannot handle non-alphabetic languages, such as Chinese or Japanese.
Also, we fail to identify systematic differences in the languages, such as “w” in Polish consistently corresponding to “v” in Czech or Slovak. We believe that the method would highly benefit from being able to find such correspondences automatically in the monolingual data, e.g. by exploring the distributions of character unigrams or ngrams, and/or by employing an EM-like approach to find a likely mapping.
Finally, the measure completely ignores the fact that words with similar meanings can be expected to appear in similar contexts. While reflecting on that fact would most probably make the computation of the similarity much more complicated and slower, it could allow the method to detect even corresponding words that are dissimilar according to the string similarity measures. A more viable approach could be to at least account for the fact that a given target word is likely to appear in similar target contexts, as mediated e.g. by a language model. 4
The MonoTrans translation system consists of two components: a training component, and a translation component.
The training component creates a pair of word frequency lists, based on source and target monolingual corpora. Any monolingual corpora can be used for the training, with larger corpora leading to better results. The frequency similarity measure works more reliably when the source and target corpora are of similar sizes, at least in orders of magnitude. However, if the source language is very low on available resources, the input text to be translated can by itself also serve as the only source corpus.
The translation component then performs a wordbased 1:1 monotone translation, trying to translate each source word by the most similar target word from the target word list, based on the similarity measure described in the previous section. The translation of each word is performed independently. 4.1
In theory, for each input source word, the translation component could always go through all target words in the target language word list, measure the similarity of the source word and each candidate target word, and then emit the most similar target word as the translation.
However, this is only feasible in cases where the target word list is very small, containing hundreds or at most thousands of words, allowing us to translate each source word in a matter of seconds at most. Once the target word list goes into tens of thousands of words and beyond (which it usually does in our experiments), the translation times become far too long for an exhaustive search to be practical.
Therefore, we introduce a range of heuristics and technical measures, both lossless and lossy, to sufficiently speed up the translation process while trying to keep the translation quality as high as possible. We describe the most important two of them in the following subsections; we also use other less interesting measures, such as caching of method calls. 4.2
The main speedup comes from a hard partitioning of the word lists, which is the only lossy procedure we employ. Following our observations in Section 3, we assume that for a pair of corresponding source and target words: • the lengths of the devowelled transliterated words differ by at most 1, • the first two characters of the devowelled transliterated words are identical.
None of these assumptions hold universally, but we believe that they do hold for a vast majority of words that can be translated by our system (i.e. words that are sufficiently similar to their target counterparts). Most importantly, they let us only go through a tiny part of the target word list when translating a source word, bringing a key speedup to the translation system.
Thus, instead of using a flat word list, the training component stores each word in a specific partition, addressed by a compound key, consisting of the first two transliterated consonants of the word and the length of the transliterated devowelled word. The translation component then only traverses three of these partitions, corresponding to the first two transliterated consonants of the source word and the length of the transliterated devowelled source word, increased by +1, 0, and -1. Even after the partitioning, many of the partitions are too large to be traversed exhaustively for each matching source word. We can deal with that issue thanks to the following two observations: • word frequency similarity is a powerful component of our similarity measure, • words in a language have a Zipf-like distribution, with a small number of frequent words and a high number of rare words.
Therefore, we can achieve a significant speedup by ordering the words in each partition descendingly by frequency, and introducing the following early-stopping criterion: once we reach a target word so infrequent that its frequency-based similarity to the source word alone is lower than the total similarity of the most similar target word found so far, we can stop processing the current partition, as none of the remaining target words would be able to surpass the currently best candidate; i.e., we stop once: fwt0gt < fwsrc ∧ sim f (wsrc, wt0gt ) < sim(wsrc, wt∗gt ) (11) where wt0gt is the current target word candidate, and wt∗gt is the best target word found so far.
As we, by definition, encounter frequent words more frequently than rare words, this allows us to often skip the processing of the long tail of infrequent words; it only gets processed if the source word is a rare one, or if it is not sufficiently similar to any of the frequent target words. 5
In this section, we evaluate MonoTrans both intrinsically and extrinsically. However, for the real under-resourced languages with no parallel data and no annotated corpora available, there is no way for us to perform an automatic evaluation, and a manual evaluation would be difficult to obtain for us. Therefore, as is usual in these scenarios, we simulate the under-resourced setting by evaluating on pairs of similar but resource-rich languages, allowing us to use standard automatic evaluation measures. Specifically, we use the following language groups in our experiments: • Czech (cs) and Slovak (sk), • Danish (da), Norwegian (no) and Swedish (sv), • Catalan (ca) and Spanish (es).
Please note that we hand-tuned our method partially based on brief manual inspections of the results on the sk-cs pair.
We first evaluate the quality of the MonoTrans translation
itself with BLEU [
use the first 10,000 target sentences and the last 10,000 source sentences for training, and then evaluate the sourceto-target translation quality on the last 10,000 sentences; i.e. the source side of the evaluation data is used for training, but the target side, which serves as the reference translation, is not.
Table 1 shows the BLEU scores achieved by MonoTrans, compared to the SrcLex baseline, i.e. to performing no translation at all; thanks to the high similarity of the languages, even the baseline achieves a non-trivial translation score. The BLEU scores are rather low, reaching 13.4 on average, whereas a state-of-the-art MT system trained on large amounts of parallel data could easily reach scores around 30 BLEU points (or probably even more, provided that the source and target languages are very similar). However, we can see a large and consistent improvement over the baseline of 3.4 BLEU points in average. We also report the relative improvement over the baseline which, thanks to the very low scores achieved by the baseline, is very high, reaching up to 90% (for sv-no) and 43% on average.
Generally, we do not expect very large corpora to be
available for under-resourced languages. Still, to
measure the scaling potential of our method, we also evaluated
MonoTrans trained on significantly larger Czech and
Slovak monolingual corpora. For this experiment, we used
large web corpora, namely CWC for Czech [
The results in Table 2 show that increasing the data size improves the translation quality, with the improvement over the SrcLex baseline being nearly doubled. However, considering the factor by which we increased the training data size, we find the improvement to be rather moderate.
Next, we tried to downscale the training data instead, training and evaluating the system identically to Table 1 but using only a part of the training data. Table 3 shows that already with 100 monolingual non-corresponding sentences for each of the languages, i.e. very modest data, improvements of over +1 BLEU over the non-translation baseline can be achieved.
Finally, in Table 4, we show several examples of the inputs and outputs of the MonoTrans system, taken from the evaluation datasets translated by the systems trained on the large datasets; the “correct” translation is not the reference translation, but a corrected version of the MonoTrans output, and errors are highlighted. We can see many correctly translated words in the outputs, as well as many words correctly left untranslated. Moreover, many of the errors can be easily accounted to the word list partitioning that we employ, making it impossible for MonoTrans to perform translations such as “len–jen”, “som–jsem”, or “po cˇúvaj– poslouchej”. This suggests that many of the errors are actually search errors, not scoring errors, and could be eliminated if we had a better way of efficiently searching for candidate target translations. 5.2
We also evaluated MonoTrans extrinsically, in the task
of cross-lingual transfer of trained NLP models across
closely related languages, inspired by the cross-lingual
parsing shared task of the VarDial 2017 workshop [
Specifically, we employ the Universal Dependencies (UD)
v1.4 treebanks [
The MonoTrans system is trained using the word forms from the training part of the source treebank and the development part of the target treebank, and applied to translate the training part of the source treebank into the target language. Then, the UDPipe tagger and parser are trained on the resulting corpus; the tagger is trained to predict the Universal POS tag (UPOS) based on the word form, and the parser is trained to predict the labelled dependency tree based on the word form and the UPOS tag predicted by the tagger. Finally, both the tagger and the parser are applied to the development part of the target language treebank, and evaluated against its gold-standard annotation.
We report the tagger accuracy in Table 5, and the parser LAS10 in Table 6. As a baseline, we also include SrcLex, i.e. using a tagger and parser trained on an untranslated source treebank, and as an upper bound, we include a supervised tagger and parser, trained on the training part of the target treebank; as the languages are very similar, the baselines are quite strong. This allows us to also compute the error reduction, i.e. the proportion of the gap between the baseline and the upper bound filled by our method.
The taggers reach an average accuracy of 77.8% and the parsers an average LAS of 47.3%, which is not much in absolute terms – when large parallel data are available, LAS scores around 60% can be reached. However, in relative terms, the scores are rather impressive, obtaining an average 23% error reduction in both the tagging and the parsing, and reaching average absolute improvements of +6.3 in tagging accuracy and +8.2 in parsing LAS. Given our setting, we find the results to be wonderful, provided that we only used small monolingual corpora to train the MT system; in fact, in the target language, we only used the evaluation input data, which is probably the lowest imaginable data requirement. 6 6.1
As we mentioned in Section 3.4, a clear shortcoming of our method is the fact that the translation is performed in a context-independent way. Employing an n-gram language model is a standard way of getting better machine translation outputs, only plaintext target-language data are needed to create one, and there already exists a plethora of state-of-the-art ready-to-use language modelling tools. Therefore, it may seem straightforward to employ a language model in MonoTrans as well.
However, there is a range of technical issues that need to be overcome. If we were able to generate a translation lexicon, we could then even easily plug it into a full-fledged MT system, such as Moses, easily combining it with a language model; however, generating the lexicon would be computationally prohibitive in our case, for reasons mentioned in Section 4.1. At best, we could potentially try to generate a translation lexicon only for the words that appear in the test data. Moreover, even using a beam search in MonoTrans decoding is too costly for us, as it prohibits the employment of the early stopping mechanisms.
So far, we have only managed to perform a set of preliminary experiments, adding a simple trigram language model and using its score as an additional scoring component; as we found that using the score directly had a too strong and negative effect on the translations, we weakend it by taking its fourth root. When evaluated on the large Czech and Slovak corpora in both directions, we observed 10Labelled Attachment Score, i.e. the number of correctly predicted labelled dependency relations in the output tree. only negligible improvements around +0.1 BLEU. We believe that this is mainly due to the fact that our approach in these preliminary experiments was too rough and simplistic, and that with proper tuning and a more sophisticated implementation, clear improvements may be gained. Based on inspection of the translation outputs, as well as from the examples in Table 4, it is clear that the word list partitioning is way too crude, preventing the system from generating the correct translation in many cases, even though its similarity to the source word is sufficiently high. On the other hand, it is completely impossible for the system to search through all possible translations, and some kind of harsh pruning of the search space is vital.
As a quick remedy, we tried to use the trigram language model to generate additional translation candidates. Specifically, for each source word we also investigated N candidate translations taken from N words that are, according to the language model, the most likely to follow the words selected as translations of the previous words. With N = 20, we observed a promising improvement of +0.6 BLEU for cs-sk, while the translation times remained competitive (they doubled). With N = 1000, the improvement for sk-cs further jumped to +1.3 BLEU; however, at this point, the translation became about 50 times slower (taking 10 hours to translate 10,000 sentences), showing that this approach is somewhat promising in terms of translation quality but too computationally demanding. For skcs, negligible or no improvements were observed.
An interesting possibility of clustering the search space which was suggested to us is to use a standard clustering algorithm, such as k-means or hierarchical k-means, with the word similarity used as the distance of the target words. This is expected to be permissibly fast to compute as well as to allow a sufficiently fast search for translation candidates; however, due to time constraints, we have not been able to design an experiment to test that. 7
We presented MonoTrans, a data-driven translation system trained only on plaintext monolingual corpora, intended for low-quality machine translation between very similar languages in a low-resource scenario.
We showed that even with very small training corpora available, the system shows respectable performance according to both intrinsic and extrinsic evaluation, consistently surpassing the no-translation baseline by large margins. Moreover, we showed that the system performance scales with larger training data, even though rather slowly.
In particular, when evaluated extrinsically as a component of cross-lingual tagger and parser transfer, employing MonoTrans leads to high improvements in both tagging accuracy and parser LAS with respect to the baselines, achieving an average 23% error reduction in both of the tasks when supervised models are taken as the upper bounds.
This work has been supported by the grant No. DG16P02B048 of the Ministry of Culture of the Czech Republic, the grant No. CZ.02.1.01/0.0/0.0/16_013/0001781 of the Ministry of Education, Youth and Sports of the Czech Republic, and the SVV 260 453 grant. This work has been using language resources and tools developed, stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (project LM2015071). We would also like to thank the anonymous reviewers and our colleagues from the ÚFAL MT group (especially Jindrˇich Libovický) for helpful comments and suggestions for improvement.