This paper describes our contribution to the Shared Task on Translation Inference across Dictionaries (TIAD-2019). In our approach, we construct a multi-lingual word embedding space by projecting new languages in the feature space of a language for which a pretrained embedding model exists. We use the similarity of the word embeddings to predict candidate translations. Even if our projection methodology is rather simplistic, our system outperforms the other participating systems with respect to the F1 measure for the language pairs which we predicted.
1 Note that this technical de nition is broader than the de nition typically applied in linguistics: In a strict sense, cognates are only those of the rst case. However, for languages with a long and intense history of contact, the di erentiation between cognates in a strict sense and mutual loans is not always clear-cut. A typical case is the large number of common roots in Turkic and Mongolian which can be variously attributed to either a loan between (proto-) languages or to their common (but hypothetical) ancestral language. In both situations, however, the linguistic characteristics (systematic phonological similarity and semantic relatedness) are comparable, thus motivating a generalized, technical de nition of cognates. (in one sense of the German word) are semantically identical, and they re ect the same sound correspondence of k and ch that we also nd in word pairs such as Kinn/chin. An example for the second case is English bank (as in river bank) and German -bank in compounds such as Sandbank `sandbank'. Again, senses are similar (an accumulation of soil in or at a water stream), but here, the English word is a loan (either from Low German or Scandinavian). For Bank and bank as ` nancial institute', the situation is more complex: In both German and English, this is a loan from Italian banca, but the origin of this word is a Germanic (Langobardian) word with the meaning `bench', i.e., the place where clients had to wait in order to receive nancial support. Cognate candidates can be identi ed by means of phonological and semantic similarity metrics, and the latter are the basis for the implementation that we describe with this paper. Our research has been conducted in the context of the Independent Research Group `Linked Open Dictionaries' (LiODi), funded by the German Federal Ministry of Education and Science (BMBF), and applied to language contact studies in the Caucasus area.
Out of the context of this project, one aspect that we appreciate in the TIAD shared task is that it does not depend nor require the use of parallel corpus data. This is in fact the situation for most languages studied in our project. For many low-resource languages, the only language data that is available is provided in the form of dictionaries (or plain word lists) on the one hand and in the form of monolingual, non-translated text on the other, often in the form of eld recordings. Parallel (translated) data, however, is scarce { if available at all (and if so, often limited to religious texts, e.g., translations of the Bible with limited representativeness for the language as a whole and not necessarily written by a native speaker). The technological challenges of the TIAD task thus closely mirror the technical (data) challenges in the LiODi project, except that it is limited for translation equivalence whereas we aim to include less direct semantic relations, as well.
In our approach, we make use of the similarity of cross-lingual word
embeddings { genuine and projected ones { in order to build new bilingual dictionaries.
The projection of words of one language into the embedding space of another and
thus the generation of a multi-lingual word embedding space is a widely discussed
topic. For an overview of di erent methodologies of cross-lingual word
embedding generation, we refer to [
Despite its simplicity, our implementation achieves good results in
comparison to the other systems submitted to TIAD. When similarity beyond translation
equivalence is excluded (as by applying a selection threshold), our system is
basically a neural reconstruction of [
We use the TSV edition of the dictionaries provided by the task organizers. Whereas we only use the languages and language pairs provided in these dictionaries, it would be possible to add more language pairs to be processed by our approach, as long as they are provided in the TIAD-TSV format. In the context of the H2020 project `Pret-a-LLOD. Ready-to-use Multilingual Linked Language Data for Knowledge Services across Sectors', we are currently in the process of creating such translation data, with OntoLex-Lemon and TIAD-TSV data for 130 language varieties and 373 language pairs already being available from http://github.com/acoli-repo/acoli-dicts/. For the TIAD-2019 task, selected parts of this data have been taken into consideration, however, only for estimating prediction thresholds (Sect. 4.1), not as a source of external lexicographical information.
For initializing the embedding space, we employ readily available monolingual
embeddings from a particular `reference language'. Di erent reference languages
would be feasible in this context, and many di erent types of embeddings are
available for each. Here, we build on the 50 dimensional GloVe v.1.2
embeddings for English, trained on Wikipedia and Gigaword [
In order to make our approach computationally more e cient, we prune the embedding space before running our experiments from words not occurring in the (English) dictionary data we are working with. We add multi-word expressions consisting of words present in the model by taking the sum of their representations. As we want to project other languages into the semantic space of our reference language, we need to store the information to which language a word of the model belongs to. We solve this by adding a language tag to the word:
2 Download link: http://nlp.stanford.edu/data/glove.6B.zip
We developed a baseline approach, and submitted the results of this approach to the Shared Task organizers. Later on, we elaborated our baseline and compared its performance to our baseline's performance by applying an interal evaluation procedure (4.3). The results of our elaboration were not submitted to the Shared Task organizers due to time constraints. 3.1
Using the pretrained embedding model for our reference language, we generate word embeddings for all the other languages of the provided dictionaries. We project the words of these languages into the embedding space of the reference language, e.g. for the Basque word abentura and its English translations adventure and venture (according to the Basque-English Apertium dictionary), we calculate the new embedding by summing up the embeddings of the possible translations into English:
In a rst iteration, we thus project all those languages into the embedding space of our reference language English for which a dictionary with English translations exist (here: EU ! EN, EO ! EN).3 In order to improve coverage, we also included the inverse direction (EN ! CA, EN ! ES, EN ! GL). We refer to these languages (CA, EU, ES, EO, GL) as ` rst-level' languages. In the second (and subsequent) iterations, we then project all the languages into the enriched embedding space for which at least one dictionary exists that connects it with another language in the embedding space. If more than one dictionary does exist (e.g., for French EO ! FR, FR ! ES, FR ! CAT), we sum over all translations in the embedding space. AN, AST, FR and IT are the `second-level languages' for which embeddings are created in this way. Note that once embeddings for a particular language are created, they are not updated nor extended any more, even if in a further iteration additional dictionaries also containing this language are encountered. For example, we obtain EO embeddings from EO ! EN and nothing else, even if after adding CA to the embedding space, EO could be reached via EN ! CA ! EO, as well. This also means that lexical material for Esparanto provided by the CA ! EO dictionary which is not covered in the EO 3 In the following, we refer to the languages solely by using the language abbreviations used in the Shared Task description (https://tiad2019.unizar.es/task.html) ! EN dictionary will not be represented in the embedding space. Again, this is an implementation decision taken in the interest of time and to be reconsidered in subsequent research. With a larger set of dictionaries and language pairs, we would continue these iterations until no more languages can be reached in the language graph represented by the dictionaries.
In order to generate a new bilingual dictionary A ! B, we use this enriched, multilingual word embeddings and predict candidate translations of a word a in language A by choosing the words in language B whose embeddings are most similar to the embedding of a, and which have the same part-of-speech as a. As similarity measure we chose cosine similarity, which is ignorant of vector length, and faster in lookup than Euclidian distance. This also allows us to add rather than average in the process of projecting new language into the embedding space. It should be noted that speed is a decisive criterion in the intended application of the technique, as we aim to provide on-the- y cognate detection. 3.2
We elaborated our baseline approach by taking into consideration the sense(s) of a word to be translated. We follow the de nition of senses in the TIAD-OntoLex edition of the Apertium dictionaries by assuming that in a bilingual dictionary, each translation b1; b2; ::: provided for a source language word a entails a speci c sense ab1 ; ab2 ; :::. The number of OntoLex senses in the TIAD data is thus identical to the number of translations provided. Unfortunately, the Apertium data does not provide sense-level linking beyond di erent bilingual dictionaries, so that sense-level linking cannot be used as a feature for translation inference. We do not have information on whether sense ab1 induced from the A ! B dictionary has any relation with the sense ac15 induced from a A ! C dictionary.
However, if the basic assumption is correct, then we can approximate the factual number of senses of a source language word a as the maximum number n of translations in any bilingual dictionary. Assuming that translations referring to the same word sense tend to be closer in the embedding space, we can use this information for disambiguation and try to identify the most representative translation for every sense. We do so by performing a lookup in the dictionaries to get all translations, and then cluster (the embeddings of) these translations into n clusters. For every cluster, we then return the target language word(s) closest to the cluster center.
We generate \sense embeddings" for a as follows:
(a; b11); :::; (a; b1n); (a; b21); :::; (a; b2m), etc., where bij : the j'th translation of a in a dictionary A ! Bi (or Bi ! A) N : number of dictionaries where A appears as source or target language; i = 1; :::; N . 2. Apply k-means clustering4 over all possible translations bij , where k = maxiJi, where Ji: number of entries containing a in A ! Bi. 3. For each of the k cluster centers, take the closest embedding (irrespective of its language) as a sense embedding of word a. This is the basis for nding the closest target language embedding, resp., word, per sense cluster. We then predict possible translations of a given word in a source language into a speci ed target language by looking up the most similar word embeddings of the target language to each of the sense embeddings of the word in the source language.
While the baseline predicted n words in the target language as candidate translations, we now predict at least n=k words for each of the k sense embeddings, with distance relative to the `sense cluster' rather than the source language embedding. 4
We rst ran an internal evaluation in order to get an idea of the quality of our approach. After that, we determine a cosine similarity threshold for our predicted translations in order to select the best translation candidates. Later, we use the internal evaluation strategy again to compare our baseline with the elaboration. Unlike most other systems in TIAD-2019, we provide only the data points exceeding our thresholds, whereas a lower threshold (and parameter estimation by the task organizers) may have produced better results on the blind test set. 4.1
Our baseline approach returns the n most similar word embeddings of the target
language to the word embedding to be translated, but obviously, not all of these
might be good translation candidates. In order to lter out good candidates,
we determine a threshold for the cosine similarity value. Therefore, we take the
EN ! PT, PT ! FR, FR ! EN dictionaries available from FreeDict,5 using
the TIAD-TSV version we provide under
https://github.com/acoli-repo/acolidicts. If both translation directions are available for a language pair, we join the
4 We use the scikit-learn implementation of k-means clustering
(https://scikitlearn.org/stable/modules/generated/sklearn.cluster.KMeans.html) with default
parameters.
5 http://www.freedict.de/
two dictionaries to obtain a higher coverage. As expected, the results in table
1 show decreasing cosine similarity decrease is correlated with lower precision
and higher recall. We chose the cosine similarity threshold 0:97, a rather high
threshold in order to achieve su cient precision, without su ering too much from
the drop in recall in terms of F1. Yet, considering recall here is debatable: It is
not as expressive in this scenario as it requires a full coverage of the languages
by the dictionary, which is not the case for dictionaries in general (see also the
discussion in the TIAD-2017 overview report [
The \One Time Inverse Consultation" (OTIC) method [
In our approach, this resembles the way how multilingual embeddings are constructed: A non-English word is represented as the sum (of the embeddings) of the (directly or indirectly associated) English words. If two words are identical in their English translations, they thus have cosine similarity 1, every English translation they do not share leads (when added to the embedding) to a slight decrease in this score. In our approach, this decrease will be smaller for semantically related words and larger for non-related words, but if we assume that, on average, the deviation from a perfect match per non-shared English translation is equal, the decrease in cosine similarity will directly re ect the number of non-shared English translations relative to the number of shared English translations. We suspect that the observed drop in F1 in comparison to OTIC is due to the construction method of the multilingual embedding space, where dictionaries connecting two rst-, resp., two second-level languages are being ignored, as well as due basing this on English as the only pivot language, whereas OTIC can adopt di erent pivots depending on the structure of the dictionary graph.
In the light of this explanation, it is somewhat surprising to see that the evaluation results of the Shared Task organizers seem to con rm the trend that 2nd level languages predict better translation candidates, as previously noticed in Sect. 4.3: For PT ! FR, we get the highest F1 value, even if for both of these languages, the embeddings are produced in the second iteration of our algorithm.
Overall, the scores achieved by all systems (as well as the baselines) in the TIAD-2019 shared task were considerably low, possibly indicating conceptual di erences in the composition of Apertium data and the blind test set. We thus performed an additional evaluation on the Apertium data itself. The senseclustering extension was evaluated only in this internal evaluation.
To assess the quality of a newly generated bilingual dictionary by our approach, we leave out one of the provided dictionaries when calculating the embeddings for the languages based on the reference language model and try to reconstruct it using a similarity threshold of 0.97 for predicting translation pairs. Only in this evaluation, we compare our base implementation with the sense-clustering extensions, see Tab. 3.
For more than two thirds of the language pairs (72%, 13/18), the sense clustering yields a slight improvement in F1, but only at a marginal scale. In general, precision and recall vary substantially. We obtain the highest precision for EU ! EN in both the baseline and extension. The fact that high-precision language pairs show a (small) drop in precision in the sense clustering extension is possibly related to the fact that most of them involve the reference language. Dictionaries including OC have comparably high recall, PT ! CA and FR ! CA also obtain high recall. The highest precision values (over 69%) are obtained by dictionaries including EN for the baseline, precision drops for these dictionaries in the sense extension, while recall increases in most of them.
The fact that embeddings for a language are not trained on their own data as it is the case for the reference language might have a small e ect on precision, but the F1 values of language pairs that do include the reference language English are not in general higher than of pairs that do not. Our \2nd level languages" AN, AST, FR, IT, OC, PT and RO are not added in the rst iteration like CA, EO, ES, EU and GL (\1st level languages") and have to be generated using already projected word embeddings instead of those of the reference language. The number of pivots used in the projection procedure does not seem to be negatively correlated with the quality of the generated dictionaries though, at least for those languages we could evaluate in our internal evaluation (we could not include AN, AST, IT, and RO as they are connected to only one other language, resp., and thus, as there is no alternative way of getting their embeddings with our projection procedure if their dictionaries are excluded from the translation graph): The highest F1 value is obtained by predicting CA and ES, both 1st level languages, from OC, a 2nd level language, whereas the lowest F1 value is obtained by predicting FR (2nd level language) from EO (1st level language). In general, predicting dictionaries with EO as input yields low F1 values, and all except one (FR ! ES) predicted dictionary from 2nd level languages yield higher recall values than most of the other predicted dictionaries. 5
As the results of our internal evaluation (and later on also of the submission) show, the fact that a word embedding was projected and the number of pivot languages necessary for the projection do not seem to worsen the quality of the generation of translation pairs. We have therefore concluded that our approach is a viable way for generating new translation pairs. We might, however, expect better results when using more sophisticated projection methodologies, for example by learning projection matrices jointly for the languages to be projected into the English embedding space - this remains to be tested. A critical point of our algorithm is to generate a multi-lingual embedding space in a greedy fashion, other languages are projected into the space of the reference language: In each iteration, we only consider the dictionaries whose target language is in the language set of already added languages plus the reference language, e.g. we obtain EO from EO ! EN and nothing else. We did not exploit yet any data provided by the dictionaries EO ! CA, EO ! ES and EO ! FR; the integration of these dictionaries and therefore, the usage of all available resources would be a more comprehensive approach and probably improve the quality of our generation of candidate translation pairs. Choosing an alternative pretrained model for the reference language with a higher-dimensional embedding space would be another way to improve { GloVe embeddings are available in higher dimensions as well, and likewise, alternative embeddings could be explored. Furthermore, one could use more than just one reference language, and create a joint embedding space by means of either concatenation (e.g., of English-based and Spanish-based embeddings for each word) or by training a joint embedding space as mentioned above.
The sense clustering we used in the elaboration of our baseline approach fails to yield signi cant improvements. One possible explanation for this is that we have too little input for clustering. Another cause might be that senses are represented by written representation pairs as described in Sect. 3.2. This results in more emphasis on certain \senses" (pairs of written representations) when predicting translations. In general, the remarkably poor performance of all submitted systems (and the provided baseline implementations) on the blind test set may re ect the possibly quite distinct nature of training and blind test data: The blind test data originates from proprietary learner dictionaries provided by KDictionaries, whereas the Apertium dictionaries that constitute the training data have been designed for machine translation. Both are similar on a super cial level, but it is not unlikely that learner's dictionaries put a stronger emphasis on covering the di erent semantic senses in their translations in a near-exhaustive way (regardless of their real-world frequency), whereas MT dictionaries do have a preference for covering prototypical senses (being ignorant against infrequent/domain-speci c uses may actually improve the system for most of its applications). In order to quantify the impact of sense granularity independently from proprietary data, we would encourage the task organizers to provide an provide open test set along with the blind test set.
As mentioned above, we developed our system originally in the context of our research project for the purpose of facilitating the search for semantically similar words. Vector representations of words in a semantic space are particularly promising when searching for remote semantic links, e.g., as required for cognate detection, but maybe less so for nding literal translations. Yet, as the evaluation indicates, our system also produces acceptable results for the task of translation inference across dictionaries in the sense that we rank among the best-performing implementations for this task.
Our contribution and the key bene t of our approach and is to be seen in the fact that it is slim, fast, and trivially extensible beyond literal translations: Translation inference (like cognate detection) requires a simple lookup in the embedding space. We see that as an important component of on-the- y cognate detection, as it allows to identify semantically related form forms over a multilingual graph of dictionaries without actually traversing this graph at query time.
The research described in this paper was primarily conducted in the project `Linked Open Dictionaries' (LiODi, 2015-2020), funded by the German Ministry for Education and Research (BMBF) as an Independent Research Group on eHumanities. The conversion of FreeDict dictionaries into TIAD-TSV data that we used for estimating the prediction threshold was performed in the context of the Research and Innovation Action \Pret-a-LLOD. Ready-to-use Multilingual Linked Language Data for Knowledge Services across Sectors" funded in the European Union's Horizon 2020 research and innovation programme under grant agreement No 825182.