We describe four systems to generate automatically bilingual dictionaries based on existing ones: three transitive systems di ering only in the pivot language used, and a system based on a di erent approach which only needs monolingual corpora in both the source and target languages. All four methods make use of cross-lingual word embeddings trained on monolingual corpora, and then mapped into a shared vector space. Experimental results con rm that our strategy has a good coverage and recall, achieving a performance comparable to to the best submitted systems on the TIAD 2019 gold standard set among the teams participating at the TIAD shared task.
Most research and development in natural language processing (NLP) and text
mining was initially conducted for English. In the last decades, there was a surge
in the application of NLP to other European languages as well as to the
mostspoken languages worldwide (e.g., Chinese [
For this purpose, most methods rely on comparable corpora [
In [
The objective of TIAD shared task is to generate automatically bilingual dictionaries based on existing ones. In particular, participants were required to generate new translations automatically between English (EN), French (FR) and Portuguese (PT) based on known translations contained in the Apertium RDF graph4. As these three languages are not directly connected in this graph, no translations can be obtained directly among them. The use of other freely available sources of background knowledge to improve performance is allowed, as long as no direct translation among the target language pairs is applied.
We presented four similar systems to the TIAD 2019 shared task: three transitive systems di ering only in the pivot language used, and a system based on
a di erent approach which only needs monolingual corpora in both the source
and target languages. Every system makes use of cross-lingual word embeddings
trained on monolingual corpora, and then mapped into a shared vector space
using VecMap [
Data processing In order to obtain similar models, and also to cover a general vocabulary in each case, we decided to employ the di erent editions of Wikipedia as corpora for learning the word embeddings. For English, Portuguese, and Spanish, we used the Wikipedia dumps from January 2018, while for French and Catalan we have downloaded the January 2019 data.
Since we work with dictionary data, we have pre-processed each corpus to
obtain morphosyntactic information as well as to reduce the vocabulary size of
each model. Thus, we can obtain the PoS (Part of Speech) tag of each word
and avoid the extraction of in ected forms that do not appear in dictionaries.
This process was carried out using LinguaKit [
After that, we used these modi ed corpora to train distributional semantics
models using word2vec [
Once we built the monolingual models for each language, we mapped them
into shared bilingual spaces using VecMap [
Algorithm With a view to exploring solely the performance of cross-lingual word embeddings in this task, it must be noted that our approach only uses the rst and last columns of the input data (the source lemma and its PoS tag), so we do not utilize information such as the sense of each entry or its translation in other languages. Besides, the strategy was principally designed to obtain translations of single lexical words, so multiword expressions (specially compound proper nouns and non-compositional expressions such as idioms) are not well covered by the algorithm. These are, however, interesting cases for further research.
To translate between a source (src) and a target (tgt ) language, our algorithm uses a pivot language (pvt ) relying on the referred cross-lingual word embeddings and computing the similarity by means of the cosine distance between the two word vectors. We take the following steps: 1. For each single word (lemma PoSTag, except proper nouns) in the input dictionary (e.g., EN-GL), we rst check whether it appears in our source vocabulary with the same PoS tag. In this case, we select the two most similar entries with the same PoS tag in the pivot vocabulary. Then, for each of these words we get the two closest entries in the target model (also with the same PoS tag). If any of the words appears in the models with the same morphosyntactic category, the most similar entry (adjective, noun, verb, or adverb) is selected. At the end of this process, we have 0 to 4 candidate translations for each input word, together with a con dence value (the cosine distance between the words in the pvt and tgt models). If no translation is found, the algorithm applies a default rule which uses the input lemma as the translation with a 0 con dence value. 2. For single proper nouns (composed by just one word), we rst check whether they appear in the pivot language (both in upper-case and lower-case) with a similarity greater than 0.5. In this case, we use the same procedure between pvt and tgt, selecting the target entry with a con dence value of 1. If the input proper noun does not appear in the pivot or target languages, we simply select the most similar proper noun (from the closest 50 words). If no proper noun is found, we apply the default rule. 3. To translate multiword expressions (MWEs) we apply a basic approach which uses a list of MWEs in the target language extracted from the input dictionaries (mwe-list ). Thus, for each input MWE, we select its two longest words (src 1 and src 2 ), and applying a similar strategy to that of single words, we select two candidate translations for each one: w1 and w1b for src 1, and w2 and w2b for src 2. Then, using the mwe-list in the target language, we select the rst MWE which contain both w1 and w2. Otherwise, we look for expressions containing other combinations of the candidate translations. If no translation is found, we also apply the default rule.
The output of this process contains, for each input entry, 1 to 4 candidate translations with their con dence value. In a post-processing step we select the rst candidate as the target translation for lemma PoSTag pairs with only one sense in each input dictionary. However, as our method entirely relies on the input word and PoS tag (and not on its translation or its semantic information), it produces the same output for homonyms and the di erent senses of polysemous words. To alleviate this issue, the post-processing step respectively selects the third, second, and fourth candidates (if any) as the translations of other entries of the same word form.
Using di erent pivot languages, we applied this algorithm in four runs: { LyS: this run uses, for each translation direction, the third language of the shared task as pivot. Thus, for EN!FR, Portuguese was the pivot language, while for FR!PT and PT!EN, English and French were the pivot languages, respectively. { LyS-CA: in this system, Catalan was used as the pivot language for each translation. { LyS-ES: this approach is identical to the previous one, but with Spanish (instead of Catalan) as pivot. { LyS-DT: this last strategy does not use a pivot language, but only the source and target cross-lingual word embeddings models. As mentioned, and to follow the guidelines of the shared task, the mapping of these models was carried out without any bilingual information.
Except for the EN$FR translations (where we used the EN10 model), the full English model was used in all the other directions (see Table 1). 3
Evaluation of the results was carried out by the organisers against a goldstandard set built by extracting translations from manually compiled pairs of K Dictionaries5 (KD). As the coverage of KD is not the same as Apertium, the subset covered by Apertium was taken to build the gold standard, i.e., those KD translations for which the source and target terms are present in both Apertium RDF source and target lexicons. The gold standard set was not available to participants, so that we could not carry out a systematic error analysis which may be useful for further research.
Due to a misunderstanding, our team, like other participants, understood that evaluation would be performed in the direction EN!FR, FR!PT and
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.00.0
Coverage Precision F1 Recall
0.5 threshold 0.1 0.2 0.3 0.4 0.6 0.7 0.8 0.9 1.0 PT!EN. As a result, translations in the opposite direction were not sent for evaluation.
When compared to the other submissions of the shared task, the rst overview of the general results con rm that our approach has a good coverage and recall (LyS-DT achieved the best numbers in this respect), even if the precision is lower than that of the other participants. In this regard, it must be mentioned that we decided to submit every output of the systems (even those with 0 con dence), so that a better analysis of the results should take into account di erent con dence thresholds. Interestingly, no submission could beat the baselines proposed by the organisers. The results of our four runs had the same tendencies in each of the three translations, being LyS-DT the best system followed by LyS-ES, LyS-CA, and LyS, respectively. Thus, we can infer that using a pivot language worsens the translation performance. The di erences among each system, however, were not considerable, and there is no evidence that language family or corpus size have had a decisive impact, so that only a careful error analysis could shed some light into these results.
With regard to each speci c translation pair, LyS-DT was the best submitted system in the EN!FR direction (0:34 F1), and the second one in FR!PT (0:36 F1) and in PT!EN (0:27 F1), tied and below the Frankfurt team, respectively. In general, the results of our other systems were slightly below these values.
We carried out a brief analysis of the LyS-DT submission, aimed at knowing a little better the output of the system. First, we observed the impact of the con dence threshold in the nal results. Then, we analyzed the number of translations of our system, paying special attention to those entries for which our system were not designed: multiword expressions and proper nouns.
Figure 1 shows a graph for LyS-DT submission with the evolution of the four metrics used to measure the performance according to the threshold of the degree of con dence for translations. As can be seen, the results remain stable up to a value of 0.5. It is worth noting that the F1 corresponding to a threshold 0 is slightly lower than the F1 between 0.1 and 0.4, so we could have climbed to the rst position in the nal ranking of participants teams by simply considering as o cial the results corresponding to a threshold 0.1. From a threshold of 0.5 on there is a rise in accuracy at the cost of a drop in coverage and recall, with F1 also dropping since the increase in accuracy is not enough to compensate for the drop in recall. Maximum precision is obtained between thresholds 0.8 and 0.9, but, while for 0.8 the recall has only dropped by half the value for 0.5, from that point on it descends with a steeper slope.
25000 20000 s 15000 n o tirsan la 10000 t 5000
Figure 2 shows the number of translations produced by LyS-DT with each con dence value. As this gure reveals, most translations have con dence values between 0.6-0.9. Besides, it is striking that from a total of 191; 373 entries there were 4; 184 with con dence of 1, and 23; 902 with 0 (which basically means that no translation was found). Among the rst group, it is worth noting that all translations with con dence 1 were proper nouns, because they were found in both pivot and target languages with a high similarity value. However, it is likely that many of these equivalents are incorrect (see Figure 1), since our Wikipediabased monolingual models contain proper nouns in di erent languages.
With regard to the entries with 0 con dence, Table 2 contains the number of non translated elements discriminated by single or multiword expressions as well as by PoS tags. For single words, most cases were proper nouns, followed by common nouns, adverbs, adjectives and verbs. In this respect, it was expected that our approach could not nd suitable translations for many proper nouns. Also, several adverbs, adjectives, and nouns could also be wrongly lemmatized and subsequently not found in our models. However, a brief look at these data also revealed other issues: on one hand, the input dictionaries contain wrong lemma PoStag pairs (e.g., actual proper nouns such as BBC, ETA or AT&T and several adjectives wrongly labeled as nouns). On the other hand, in some dictionaries there were typos and words in other languages (e.g., the Portuguese inputs include Spanish words like garant a, asesor, cazador, or ateo, instead of their Portuguese equivalents garantia, assessor, cacador, and ateu). Therefore, they were not present in our vocabularies. Apart from these issues, several other mistranslations of LyS-DT were technical words from speci c domains (e.g., Medicine and Biology), which perhaps have low frequency in the Wikipedia corpus.
Finally, out of 13; 157 MWEs of our runs, LyS-DT could only translate 344, proving that our basic MWE approach was not enough to obtain suitable translations. A simple pre-processing of proper nouns (joint as single tokens) could improve both their precision and recall. For the other cases, specially for noncompositional expressions, more complex strategies need to be designed. 4
We consider that our participation in the TIAD shared task has been fruitful, as one of our systems has obtained the rst position among the participant teams (tied with Frankfurt), while the other three results we sent were placed from third to fth position. However, our systems failed to beat the \baselines" established by the organisers. In this regard, we must clarify that these are not really baselines but quite sophisticated state-of-the-art methods (one based on multilingual word embeddings and the other one based on the degree of overlap with the translations obtained by means of a pivot language).
Besides, it is worth mentioning that our approach only makes use of the lemma and PoS tag of each entry, so di erent senses of a word are simply inferred by their distributional contexts. In this respect, it could be interesting to make use of contextual information for each input word, in order to automatically select a speci c sense. Also, the use of contextualized distributional models could be an interesting topic for further research.
Finally, in future work we plan to improve our method by dealing with constructions that in the current version are not processed properly. In particular, we intend to design di erent strategies, both compositional and non-compositional, for the processing of compound proper nouns as well as for obtaining better vector representations of multiword expressions.