Translations capture important information about languages that can be used as implicit supervision in learning linguistic properties and semantic representations. In an information-centric view, translated texts may be considered as semantic mirrors of the original text and the signi cant variations that we can observe across various languages can be used to disambiguate a given expression using the linguistic signal that is grounded in translation. Parallel corpora consisting of massive amounts of human translations with a large linguistic variation can be applied to increase abstractions and we propose the use of highly multilingual machine translation models to nd language-independent meaning representations. Our initial experiments show that neural machine translation models can indeed learn in such a setup and we can show that the learning algorithm picks up information about the relation between languages in order to optimize transfer leaning with shared parameters. The model creates a continuous language space that represents relationships in terms of geometric distances, which we can visualize to illustrate how languages cluster according to language families and groups. Does this open the door for new ideas of data-driven language typology with promising models and techniques in empirical cross-linguistic research?
Our primary goal is to learn meaning representations of sentences and sentence fragments by looking at the distributional information that is available in parallel corpora of human translations. The basic idea is to use translations into other languages as \semantic mirrors" of the original text, assuming that they represent the same meaning but with di erent symbols, wordings and linguistic structures. For this we discard any meaning diversions that may happen in translation due to target audience adaptation or other processes that may in uence the semantics of the translated texts. We also assume that the material can be divided into meaningful and self-contained units, Bible verses in our case, and focus on the global data-driven model that hopefully can cope with instances that violate our assumptions.
Our model is based on the intuition that the huge amount of variation and the cross-lingual di erences in language ambiguity make it possible to learn semantic distinctions purely from data. The translations are, thus, used as a naturally occurring signal (or cross-lingual grounding) that can be applied as a form of implicit supervision for the learning procedure, mapping sentences to semantic representations that resolve language-internal ambiguities. With this approach we hope to take a step forward in one of the main goals in arti cial intelligence, namely the task of natural language understanding. In this paper, however, we emphasise the use of such models in the discovery of linguistic properties and relationships between languages in particular. Having that in mind, the study may open new directions for collaborations between language technology and general linguistics. But before coming back to this, let us rst look at related work and the general principles of distributional semantics with cross-lingual grounding.
The use of translations for disambiguation has been explored in various
studies. Dyvik [
The idea of cross-lingual disambiguation is simple. The following example illustrates the e ect of disambiguation of idiomatic uses of \put o " through translation into German: English: I don't want to put you o . The meeting has been put o again. German: Ich will dich nicht abschrecken. Das Tre en wurde wieder verschoben.
Using the general idea of the distributional hypothesis that \you shall know
a word by the company it keeps" [
Our framework will be neural machine translation (NMT) that applies an
encoder-decoder architecture, which runs sequentially through a string of input
symbols (for example words in a sentence) to map the information to dense
vector representations, which will then be used to decode that information in
another language. Figure 1 illustrates the general principle with respect to the
classical Vauquois triangle of machine translation [
Data-Driven Machine Translation
Man, this is so boring.
Hmm, every time he sees “banco”, he either types “hbeasneke”sor“b“baenncochd”e ”b,ut if he always types “bank”, never “bench” lahnugmuangetrdanastlaations
meaning Translated documents source language
Learning Algorithm understanedinndgceorndesepernrnveeeestwecutnrooatrarlk-tbidoaencsoedderspeaking
Ey Z
C target language Ex y1 y2
y3 h1 x1
Translation models are precisely the kind of machinery that tries to transfer the meaning expressed in one language into another by analysing (understanding) the input and generating the output. NMT tries to learn that mapping from data and, thus, learns to \understand" some source language in order to produce proper translations in a target language from given examples. Our primary hypothesis is that we can increase the level of abstraction by including a larger diversity in the training data that pushes the model to improve compression of the growing variation and complexity of the task. We will test this hypothesis by training multilingual models over hundreds or even almost a thousand languages to force the MT model to abstract over a large proportion of the World's linguistic diversity.
As a biproduct of multilingual models with shared parameters, we will obtain a mapping of languages to a continuous vector space depicting relations between individual languages by means of geometric distances. In this paper, we present our initial ndings when training such a model with over 900 languages from a collection of Bible translations and focus on the ability of the model to pick up genetic relations between languages when being forced to cover many languages in one single model.
In the following, we will rst present the basic architecture of the neural translation model together with the setup for training multilingual models. After that we will discuss our experimental results before concluding the paper with some nal comments and prospects for future work. 2
Neural machine translation typically applies an end-to-end network architecture
that includes one or several layers for encoding an input sentence into an
internal dense real-valued vector representation and another layer for decoding that
representation into the output of the target language. Various variants of that
mo del have b een prop osed in the recent literature [
Discrete input symb ols are mapp ed via numeric word representations
(emb eddings E ) onto a hidden layer (C ) of context vectors (h), in this case by a
bidirectional RNN that reads the sequence in a forward and a reverse mo de.
The enco ding function is often mo deled by sp ecial memory units and all mo del
parameters are learned during training on example translations. In the simplest
case, the nal representation (returned after running through the enco ding layer)
is sent to the deco der, which unrolls the information captured by that internal
representation. Note that the illustration in Figure 1 includes an imp ortant
addition to the mo del, a so-called attention mechanism. Attention makes it p ossible
to fo cus on particular regions from the enco ded sentence when deco ding [
variation enforces
Hmm, everyHtmimme, heevesreyHetmsimme, heevesreyHetmsimme, heevesreyetsime he sees DaMatna,thD-is DisaMsroatinabv,otrhD-enisDgi.nsaMsroatinMabv,otrhD-einasDgi.nsacMsroathinMabv,oitrhn-einasDgei.nscsr“hhnobheeeTiaMvbsvsaneoielkrwrrtne”ei“narsaobgyreoe“.nsbn“cnbatcny“hhnenhbpheeencsT”aMevgocsansehiellkdrwr“ene”ab“asaobayrree“ntsb”bn“ckn,ibatuc”ay“hhnoe,nhtbpheeencsT”iaefvbcosannsehiellkdrwr”“nsaeb“asaobaytree“ntsb”brn“kn,ibatuc”ay“hhnoe,nhtbpeeencsT”iaefvcocsannsehellkdrwrt”“aeb“iasaobaoyre“ntsb”bn“kn,ibnatuc”yoe,nhtpncs”iefconshld”“aebant”bk,iu”o,tif nm
“banco”, he“ebiathnecro”ty,phes“ebiathnecro”ty,phes“ebiathnecro”ty,pheseither types humlaangturagnesladtaiotnas
Hmm, everyHtmimme, heevesreyHetmsimme, heevesreyHetmsimme, heevesreyetsime he sees “banco”, he“ebiathnecro”ty,phes“ebiathnecro”ty,phes“aebiathdnecrod”ty,pheseither types “bank” or “b“ebnacnhk” or b“bu“etbniafcnhk” or b“bu“etbniafcnhk” or b“buet nifch” but if Man, this isMsoanb,otrhinsgi.sMsoanb,otrhinsgi.sMsoanb,otrhinsgi.s shoebsoereinsg“.bahnecosedes “b”,ahnecosedes “b”,ahnsecoisgedesn“b”i,afnicco aden”t, he always tyhpeeasl w“baaynskt”yh,peeasl w“baaynskt”yh,peeasl w“baaynskt”y,pes “bank”, never “bencnhe”ver “bencnhe”ver “bencnlhei”vner g“beuncihs” tic TranslatedTdroacnusmlatendtTsdroacnusmlatendtTsdroacnusmlatendtsdocumdenitsversity TranslatedTdroacnusmlatendtTsdroacnusmlatendtTsdroacnusmlatendtsdocuments eaning neural network
DAN I Jeg ved don't ikke,
GER Ich
I don't know what nde keine Worte. to sa
DUT I Ik heb don't know what er geen woorden to sa vo or.
Fig. ultilingual
Neural
MT and training data with language ags.
All parameters of the network are trained on large collections of human translations (parallel corp ora) typically by some form of gradient descent (iterative function optimisation) that is backpropagated through the network. The attractive prop erty of such a mo del is the ability to learn representations that re ect semantic prop erties of the input language through the task of translation. However, one problem is that translation mo dels can b e \lazy" and avoid abstractions if the mapping b etween source and target language do es not require any deep understanding. This is where the idea of multilinguality comes into the picture: If the learning algorithm is confronted with a large linguistic variety then it has to generalize and to forget about language-pair-speci c shortcuts. Covering substantial amounts of the world's linguistic diversity as we propose pushes the limits of the approach and strong abstractions in C can be expected. Figure 2 illustrates the intuition behind that idea.
Various multilingual extensions of NMT have already been proposed in the
literature. The authors of [
Our question is whether we can use a standard NMT model with a much larger
coverage of the linguistic diversity of the World in order to maximise the variation
signalling semantic distinctions that can be picked up by the learning procedures.
Figure 3 illustrates our setup based on a model trained on over 900 languages
from the multilingual Bible corpus [
We trained the model in various batches and observed the development of the model in terms of translation quality on some small heldout data. The heldout data refers to an unseen language pair, Swedish-Portuguese in our case (in both directions). We selected those languages in order to see the capabilities of the system to translate between rather distant languages for which a reasonable number of closely related languages are in the data collection to improve knowledge transfer.
The results demonstrate so far that the network indeed picks up the
information about the language to be produced. The decoder successfully switches to the
selected language and produces relatively uent Bible-style text. The adequacy
of the translation, however, is rather limited and this is most probably due to the
restricted capacity of the network with such a load of information to be covered.
Nevertheless, it is exciting to see that such a diverse material can be used in one
single model and that it learns to share parameters across all languages. One of
the most interesting e ects that we can observe is the emerging language space
that relates to the language ags in the data. In Figure 4 we plot the language
space (using t-SNE [
Helsinki Neural
MT System Bible translations in 927 languages Ey
other languages
English vector space of language embeddings train
Z C Ex multilingual
NMT model learning to translate Portuguese - Swedish (“zero-shot MT”) family for the ten language families / groups with most members in our data set.
We can see that languages roughly cluster according to the family they belong to. Note that this is purely learned from the data based on the objective to translate between all of them with a single model. The training procedure learns to map closely related languages near to each other in order to increase knowledge transfer between them. This development is very encouraging and demonstrates the ability of the neural network model to optimise parameter sharing to make most out of the model's capacity.
An interesting question coming out of this study is whether such multilingual
translation models can be used to learn linguistic properties of the languages
involved. Making it possible to measure the distance between individual languages
in the emerging structures could be useful in data-driven language typology and
other cross-linguistic studies. The results so far, do not reveal a lot of
linguistically interesting relations besides the projection of languages onto a global
continuous space with real-values distances between them. Nevertheless,
quantifying the distance is potentially valuable and provides a more ne-grained
relation than discrete relations coming from traditional family trees. It is, however,
still an open question what kind of properties are represented by the language
embeddings and further studies are necessary to see whether speci c
linguistic features can be identi ed and isolated from the distributed representations.
There is a growing interest in interpretability of emerging structures and related
work already demonstrates the ability of predicting typological features with
similar language representations [
Trans-New Guinea Otomanguean Quechuan Indo-European Austronesian Nilo-Saharan Afro-Asiatic Mayan Niger-Congo Creole
Massively parallel data sets make it now possible to study speci c typological
structures with computational models, for example tense and aspect as in [
Another direction is to explore the inter-lingual variations and language developments using, for example, the alternative translations that exist for some languages in the Bible corpus. However, even here the data is rather sparse and it remains to be seen how reliable any emerging pattern will be. Crucial for the success will be a strong collaboration with scholars from the humanities, which shows the important role of digital humanities as a eld. 4
In this paper, we present our experiments with highly multilingual translation models. We trained neural MT models on Bible translations of over 900 languages in order to see whether the system is capable of sharing parameters across a large diverse sample of the World's languages. Our motivation is to learn languageindependent meaning representations using translations as implicit semantic supervision and cross-lingual grounding. Our pilot study demonstrates that such a model can pick up the relationship between languages purely from the data and the translation objective. We hypothesise that such a data-driven setup can be interesting for cross-linguistic studies and language typology. In the future, we would like to investigate the emerging language space in more detail also in connection with alternative network architectures and training procedures. We believe that empirical methods like this one based on automatic representation learning will have signi cant impact on studies in linguistics providing an objective way of investigating properties and structures of human languages emerging from data and distributional patterns.
We would like to thank the anonymous reviewers for their valuable comments and suggestions as well as the Academy of Finland for the support of the research presented in the paper with project 314062 from the ICT 2023 call on Computation, Machine Learning and Arti cial Intelligence.