This study investigates the impact of cross-linguistic similarities on idiom representations in mBERT, focusing on English and German idioms categorized by diferent degrees of similarity. We aim to determine whether diferent degrees of cross-linguistic similarities significantly afect mBERT's representations and to observe how these representations change across its 12 layers. Contrary to our initial hypothesis, cross-linguistic similarity did not uniformly impact idiom representations across all layers. While early and middle layers showed no significant diferences among idiom categories, higher layers (from Layer 8 onwards) revealed more nuanced processing. Specifically, significant diferences between the control category and idioms with similar meaning (SM), as well as between idioms with similar lexical items (SL) and those with similar semantics (SM) were observed. Our analysis revealed that early layers provided general representations, while higher layers showed increased diferentiation between literal and figurative meanings. This was evidenced by a general decrease in cosine similarities from Layer 5 onwards, with Layer 8 demonstrating the lowest cosine similarities across all categories. Interestingly, a trend suggests that mBERT performs slightly better with more literal hints. The order of cosine similarity for the categorizations was: idioms with a degree of formal similarity, control idioms, idioms with both formal and semantic similarity, and finally idioms with only semantic similarity. These findings indicate that mBERT's processing of idioms evolves significantly across its layers, with cross-linguistic might afect more significantly in higher layers where more abstract semantic processing likely occurs.
case study in diferent experimental linguistics settings.
Idioms are one of the most studied linguistic concepts Recent advancements in Large Language Models (LLMs)
that broadly can be defined as multi-word expressions and their widespread application have prompted linguists
that are often fixed in terms of their syntactic and lexical to investigate the performance of these models across
aspects, while they usually carry meanings that cannot be various linguistic concepts, including idioms [14, 15, 16].
directly deduced from the meaning of individual words In addition, in the case of multi-lingual models, an
inthey contain [
1. Does cross-linguistic similarity have a
significant impact on the representation of idioms in
mBERT?
2. Does the degree of cross-linguistic similarity and
the representation of the model change across the
12 layers of mBERT?
languages, it should perform consistently across all cross- ies on idioms, one of the aspects that have been studied
linguistic categories, similar to how it represents idioms is the concept of cross-linguistic similarity or
translatafrom the language it has been given, that is English in bility. Among language speakers, the degree of
transthis case. However, if it primarily retrieves data from latability of an idiom in their L1 and L2 seems to play a
specific languages, we expect to observe significant per- significant role in how they interpret and understand the
formance diferences among the categories, potentially idioms [
such as BERT and mBERT [30] are also able to capture idioms [26, 27, 28]. Studies on the internal mechanisms of how transformer-based language models process id- To investigate our research questions concerning the imioms demonstrated that BERT, Multilingual BERT, and pact of cross-linguistic similarity on the representation DistilBERT represent idioms distinctively compared to of idioms in mBERT and how this representation changes literal language [16]. These studies also observed that across the model’s 12 layers, a list of idiomatic expresthe semantic meaning of idioms is captured more efec- sions was compiled. the dataset consists of 72 idioms: tively in deeper layers of the models. They found that 54 from German and 18 from English, the latter servwords within idioms receive less attention from other ing as a control group. The German idioms are classified words in the sentence compared to words in literal con- based on their similarity with English idioms, using three texts. However, [14] argue that LLMs capture MWE se- categories of cross-linguistic correspondence. The first mantics inconsistently, as shown by reliance on surface category includes idioms with the highest degree of forpatterns and memorized information. MWE meaning is mal and semantic similarity. These idioms, such as die also strongly localized, predominantly in early layers of Ruhe vor dem Sturm, have a corresponding form in Enthe architecture. They also discuss that representations glish when translated word-for-word, e.g., the calm before benefit from specific linguistic properties, such as lower the storm. In addition to the formal similarity, the meansemantic idiosyncrasy and ambiguity of target expres- ing of the idiom in the target language is also similar to sions. that of the originating language, in this case referring
Moving from LLMs and idioms, there are diferent to a period of calmness before argument or trouble. The arguments on how models such as BERT work [31], second category focuses on formal similarities without and in the case of multi-lingual approaches, how multi- semantic correspondence. For instance, jemanden auslingual they are [17, 18, 32]. Works on the mechanisms of nehmen wie eine Weihnachtsgans (’to gut someone like a BERT demonstrate that it captures significant linguistic Christmas goose’) refers to financially exploiting someinformation, with lower layers focusing on local syn- one. In English, there is an idiom that contains the word tactic relationships and higher layers encoding more "goose" - to cook one’s goose - but it refers to sabotaging complex linguistic features. The self-attention heads someone’s plans, demonstrating some degrees of formal in BERT show specialization for certain linguistic func- and lexical similarity without semantic alignment. The tions, though many exhibit redundant patterns, suggest- third category encompasses idioms with semantic simiing overparameterization. While BERT demonstrates larities but no formal correspondence. For example, the some ability to capture world knowledge, its reasoning German idiom Den Löfel abgeben (’to pass the spoon’) capabilities appear limited. Despite impressive perfor- and the English idiom to kick the bucket both convey the mance on many NLP tasks, BERT shows limitations in meaning of dying, while sharing no formal similarities. handling negation, numerical reasoning, and complex After categorizing the idioms, the German idioms were inference, often relying on shallow heuristics [31]. Inves- literally translated into English. We literally translated tigations on mBERT across 39 languages found that it the idioms to ensure all expressions can be fed to the performs well on high-resource languages but struggles model in a single language. This approach allows us to with low-resource languages. For languages with lim- control for the language space in which idioms are preited Wikipedia data (which was used to train mBERT), sented, given that in more complex tasks diferent subsets performance drops significantly, especially for tasks like of mBERT can afect how idioms are represented [ 33]. named entity recognition. This suggests that the qual- Additionally, for each idiom, a brief entity or description ity of representations learned by mBERT is not uniform is selected reflecting its figurative meanings. For examacross all 104 languages it supports [32]. Additionally, ple, for "the calm before the storm", "episodic tranquility" [18] conducted a series of probing experiments to under- is chosen, which refers to the figurative interpretations stand mBERT’s cross-lingual abilities. They found that of the idiom. Table 1, summarizes the proposed categomBERT performs surprisingly well on zero-shot cross- rizations, the original and translated idioms, along with lingual model transfer, even between languages with dif- their figuratively related entities. ferent scripts. Their analysis suggests that mBERT learns multilingual representations that go beyond simple vo- 4. Model, and Experiment cabulary memorization. However, they also note that transfer works best between typologically similar languages, indicating some limitations in mBERT’s ability to generalize across very diferent language structures.
their figurative meanings, the dataset was processed using the "bert-base-multilingual-uncased" model [34] without any fine-tuning. This model consists of 12 hidden layers, each containing 768 neurons, and the activity of each layer was extracted for the CLS token. Embeddings for the CLS token from each of the 12 layers for every idiom and its associated meanings were extracted. The model is pretrained on the 102 languages with the largest Wikipedias, which includes both German, the language from which our idioms are derived, and English, which is the target language for the translation of the idioms and used for deriving the embeddings. For each sample, the embeddings of the [CLS] token from all 12 layers of mBERT are extracted. The [CLS] token was chosen because it is designed to capture sentence-level semantics in BERT models [35]. Using the [CLS] token’s embedding from models can be used as a powerful method for semantic comparison of texts, which can then be compared using similarity measures. vIdiom · vMeaning Cosine Similarity = cos( ) = |vIdiom||vMeaning| (1)
In Equation 1, v stands for word embedding, which is a vector with a length of 768. To interpret the result of the cosine similarity in the context of word embedding, a score of 1 means the vectors are identical, 0 means the vectors are orthogonal (no similarity), and -1 means the vectors are opposed.
After deriving the CLS embeddings from all layers of mBERT for the translated idioms and their corresponding ifgurative meanings, the cosine similarities among the derived embeddings were calculated. Figure 1 illustrates the cosine similarities across diferent layers of mBERT for each idiom category. As it can be seen, the first layer of mBERT showed identical cosine similarities (equal to 1) for all idioms, representing the entry point of the model. Therefore, this layer is excluded from subsequent analyses to avoid skewing our results.
0 1 2 3 4 5 6 7 8 9 10 11
Layer
4.1. Similarity Calculation cosine similarities exhibited notable variations across layIn the next step to measure how similar BERT’s under- ers. Layer 3 demonstrated the highest cosine similarities standing is of each idiom, the similarity of embeddings across all categories; while layer 8 showed the lowest for each idiom with its figurative meanings was calcu- cosine similarities for all four categories. In addition, as lated. Cosine similarity is used, a widely used method it can be seen from Layer 5 onwards, we observed a genbecause of its efectiveness and it is mainly used to de- eral decrease in cosine similarities, suggesting increasing termine how similar or related two words are based on diferentiation between CLS representation of idioms and their vector representations [36, 37]. their corresponding figurative meanings in higher layers.
To test our hypothesis on how the embeddings of mBERT would change given the proposed cross-linguistic
To examine the predicted Cosine Similarity of Figurative CLS representations for each combination of Category and Layer, the estimated marginal means using the emmeans package [41] in R computed. In this analysis, the changes in the cosine similarities were compared among the categories, in diferent layers. The results of the pair-wise comparisons indicate that, For Layers 1-7, there are no significant diferences between categories (all p-values > 0.05), this can be also seen in figure 3, in which almost until the 7th layer all of the lines align with each other. However, from layer 8 a significant diference can be seen between the control category and category SM that represents the idioms with cross-linguistically similar semantics (estimate = 0.0272, p = 0.0179). In addi
Our study investigated how cross-linguistic similarities among idioms afect their representation in mBERT, with a focus on English and German idioms categorized based on three degrees of cross-linguistic similarity. This study aims to answer two main research questions concerning whether cross-linguistic similarity has a significant impact on the representation of idioms in mBERT, and how the degree of cross-linguistic similarity and the representation of the model change across the 12 layers of mBERT.
Our findings provide insights into these questions and our initial hypotheses. Contrary to our initial hypothesis, we found that cross-linguistic similarity does not have a uniformly significant impact on the representations of idioms across all layers of mBERT. The main efects of our translated idioms categorized into cross-linguistic categories (SI: formal and semantic similarity, SL: similar lexicon, SM: Similar Meaning), were not statistically significant when compared to the control category (English idioms) in the early and middle layers of the model. This result may suggest that mBERT might be utilizing knowledge from all languages in its training data as a collective pool, at least in the case of the studied idioms. This aligns with the idea that mBERT learns multilingual representations that go beyond simple vocabulary memorization, as suggested by Pires et al. [18]. However, the emergence of significant diferences in higher layers (particularly from Layer 8 onwards) might indicate that mBERT’s processing of idioms becomes more nuanced as information 6.1. Limitations and future research This research also has limitations, that can be tackled in the further and future studies. One of the primary limitations of our study is the size of the dataset. However, the dataset has a good variety of samples but a bigger dataset may improve the generalizability and robustness of our ifndings. Future research should aim to include a more extensive dataset to confirm and extend these findings.
Moreover, literally translating the idioms and the figuratively related entities, can afect on the representations of the model, and the derived cosine similarities; therefore, in further studies, it can be insightful to compare also, how the representations of the model change if the [14] F. Miletić, S. S. i. Walde, Semantics of multiword org/P16-1019. doi:10.18653/v1/P16-1019. expressions in transformer-based models: A survey, [28] V. Nedumpozhimana, J. Kelleher, Finding bert’s Transactions of the Association for Computational idiomatic key (2021).
Linguistics 12 (2024) 593–612. [29] A. Fazly, P. Cook, S. Stevenson, Unsupervised [15] M. TAN, J. JIANG, Does bert understand idioms? Type and Token Identification of Idiomatic Expresa probing-based empirical study of bert encodings sions, Computational Linguistics 35 (2009) 61–103. of idioms.(2021), in: Proceedings of the Interna- URL: https://doi.org/10.1162/coli.08-010-R1-07-048. tional Conference on Recent Advances in Natural doi:10.1162/coli.08-010-R1-07-048. Language Processing (RANLP 2021), Virtual Con- [30] J. Devlin, M. Chang, K. Lee, K. Toutanova, ference, September, ????, pp. 1–3. BERT: pre-training of deep bidirectional trans[16] Y. Tian, I. James, H. Son, How are idioms processed formers for language understanding, CoRR inside transformer language models?, in: Proceed- abs/1810.04805 (2018). URL: http://arxiv.org/abs/ ings of the 12th Joint Conference on Lexical and 1810.04805. arXiv:1810.04805. Computational Semantics (* SEM 2023), 2023, pp. [31] A. Rogers, O. Kovaleva, A. Rumshisky, A primer 174–179. in bertology: What we know about how bert [17] H. Gonen, S. Ravfogel, Y. Elazar, Y. Goldberg, It’s works, 2020. URL: https://arxiv.org/abs/2002.12327. not greek to mbert: inducing word-level trans- arXiv:2002.12327. lations from multilingual bert, arXiv preprint [32] S. Wu, M. Dredze, Are all languages created equal in arXiv:2010.08275 (2020). multilingual bert?, arXiv preprint arXiv:2005.09093 [18] T. Pires, E. Schlinger, D. Garrette, How mul- (2020).
tilingual is multilingual bert?, arXiv preprint [33] J. Libovicky`, R. Rosa, A. Fraser, How languagearXiv:1906.01502 (2019). neutral is multilingual bert?, arXiv preprint [19] C. Cacciari, P. Tabossi, The comprehension of id- arXiv:1911.03310 (2019).
ioms, Journal of memory and language 27 (1988) [34] J. Devlin, M. Chang, K. Lee, K. Toutanova, 668–683. BERT: pre-training of deep bidirectional trans[20] S. Irujo, Don’t put your leg in your mouth: Transfer formers for language understanding, CoRR in the acquisition of idioms in a second language, abs/1810.04805 (2018). URL: http://arxiv.org/abs/ tesol Quarterly 20 (1986) 287–304. 1810.04805. arXiv:1810.04805. [21] J. I. Liontas, Killing two birds with one stone: Un- [35] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, derstanding spanish vp idioms in and out of context, Bert: Pre-training of deep bidirectional transformHispania (2003) 289–301. ers for language understanding, arXiv preprint [22] D. Titone, G. Columbus, V. Whitford, J. Mercier, arXiv:1810.04805 (2018).
M. Libben, Contrasting bilingual and monolingual [36] N. Reimers, I. Gurevych, Sentence-bert: Sentence idiom processing. (2015). embeddings using siamese bert-networks, arXiv [23] H. Bortfeld, Comprehending idioms cross- preprint arXiv:1908.10084 (2019). linguistically., Experimental psychology 50 (2003) [37] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, 217. Y. Artzi, Bertscore: Evaluating text generation with [24] M. S. Senaldi, D. A. Titone, Less direct, more analyt- bert, arXiv preprint arXiv:1904.09675 (2019). ical: Eye-movement measures of l2 idiom reading, [38] D. Bates, M. Mächler, B. Bolker, S. Walker, Fitting Languages 7 (2022) 91. linear mixed-efects models using lme4, Journal of [25] M. S. Senaldi, J. Wei, J. W. Gullifer, D. Titone, Statistical Software 67 (2015) 1–48. doi:10.18637/ Scratching your tête over language-switched id- jss.v067.i01. ioms: Evidence from eye-movement measures of [39] R Core Team, R: A Language and Environment for reading, Memory & cognition 50 (2022) 1230–1256. Statistical Computing, R Foundation for Statistical [26] V. Nedumpozhimana, F. Klubička, J. D. Kelleher, Computing, Vienna, Austria, 2023. URL: https:// Shapley idioms: Analysing bert sentence embed- www.R-project.org/. dings for general idiom token identification, Fron- [40] D. J. Schad, S. Vasishth, S. Hohenstein, R. Kliegl, tiers in Artificial Intelligence 5 (2022) 813967. How to capitalize on a priori contrasts in linear [27] G. Salton, R. Ross, J. Kelleher, Idiom token classi- (mixed) models: A tutorial, Journal of memory and ifcation using sentential distributed semantics, in: language 110 (2020) 104038.
K. Erk, N. A. Smith (Eds.), Proceedings of the 54th [41] F. M. S. S. R. Searle, G. A. Milliken, Population Annual Meeting of the Association for Computa- marginal means in the linear model: An alternative tional Linguistics (Volume 1: Long Papers), Asso- to least squares means, The American Statistician ciation for Computational Linguistics, Berlin, Ger- 34 (1980) 216–221. URL: https://www.tandfonline. many, 2016, pp. 194–204. URL: https://aclanthology. com/doi/abs/10.1080/00031305.1980.10483031. Groups idiom 1.00 -0.00 -0.00 -0.00 0.00 0.00 0.00 -0.04 -0.07 -0.10 -0.15 -0.13 -0.12 -0.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -0.00 0.01 0.00 -0.00 0.01 -0.00 -0.01 0.01 -0.01 -0.01 0.00 -0.03 -0.01 0.01 -0.02 -0.01 0.01 -0.02 -0.01 0.00 -0.02
Fixed Efects