The WiC-ITA shared task aims to determine whether a word appearing in two distinct sentences carries the same meaning. The task consists of two subtasks: binary classification (Subtask 1) and ranking (Subtask 2). Each subtask is designed in both a monolingual (Italian) and multilingual (Italian-English) setting. In this report, we present the results of our participation in WiC-ITA. In our experiments, we leverage the condition number of the cosine similarity matrix between XLM-R embeddings and demonstrate competitive performance, ranking among the top positions in both the monolingual and cross-lingual setting. Our results indicate that semantic information is present not only in the last layers but also across the middle layers of XLM-R and throughout the entire architecture. This suggests potential avenues for future research to explore the use of the complete set of embeddings, rather than solely relying on the embeddings extracted from the last layer(s).
novel benchmark for evaluating WiC for both a monolingual (L) setting in Italian and a cross-lingual (XL) setting
niques has improved the modeling
Despite the growing popularity of contextualized
embeddings in research fields such as Word Sense
Disambiguation or Lexical Semantic Shift Detection (Scarlini
et al., 2020 [3]; Montanelli and
embedding model to compute a continuous similarity score. This score indicates the extent to which the target BERT is a powerful contextualized model that leverages carries the same meaning in the sentences 1 and 2. the Transformer encoder to capture the contextual se- More precisely, consider a sentence that contains mantics of words (Devlin et al., 2019 [2]; Vaswani et al., the word . Given a contextualized model , a vector 2017 [13]). Typically, the success of BERT is attributed representation of is extracted from every layer of the to its multi-layer (e.g., 12) and multi-head (e.g., 12) self- model . This way, the word in the sentence is assoattention blocks. However, most of the SOTA work only ciated with a set of contextualized embeddings denoted uses the outputs of the final layer(s) (i.e., word embed- by . It’s worth noting that ∈ R× , where is the dings) as input for solving NLP tasks, while ignoring the number of encoders of the model (e.g., 12) and is the output of the earlier layers. As a result, the role of difer- dimension of the embeddings (e.g., 768). As a result, we ent embedding layers for representing the semantics of denote as 1 and 2 the contextualized embeddings of word occurrences is still unclear. Recently, a limited num- extracted from the sentences 1 and 2, respectively. ber of studies have been conducted to explore the nature In order to evaluate the similarity of the word in and characteristics of the BERT embeddings. In particu- the contexts 1 and 2, we collect the pairwise cosine lar,Jawahar et al. (2019) [14] indicate that BERT’s lower similarities between 1 and 2. We denote as the layers capture surface features pertaining to phrase-level similarity matrix between 1 and 2 (see Figure 3 as an information, middle layers capture syntactic features, example 1) . Our hypothesis is that taking into account and higher layers capture semantic features. Devlin et information from all layers at once will provide a richer al. (2019) [2] report that combining the last four hid- and more comprehensive rapport of the nature of usage den layers could be beneficial for mainstream tasks such similarities of a word between the two sentences. We as Named Entitiy Recognition. Ethayarajh (2019) [15] hypothesize that because many layers are known to capdemonstrates that the geometry of the embedding space ture relevant semantic information, we should consider exhibits anisotropy, meaning that the embeddings of all as many of them as possible together, as they may conlayers occupy a narrow cone within the vector space. tain more comprehensive information than a single layer Other work involves probing tasks, as proposed in He- comparison approach. witt et al. (2019) [16]. These tasks consist of training an In order to tap into this pool of similarity scores enauxiliary classifier on top of a model, where the contextu- coded within (that contains 144 times more information alized embeddings serve as features to predict syntactic than a single layer) we use a measure called the condition (e.g., part-of-speech tags) and semantic (e.g., word rela- number. The condition number of a matrix, which was tions) properties of words. The idea is that if the auxiliary already successfully applied in other domains in NLP classifier accurately predicts a linguistic property, we can (Dubossarsky et al., 2020 [19]), provides us with a unified assume that the property is encoded in the tested model. measure that takes into account the many similarities
gate the capability of word sense prediction and indicate and 2 throughout the diferent layers. that earlier-layer embeddings contain significantly more Originally, the condition number of a matrix was semantic information than conventionally believed. used to measure its sensitivity to perturbations, or small
Thus, our experiments are motivated by the latter find- changes, in its input. A large condition number indicates
ing and inspired by linguistic research that highlights that the matrix is ill-conditioned, meaning it is sensitive
the influential role of morphology and syntax in shaping to small perturbations. On the other hand, a small
condiword meanin
on the similarity of two sets of word vectors. In particular, given a pair of sentences ⟨1, 2⟩, and a target word , we use the output embeddings of a contextualized varied similarity scores indicate more unrelated usages with ∈ 1, ..., . Additionally, we compute the coresulting in a higher (and worse) condition number. sine similarity CS between the word embeddings
multiplication of the matrix’s norm by the norm of its and 2, respectively. reciprocal (i.e., the inverse of the matrix). The norm In line with the WiC-ITA guidelines, we compute the could be Euclidean norm, Max norm, Frobenius norm, etc. Spearman correlation between the estimated similarity
(COND) of the similarity matrix using the Frobenius metric for Subtask 2. norm as follows: In Subtask1, our binary predictions are derived from the similarity scores obtained in Subtask 2. We employ a COND () = ‖‖ · ‖ − 1‖ threshold-based classifier, selecting the threshold value that optimizes the F1 score on the set of sentence pairs used as training set.
similarity matrix , we assess the degree of semantic similarity of a word in each pair ⟨1, 2⟩ as = COND() 2. 4. Experimental setup
by considering only a subset of . We test COND , In this task, we compared two diferent contextual
from the first, middle, and last four layers of the model 2019 [2]), and XLM-R (Conneau et al., 2020 [20]). We , respectively. use the Transformers library by HuggingFace to extract
lines the cosine similarity (CS) of the embeddings ex- models without performing any fine-tuning stage
CS(1[], 2[]) = , Given a target word and a pair ⟨1, 2⟩. The acqui‖1[]‖‖2[]‖ sition of contextual embeddings is done by feeding the models with the sentences 1 and 2 individually. For every sentence, we extract the token embedding for the target word from each layer of the model. Due to the
COND metric. We chose to associate smaller condition numbers with unrelated usages (annotated as 1), while larger numbers with identical usages (annotated as 4). Measures COND COND COND CS10 CS11 CS
CS9 CS8 CS7 CS6 CS12 CS5
CS4 COND
CS3 CS2 CS1 byte-pair input encoding scheme employed by BERT-like Train sets. models, some tokens may not correspond to complete Motivated by the superior results achieved during the words but rather to word pieces. In such cases, when development phase, we relied on XLM-R for our final a word is split into multiple tokens, we build a single submissions. In particular, we submit the predictions word embedding by averaging the embeddings of its con- obtained with COND, COND , COND . However, in stituent word pieces. Table 2, it is worth noting that COND also emerged as the
Finally, to assess the graded word similarity in the leading measure for the mBERT model, proving its concontext of a pair of sentences, we calculate similarity sistency. Moreover, we note that for the WiC-ITA task, scores between the contextualized embeddings of the the embeddings from the last layer of both XLM-R and target word under consideration (See Section 3). BERT, as well the embeddings derived by the aggregation of the last four layers, are not as efective as those from other layers. For instance, it is interesting to observe that 5. Experimental results layer 8 seems to be efective for Subtask1.
In our submissions, we rely on XLM-R as it proved to task, we ranked 2nd for L-Subtask1, 1st for XL-Subtask1, be more efective than mBERT. To maximize the perfor- 2nd for L-Subtask2, and 1st for XL-Subtask2. The leadermance of our system, we leverage the available train board is reported in Table 3. and dev set as a whole. In particular, we randomly gen- Our final results at WiC-ITa demonstrate that COND erate 100 diferent train-test splits, with sizes of 2000 efectively captures semantic features of word meanings and 1305 respectively (equivalent to 60% and 40% of the and can be successfully applied to tasks like WiC. Based full dataset). We conduct cross-validation on these 100 on our development results, we assert that COND consplits to validate the use of COND for Subtask2. Addi- sistently outperforms the CS measure computed over tionally, we leverage cross-validation to determine the individual contextualized embeddings, for Subtask 1 and optimal threshold for Subtask1, meaning that we rely on 2 in both in L and XL setting. This is particularly interthe average of the 100 best thresholds obtained during esting considering that CS is commonly utilized in NLP cross-validation. The average scores of Spearman cor- tasks to capture contextual semantics in contextualized relation, Precision, Recall, and F1 score are presented embeddings. in Table 1 for each tested measure. For Subtask1 and 2 Finally, COND consistently achieves good results and for both the L and XL setting, our three submissions by considering medium layers alone. These results are correspond to the top three measures based on the F1 in line with the findings of Coenen et al. (2019) [ 17], score and Spearman correlation, respectively (i.e., COND, and suggest that the middle layers of BERT-like models COND , COND). contain valuable information for efectively representing
For the sake of comparison, Table 2 presents the pre- meaning. Therefore, future work should explore the apliminary performance achieved during the development plication of COND for WiC and other related NLP tasks phase with both XLM-R and mBERT over the Dev and such as Lexical Semantic Change Detection (Montanelli
1–6 Preliminary performance achieved during the development phase with both XLM-R and mBERT over the Dev and Train sets. We report in bold the best result for each metric, model, and data set.
Evaluation Phase Subtask1
Subtask2
Teams BERT 4EVER BERT 4EVER BERT 4EVER
LG extremITA extremITA
Baseline The Time-Embedding Travelers The Time-Embedding Travelers The Time-Embedding Travelers
Run run1 run2 run3
LG COND COND
COND camoscio lora it5
L-WiC
condition number of the cosine similarity matrix between ported by the Swedish Research Council (2019–2022; con
tract 2018-01184), and in part by the research program results support our initial hypothesis that leveraging all