In this paper we describe our participation in the CONcreTEXT task of EVALITA 2020, which involved predicting subjective ratings of concreteness for words presented in context. Our approach, which ranked first in both the English and Italian subtasks, relies on a combination of context-dependent and context-independent distributional models, together with behavioural norms. We show that good results can be obtained for Italian, by first automatically translating the Italian stimuli into English, and then using existing resources for both Italian and English.
In our everyday life we rarely encounter words in
isolation. Instead, we typically process words as
part of sentences or phrases, and these linguistic
contexts shape our understanding of individual
words. However, for various reasons, the
overwhelming majority of behavioural norms that
have been collected so far focus only on single
words or word pairs
Thus, the EVALITA 2020
In this paper we describe our computational models, based on pre-trained distributional models and behavioural norms, which ranked first in both the English and Italian tracks of the competition1. We find that the best performance can be obtained by employing a combination of transformer models, developed in the last 2 years. Moreover, for Italian, it is possible to reach good levels of performance by relying on both the original stimuli and their English translation, which allows access to resources for both languages. 1.1
In order to predict concreteness in context, we use information derived from three type of sources, namely behavioural norms and distributional models, both context-independent (i.e., a model outputs the same vector representation for a given word, regardless of the context in which the word is encountered), and context-dependent (i.e., a model outputs a potentially different representations for a given word, as a function of the context in which the word is presented).
Firstly, we employ behavioural norms collected
for a wide variety of psycholinguistic factors. Of
particular interest to us are norms for concreteness
We focus on these specific factors since they are meaningfully related to word concreteness (see the previous references).
Secondly, we employ context-independent
distributional models, namely Skip-gram
Thirdly, we employ context-dependent
distributional models, namely BERT (Devlin et al.,
2018), RoBERTa (Liu et al., 2018), AlBERTo
We tested (combinations of) three groups of predictors. The first group was derived from large datasets of ratings for concreteness, semantic diversity, age of acquisition, emotional dimensions, and sensorimotor dimensions, as well as frequency and contextual diversity counts based on the SUBTLEX-UK and BNC corpora (see the references from the beginning of the previous section). In order to extend the coverage of the subjective ratings, we did not directly use them as predictors of concreteness in context. Instead, we relied on the Skip-gram, GloVe, and ConceptNet NumberBatch models, as a means of estimating the subjective ratings for more than 100,000 words, via linear regression. For the frequency and contextual diversity counts, we kept the original values, as they already have very good coverage. The intersection of the two datasets, which includes more than 70,000 words, served as the basis for our predictors of concreteness. More specifically, for each variable V (e.g., semantic diversity), we generated four predictors, namely V(w), V(c), V(w) * V(c), and abs(V(w) - V(c)), where: • V(w) denotes the value of V corresponding to the word w (e.g., w = “offend”). If w is not present in our norms, we set V(w) • to the average value of V, computed over the entire norms; V(c) denotes the value of V corresponding to the context c in which the word w is encountered (e.g., w = “offend”, c = “Do not insult or ___ anyone .“). Computing this value involves calculating the average V(c) = ∑!"#$ "($!) , where V(ci) is the & value of V corresponding to the i-th context word, calculated as described previously, and N is the number of words that make up the context.
These predictors allowed us to include both the individual contributions of word w and its context c, as well as certain interactions between w and c.
The second group was derived from Skip-gram,
GloVe, and ConceptNet NumberBatch
embeddings, as well as from the concatenation of the
three types of embeddings. The vocabulary of the
four models is that described in the discussion
above. Given the large number of dimensions
involved (i.e., 300 + 300 + 300 + 900 = 1,800), we
first extracted the top 20 principal components
from each model (although comparable results
can also be obtained by using a larger number of
components). Then, for each variable V (e.g., PC3
from the GloVe model) we generated four
predictors, namely V(w), V(c), V(w) * V(c), and abs(V(w)
- V(c)), following the same procedure as in the
previous discussion. In addition, based on
The third group was derived from the BERT, GPT-2, Bart, and ALBERT models. We used the standard (base) versions of each model (i.e., without task-specific fine-tuning), as described in the original papers, and obtained from the Hugging Face repository (https://huggingface.co/models).
Unlike for the previous two groups, the predictors consist only of a word’s activations from the last hidden layer (i.e., for the GPT-2, Bart, and ALBERT models), or averaged from the last four hidden layers (i.e., for the BERT model).
Importantly, for each group of predictors we generated two sets of variables, based on two versions of the target words (i.e., the words rated by the participants). In the first set we used the uninflected form of the target words, taken from the TARGET column. In contrast, in the second set of we used the inflected form of the target words, taken from the words in the TEXT column located at the positions specified in the INDEX column. More details can be found in Table 1.
For predicting ratings of concreteness in context, we employed ridge regression, with large values of the parameter lambda (i.e., strong regularization), after standardized all the variables. 1.3
Our approach was similar to that for English, but
with certain significant changes, as follows:
• for the first group of predictors, we began
by automatically translating the Italian
stimuli (i.e., the TARGET and TEXT
columns) into English, using the MarianMT
translation model
As in the case for English, we generated two sets of predictors, using either the uninflected or inflected forms of the target words, together with their corresponding English translations. More details can be found in Table 1.
Once more, we employed ridge regression, with large values of the parameter lambda (i.e., strong regularization), after standardizing all the variables. 2
The results for English and Italian are shown in Figures 1 and 2, respectively, for various sets of predictors and regularization strengths. Results are averaged over 1,000 rounds of 5-fold crossvalidation, using only the training dataset.
For English, the results indicate that contextdependent models (Fig. 1c-d) outperform behavioural norms (Fig. 1a) and context-independent models (Fig. 1b). For the latter, even though we introduced contextual variables by averaging a given variable (e.g., concreteness) over the words that make up the context, it appears that this simple average does not properly capture contextual information and/or interactions between single word and contextual information. The addition the behavioural norms and/or context-independent models has a negligible effect on performance (Fig. 1e). In this respect, the excellent results for context-dependent models are likely due to several factors, such as the highly non-linear integration of contextual information, the use of attention mechanisms, and that of more sophisticated learning objectives (e.g., next sentence prediction).
Interestingly, predictors based on inflected targets consistently outperform those based on uninflected targets, especially for the context-dependent models. This shows that morphological information can be quite valuable. Also, even for the largest sets of predictors, consisting of more than 3,200 variables per 80 data points, the degree of regularization appears to matter very little, indicating surprisingly small levels of overfitting.
In the case of Italian, the findings are somewhat different from those for English. Performance is roughly 10% lower than that for English. This is expected, given that perfect translation from Italian to English is impossible, and that the majority of predictors depend on this translation. The gaps in performance between predictors for inflected vs uninflected targets (Fig. 2c-d), and between the various classes of predictors (Fig. 2a-e), are also smaller. Moreover, the performance of contextdependent models can be increased to a small degree by adding behavioural norms and/or contextindependent models (Fig. 2f).
Our best models, as described in Figures 1 and 2, ranked first in both the English track (ρ = .83), and the Italian track (ρ = .75). The two correlations are smaller than those for the best models in the two figures, but this is likely to be an effect of distributional differences between the training set and the test set. 3
Our results suggest that a variety of approaches can be quite successfully employed in order to predict concreteness in context. The most effective predictors are those derived from context-dependent models (e.g., BERT), but relatively good results can be obtained also by using context-independent models (e.g., Skip-gram) and behavioural norms (e.g., ratings of semantic diversity).
Such an approach works very well for English, but less so for Italian, where the range of available predictors (i.e., pre-trained distributional models and large behavioural norms) is limited. One surprisingly effective solution to this problem is to simply translate the Italian stimuli into English, by relying on a neural machine translation system (e.g., MarianMT), and then make use of existing predictors for English. As an alternative to translating stimuli, it would be interesting to test whether comparable results can be obtained using multilingual versions of context-dependent models, such as BERT.
We would like to thank the anonymous reviewers, for their comments and suggestions, as well as the organizers of the competition, for their support. Behavioural norms (frequency, etc.) FastText (Common Crawl + Wikipedia) ConceptNet NumberBatch (ConceptNet + Skip-gram + GloVe) Concatenation of FastText and ConceptNet NumberBatch ALBERT (last hidden layer) AlBERTo (last hidden layer) Bart (last hidden layer) BERT (last hidden layer) GPT-2 (last hidden layer) RoBERTa (last hidden layer)
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In T. Linzen, G. Chrupała, & A. Alishahi (Eds.), Proceedings of the EMNLP Workshop BlackboxNLP (pp. 353-355). Stroudsburg, PA: ACL.