English. This paper describes several approaches to the automatic rating of the concreteness of concepts in context, to approach the EVALITA 2020 “CONcreTEXT” task. Our systems focus on the interplay between words and their surrounding context by (i) exploiting annotated resources, (ii) using BERT masking to find potential substitutes of the target in specific contexts and measuring their average similarity with concrete and abstract centroids, and (iii) automatically generating labelled datasets to fine tune transformer models for regression. All the approaches have been tested both on English and Italian data. Both the best systems for each language ranked second in the task.
The characterization of the conceptual concreteness of a word in context is a task that requires a level of analysis that goes well beyond the identification of the properties of the referent (or denotation) of the target word. The overall linguistic context should be taken into consideration as well, along with its interaction with the target word. Even addressed in the most simplistic way, i.e. ignoring the context and focusing solely on the target word in isolation, it is a daunting task in which the machine is asked to draw inferences on a level of semantic representation that the speaker builds by integrating experiential and linguistic information (Vigliocco et al., 2009). Moreover, figurative
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). uses of words (e.g., metaphors) determine important shifts in their concreteness values. For example, the word head in the sentence Take your safety pins and attach one card to the head of your bed can be considered as highly concrete, as it describes a physical object. Conversely, the same word in the sentence The pope is also head of the world’s smallest sovereign state, The Vatican has a more abstract meaning, denoting the title of a person. Similarly the verb fly is more concrete in the sentence The plane flies in the sky than in the metaphorical sentence Time flies.
The context-sensitive nature of word
concreteness is one of the key elements that make its
identification very interesting and complex from a
Natural Language Processing (NLP) perspective
As it is common for other NLP and NLP-related
tasks and topics, an invaluable source of
knowledge that can be used both to train models and to
gain some insights on the nuances of the
problem itself can be found in the psycho-linguistic
tradition, and especially in those normative
studies built to analyze collections of human-elicited
concreteness judgements
The CONcreTEXT task
In order to address this task, we propose three families of distributional semantic methods relying on several existing concreteness norms. Our general approach revolves around the idea that taking into account both the context and the target word, as well as words that play a similar role in the same context, may help us in overcoming limitations due to scarce training data, and may prove beneficial for predicting more accurate ratings.
The paper is organized as follows: Section 2 describes the proposed approach based on both supervised and unsupervised methods. Section 3 presents the results, which are discussed in Section 4. Finally, Section 5 draws some conclusions. 2
We propose three different “CAPISCO” (for CA’ Foscari and PISa COncretext project) approaches for predicting the concreteness of a word in a given context of occurrence. Each method exploits the assumption that the concreteness of a word is influenced by its surrounding context. We explore both unsupervised and supervised techniques. In fact, two such approaches are unsupervised, and exploit either pre-trained word embeddings, or pre-trained transformer language models, while the third method is supervised: NON-CAPISCO – the concreteness of the target word is modelled as a function of its concreteness value in isolation and of the average concreteness of its surrounding context.
CAPISCO-CENTROIDS – the concreteness of the target word is estimated as a function of the concreteness values of its closer synonyms according to a pre-trained transformer language model. Crucially, the concreteness ratings of the target synonyms are estimated by computing their distance from two reference points in the distributional space corresponding to the centroids of the highly concrete and highly abstract terms.
gressor is trained to predict concreteness ratings. Specifically, we fine-tune a transformer model to predict the target concreteness of the sentence, exploiting the available dataset augmented with new data automatically generated from several different norms of concreteness. 2.1
The NON-CAPISCO system is rather simple, both conceptually and implementation-wise. It is based on a minor change in the baseline proposed by the task organizers.
The task baseline is computed by averaging
over the concreteness ratings of all the words in
the sentence. Ratings are obtained from the norms
by Montefinese et al. (2013) for Italian, and from
those by Brysbaert et al. (2013) for English.
Words missing from these resources are replaced
by their closest neighbor among those for which
human ratings are available. Closest neighbors are
identified using fastText
Crucially, this baseline takes into account the concreteness rating of the target word, but it has the same weight as all the other words in the sentence on the final prediction. On the other hand, we noticed that a simple method based solely on the concreteness score of the target word achieves a performance of 0:69 for Italian and 0:69 for English, much higher than that of the task baseline. This led us to surmise that, at least in the task dataset, the concreteness of the word in context is strongly affected by its value in isolation.
The NON-CAPISCO method gives more weight to the target word, by multiplying its concreteness rating for the mean concreteness of the whole sentence. On the trial dataset this combined score obtained a Spearman correlation of 0:73 for Italian and 0:73 for English. 2.2
The CAPISCO-CENTROIDS approach is based on
the assumption that semantically similar words are
expected to be similarly rated for concreteness and
that, conversely, words associated with highly
different concreteness scores should be placed far
away from each other in semantic space. This
assumption is driven by the fact that concrete (or
abstract) senses are typically found in co-occurrence
with other concrete (or abstract) ones
The first step of this method consisted in the building of two reference vectors: one representing the prototypical abstract concept; the other representing the prototypical concrete concept. To this end, we first identified highly concrete and highly abstract terms from two available resources: the Brysbaert et al. (2013) norms for English and the Della Rosa et al. (2010) norms for Italian. The latter has been preferred to more comprehensive alternatives, like the Montefinese et al. (2013) norms, due to its covering of a significant set of highly polarized words.
For each resource, the clusters of most
concrete and abstract words were identified by
fitting a mixture-of-Gaussian model on the human
judgments, and choosing the most distant
clusters. We used the expectation-maximization
algorithm available in scikit-learn.1 To set the
number of clusters and type of covariance, we chose
the pair that minimized the Bayesian information
criterion. After identifying the groups of most
polarized words in our reference norm, we used
English and Italian pre-trained word embeddings
from fastText
However, predicting the concreteness of a 1https://scikit-learn.org/ target word solely based on its proximity with the centroids could be biased by its semantic relatedness with the words used for building the centroids. To smooth this bias, the final score for a given target word was calculated as the average of the similarities of its potential lexical substitutes. BERT was used to identify the substitutes of each target word in context. Operationally, we masked the target word in each sentence, and asked the model to predict the 50 most likely words that may fill the masked token, which is likely to include the target itself. After several experiments, we chose 50 words as they gave us the best overall results. We can argue that it is probably the best trade-off between number of neighbors and their actual similarity with the target word. We used the bert-base-uncased model for English, and the bert-base-italian-xxl-uncased model for Italian. Table 1 reports some potential substitutes of the target word in the sentence.
TARGET lawsuit love
MASKED SENT.
In a typical [MASK] , the defendant frequently brings a motion [...].
Give your friends [MASK] , positivity , and compliments .
FILLERS case trial proceeding attention kindness respect
To avoid noise due to the fact that sometimes BERT predicts a token with a different syntactic role, all the fillers with a different Part-of-Speech (PoS) tag than that of the target word were filtered out. To this end, we PoS-tagged all the sentences produced by replacing the target word and kept only those with the same PoS sequence of the original sentence. This way, we obtained, for each target word, a list of lexical substitutes in a particular context. Each substitute was assigned a concreteness score based on its proximity to the two prototypical vectors. More specifically, we computed the concreteness of a word as the absolute value of the difference between its cosine with the concrete centroid and its cosine with the abstract one normalized on a 1-7 scale. Finally, each target word was assigned with a concreteness value obtained by averaging the concreteness of its substitutes. 2.3
The CAPISCO-TRANSFORMER system addresses
the problem from a supervised perspective. The
system is based on the BERT Transformer
architecture
On the one hand, we generated potential substitutes of the target word with the same techniques used in Section 2.2. In this case, we generated three sentences containing as target word the three most likely lexical substitutes of the original one. Such new target words were assigned the same concreteness rating of the original one, modified by a small random value in the range [-0.2,0.2], to avoid repetition of target values for the training set derived from the gold data.
On the other hand, we extended the dataset
with new sentences which were assigned the
concreteness scores found in the concreteness norm.
For English, we extracted from the BNC corpus
We proposed three different approaches for the
estimation of concreteness. The performances
obtained for each model for the Italian language and
for the English language are presented
respectively in Tables 2 and 3. Given the absence of
a training set, we decided to give more
emphasis to the unsupervised method (NON-CAPISCO)
based on the concreteness of target words and of
the surrounding context. It is clear that the results
of this method are highly influenced by the
annotated resources exploited to infer the concreteness.
The results revealed that while for English such
approach was quite effective, for Italian it is not,
probably due to the smaller dimension and
quality of the resources taken into consideration. In
fact, if we look at the ranking of our models in the
two languages, the results are reversed. On the one
hand, the best CAPISCO approach for English is
the NON-CAPISCO system, in which concreteness
ratings are obtained from Brysbaert et al. (2013).
Such resource counts ratings for about 40
thousand of English lemmas that have been annotated
for several variables. On the other hand, the Italian
resources
In light of the reported results, several interesting observations can be made. For both languages, our best-performing model ranked secThe whole extended dataset is then used to fine2https://huggingface.co ond overall. However, we can notice how neither numerical results nor the ranking of the system are consistent across languages. For English, the best performing system is NON-CAPISCO. The system strongly outperforms both baselines and the other two methods. We must also note that both CAPISCO-CENTROIDS and CAPISCOTRANSFORMER perform worse than one of the two baselines. On the other hand, for Italian, CAPISCO-TRANSFORMER performed best, closely followed by CAPISCO-CENTROIDS. Both outperform the NON-CAPISCO approach, and all three systems perform better than the baselines. This discrepancy may be due to several key aspects concerning both the resources used as well as some crucial differences among trial and test samples of the dataset.
We can identify several key differences among English and Italian resources that may justify such drastically different performances. While for English a comprehensive resource with 40,000 words is available, both resources for Italian are orders of magnitude smaller. In addition to this, especially for ratings contained in Montefinese et al. (2013), the distribution is unbalanced towards mid-range and high values of concreteness, while ratings for Brysbaert et al. (2013) are more evenly distributed across the spectrum. For the NON-CAPISCO system, this may lead to poor performances since for the system is more difficult to predict higher values for the Italian dataset. While predictions for the English model closely follow the distribution of ratings in the test set, predictions for Italian are unbalanced towards lower values.
On the contrary, for the CAPISCO-CENTROIDS system, this has the opposite effect. In fact, given that it is more difficult to isolate extremely abstract and extremely concrete terms, centroids built from Italian resources are closer one another, and thus prediction based on the difference between distances to the centroids almost always fall in the middle of the range, while for English the same approach has the effect of yielding results that are mostly close to the lower-end of the spectrum. This, in turn, has the effect of seemingly improving performances for Italian, because too high and too low prediction balance each other, while errors for English are more pronounced.
Finally, for the CAPISCO-TRANSFORMER system, it may be possible that the fact that English norms contains more high frequency words, may hinder the generalization capabilities of the model. In fact, if such words are found in very different sentences, all such sentences are assigned very similar concreteness scores and the predictions are biased towards certain values for many different sentences. Therefore, the distribution of predictions follow the same tripartite distribution of the sampled words in terms of concreteness.
Finally, we must point out that the distribution of ratings in the trial and test set are rather different, as shown in Figure 1. This may have hindered our judgment on the quality of all proposed systems, both unsupervised and supervised. 5
The models proposed are based on both supervised and unsupervised approaches. The choice was motivated by the fact that the trial dataset proposed for the task is too small to effectively train supervised learning models on it. The key assumption that drove the development is that the concreteness of a word is influenced by its surrounding context, as claimed by the task organizers as well. The best CAPISCO systems for both Italian and English ranked second in the CONcreTEXT task despite the fact that results differ a lot in terms of absolute performances and used method. For Italian, the best CAPISCO system is based on Transformers and reaches a Spearman correlation of 0:625 with gold data. The best CAPISCO model for English, on the contrary, is unsupervised and reaches a Spearman correlation with gold data of 0:785.
In the future, we plan to perform some additional hyper-parameter tuning on the models. Moreover, we would like to test this approach in similar tasks (e.g. predicting abstractness). We are confident that by exploiting the dynamic selection of training data in addition to an annotated dataset such as the test dataset provided by the task organizers would improve the results of our systems, and in particular of the transformers-based one.
We gratefully acknowledge the support of the NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Gabriella Vigliocco, Lotte Meteyard, Mark Andrews, and Stavroula Kousta. 2009. Toward a theory of semantic representation. Language and Cognition, 1(2):219–247.