Concreteness of words has been measured and used in psycholinguistics already for decades. Recently, it is also used in retrieval and NLP tasks. For English a number of well known datasets has been established with average values for perceived concreteness. We give an overview of available datasets for German, their correlation and evaluate prediction algorithms for concreteness of German words. We show that these algorithms achieve similar results as for English datasets. Moreover, we show for all datasets there are no significant differences between a prediction model based on a regression model using word embeddings as features and a prediction algorithm based on word similarity according to the same embeddings.
A number of properties of words, mainly of semantic nature, have been studied and used in psycholinguistic research for decades. These properties often referred to as (affective) word norms, include concreteness, imagery1, age of acquisition, valence, and arousal. In the present work we will focus on concreteness. Friendly et al. (1982) define concrete words as words that “refer to tangible objects, materials or persons which can be easily perceived with the senses”. Similarly, Brysbaert et al. (2014) define concreteness as the degree to which the concept denoted by a word refers to a perceptible enCopyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
1Most authors seem to use the term imagery, while others also use imageability and visualness. In German the term Bildhaftigkeit is the most common one, while also Vorstellbarkeit is found. We will use imagery throughout this paper. tity, but found that subjects largely rated the haptic and visual experiences, even if they were explicitly asked to take into account experiences involving any senses.
Concreteness seems to play an important role in
human language processing
Traditionally, word norms are obtained by asking subjects to estimate the value for each property on a Likert scale. Recently, also various approaches have been proposed to predict the concreteness of words. On three different datasets we will test two algorithms that have given very good results for English data and compare the results in section 4 after we have discussed the most common approaches to predict word concreteness (section 2) and presented the concreteness data available for German (section 3). 2
We find basically three approaches to predict the concreteness of a word: (1) adopting the concreteness value from similar, related or neighboring words; (2) identifying a dimension in word embeddings that corresponds to concreteness; (3) training a regression model on features of words 2.1
Liu et al. (2014) predict values for imagery, a word norm that strongly correlates with concreteness, by using the values from synonyms and hypernyms found in WordNet.
Rabinovich et al. (2018) predict the concreteness
of words indirectly by assigning a concreteness
value to sentences in which a word occurs. The
concreteness value of a sentence is based on the
presence of seed words. The set of seed words is
constructed by selecting words with derivational
suffixes that are typical for highly abstract nouns.
The correlation between the predicted values and
the manual assigned values from various subsets
of the dataset from Brysbaert et al. (2014) and the
4,295 concreteness values2 from the MRC
(Medical Research Council) Psycholinguistic Database
Turney et al. (2011) compute the degree of
concreteness of a word as the sum of the similarities
between the word and n abstract paradigm words
minus the sum of the similarities between the word
and n concrete paradigm words. The paradigm
words are found as follows: first one concrete and
one abstract paradigm word are selected such that
the concreteness values for all words in the training
data, predicted by using the similarity with these
two words is maximized. Then a second concrete
and abstract word are added that again maximize
the correlation. This process is repeated until n
abstract and concrete words are found. Turney
et al. (2011) limit the selection to 20 abstract and
concrete paradigm words. Using half of the MRC
data for training and half for testing, they found
Spearman correlation coefficient of 0.81 between
predicted and observed concreteness values. To
compute the similarity between words they use
count based word embeddings of 1000 dimensions
trained on a 5 1010 word web corpus. The same
approach is followed by Köper and Schulte im Walde
(2016) to predict concreteness values for German
words using word vectors trained with word2vec
2The current version of MRC has concreteness values aggregated from different sources for 8,288 words. We assume that a previous version provided concreteness values for 4,295 words. 2.2
Rothe et al. (2016) try to find low-dimensional feature representations of words in which at least some dimensions correspond to interpretable properties of words. One of these dimensions is concreteness. For training and testing they use Google News embeddings and two subsets of frequent words from the norms of Brysbaert et al. (2014). For their test set of 8,694 frequent words they found a moderate correlation with the human judgments (Kendall’s t = 0; 623). Similarly, Hollis and Westbury (2016) looked which dimension of word embeddings correlate to one of the classical word norms. They found no direct correlations, but after reducing the number of dimensions for a set of words by applying Singular Value Decomposition, they found a strong correlation between one of the dimensions and concreteness. 2.3
Tanaka et al. (2013) train a regression model to predict concreteness values. As features they use a small number of manually constructed cooccurrence features, like co-occurrence with sense verbs. For training and evaluation they use a subset of 3,455 nouns from the MRC Database. Pearson’s correlation and Kendall’s t between the values from the database and their predictions are 0.688 and 0.508, respectively.
Paetzold and Specia (2016) train a regression model to predict four word norms, among which concreteness. Like many other studies, they use the data from the MRC database. As features they use word embeddings trained on a set of various large corpora and a number of word features extracted from WordNet. For each word norm they use half of the words to train the model and half of the words for evaluation. For concreteness they find a Pearson correlation coefficient of 0.869.
Ehara (2017) trains regression models to predict four word norms for Japanese and English words. As features they use word embeddings trained with word2vec and a probability distribution of words over topics found using Latent Dirichlet Allocation. They use a subset of 1,842 words from the MRC data, from which 1,342 words are used for training and 500 for testing. When both feature sets are trained on the British National Corpus (BNC) and used in combination, the best regression model gives a Pearson correlation of 0.87 and a Spearman correlation coefficient of 0.876 on the test data.
Ljubešic´ et al. (2018) used a regression model
as well with pre-trained fastText word embeddings
Though all studies use different data and
different versions of the MRC Psycholinguistic database,
use different splits and different number of folds
for cross validation and finally use different
correlation coefficients, all studies report very similar
results. The correlations that are found are all in
the range of correlations found between various
sets of concreteness values
Both, for English and German, various word norms with concreteness values have been created, though some are quite small and only available as printed supplements to older publications.
The dataset created by Baschek et al. (1977) and Wippich and Bredenkamp (1979) consists of 1698 words (800 nouns, 400 adjectives, 498 verbs) is one of the oldest and still one of the largest word norms for German. We will refer to this dataset as the Göttingen Word Norms. We removed 40 verbs containing an underscore, especially all reflexive verbs (e.g. sich_wünschen; to wish), from the data set. For number of words the experiment was repeated and two values are given. We only use the first value in these cases.
Lahl et al. (2009) collected values for 2,654 words using crowdsourcing to build a dataset called the Web Word Norms (WWN). For the WWN 3,907 subjects committed 190,212 ratings, each for at most 50 words. On average each word has 24 ratings. They used a 11-point scale were 0 stands for the most concrete and 10 for the least concrete judgment.
Kanske and Kotz (2010) collected ratings for valence, arousal and concreteness for 1000 nouns. This dataset is known as the Leipzig Affective Word Norms. Only nouns were used to reduce the variance other word classes would introduce. The experiment was done in 2006 with 32 native speaker. On two separate days the participants rated the words 3 times on a 9-point scale, each time for one of the three ratings. This was repeated 2 years later with two groups, one with 22 repeating participants from 2006 and a second with 32 fresh participants. The words were collected from the Duden dictionary and a previous word list by the same authors. Only 1 and 2-syllable words and no compound nouns were allowed.
The Berlin Word Norms
In order to have a larger dataset for German, providing more training data for supervised prediction algorithms, we created a merged dataset.
The overlap of the datasets is quite small (see Table 1), the correlation between the values for the overlapping parts, however, is high (around 0.9). Since the Leipzig Word Norm uses low values for concrete and high values for abstract words, the correlation between this and the other datasets is negative.
For the merged data set we use the 7 point scale
where 1 means abstract and 7 means concrete. We
do not simply rescale the values but use linear
regression on the overlapping parts such that the
values for the words in overlapping parts are as close
as possible. We take the values from the
Göttingen Word Norms as an anchor and transform the
other values using the slope and the intercept. The
transformed concreteness thus is defined as
C0 = a + bC
(1)
where C is the original value. For WWN a = 0:776
and b = 0:608 and for the Leipzig Word Norms
a = 7:39 and b = 0:540. Finally, we take the
average from all datasets if a word is present in
more than one source. The dataset thus offers
empirical concreteness values for 4,182 German
words. In Table 1 we see the high correlation
of the values in the merged data with those in
the original datasets. The merged dataset can be
downloaded from http://textmining.wp.
hs-hannover.de/datasets.html
For each dataset we use two methods to predict the
concreteness values in a five-fold cross validation
scheme. We compare the method of Turney et al.
(2011) described above in section 2. Following
Turney et al. (2011) and Köper and Schulte im Walde
(2016) we use 20 abstract and 20 concrete
prototype words. As a second method we use Support
Vector Regression
All test were done using 5-fold cross validation. We use stratified sampling for the Göttingen WN and the Merged dataset to ensure that each fold has the same number of nouns, verbs and adjectives. For the other dataset we use random splits. 5
The results for all datasets and both methods are given in Table 2. We see in general very high correlation values for all datasets and both methods. All correlation values are in a similar range as the correlations between the datasets.
We can make some interesting observations. The first remarkable fact is, that for all datasets there is no significant difference between the results from the method of Turney et al. (2011) and the regression model. As far as we know, these methods have not been compared directly before. This result is quite surprising, since there are many aspects of the meaning of a word that determine the word similarity. All of these aspects are used to find the similar words on which the concreteness prediction is based in the method of Turney et al. It has to be noted that the search of the prototype words in Turney’s method is extremely slow and not feasible for large datasets.
Furthermore, we see that our implementation of the method of Turney et al. gives slightly better results for WWN and the Leipzig Word Norms than the result found by Köper and Schulte im Walde (2016), who used a random split of the unification of those two datasets (0.844 and 0.891 vs. 0.825). Besides the possibility that they have chosen a disadvantageous split, we see two differences: In the first place we used different word embeddings to compute the word similarity. Secondly, they added concreteness values from English datasets with German translations to the training data. This is only helpful if concreteness is invariant under translation. This might be not the case. 6
Datasets with concreteness values for German are smaller and less easily accessible than those for English. One contribution of the present work is that we aggregated a consistent dataset with over 4000 concreteness ratings from three different sources.
A possibility to obtain more concreteness ratings is to train a model on available ratings and predict ratings for other words. We show that prediction methods that have been tested for English only before yield similar results for German. Moreover, we show that two of the best available methods that have not been compared on the same data before, yield similar results with no significant differences on 4 different data sets.
In near future we will extend the merged dataset with values from some smaller and older studies.