The present paper is a technical report of KonKretiKa, a system for computation of concreteness indexes of words in context, submitted to the English track of the CONcreTEXT shared task. We treat concreteness as a bimodal problem and compute the concreteness indexes using paradigms of concrete and abstract seed words and distributional semantic similarity. We also conduct sigmoid transformation to achieve greater similarity to the psycholinguistically attested data, and apply dynamic adjustment of static indexes for sentential context. One of the modifications of the presented system ranked third in the task, with rs = .6634 and r = .6685 against the gold standard.
This paper is a description of the system with the
working title KonKretiKa, which was submitted
to the English track of CONcreTEXT, the shared
task on evaluation of concreteness in context
KonKretiKa stems from our previous work on computation of such indexes for the purposes of metaphor identification.
Computationally obtained indexes of
concreteness are extensively explored in experiments for
automated metaphor identification. Application of
concreteness indexes to metaphor identification
relies on the assumptions made by the theories of
embodied and grounded cognition
An established method to compute the concreteness index of a word is to collect two sets of lexemes (‘seed lists’, or ‘paradigms’) consisting of abstract and concrete words – and to measure the lexical similarity between each word in the lexicon and each of the paradigm words.
The paradigm words are automatically selected
from the MRC Psycholinguistic Database
Machine Usable Dictionary
} where ′ is a linearly ordered set of (in ascending order)
(1) (2) (3)
The present work develops and extends the
method of
We demonstrate a method for evaluating
concreteness on English data; however, it can be
transferred to any other language provided that the
following
types
of resources
are
available:
(1) a lexicon with semantic
model; and (3) a relatively large wordlist containing lexemes with different frequencies of occurrence (ipm) in order to ensure the maximum possible variation in concreteness across the lexicon.
When analyzing the distribution of psycholinguistic concreteness ratings,
Computation of the standalone concreteness and
abstractness indexes is based on paradigm lists of
concrete and abstract words; we use the English
concrete and abstract paradigms from
The indexes of concreteness and abstractness
were computed using a Continuous Skip-Gram
model
As shown in Equations 1-3, to compute a concreteness or an abstractness index ( ) of a word, we measured semantic similarity (cosine distance) Sim between the vectors of this word and each word in the paradigm (concrete or abstract, respectively), and took the mean of the ten nearest semantic neighbors (NN).
In total, we computed concreteness and
abstractness indexes for approximately 23,000
English words (nouns, verbs, adjectives, and adverbs);
this lexicon was taken from the
The obtained computational sets of concreteness and abstractness indexes were normalized to the range [1, 7]1 in order to comply with the scale set by the CONcreTEXT shared task. In order to obtain an aggregate single-value index of a word, which would be representative of both its concreteness and abstractness, we subtracted the abstractness indexes from the concreteness indexes. m e t s y
S
on ) ita lau (tcn rm ep tx emResult (rs) Result (r) fso ty ten ts an oC jud rT a 2 1 2 1 0.5 0.5 0.8 0.8 where defines the slope of the function and defines the inflection point. Consequently, we can transform the sigmoid by changing the and coefficients.
In the submissions to the CONcreTEXT shared task, we experimented with two transformations of the raw KonKretiKa curve (Figure 2). In the first transformation, we applied a heuristically chosen combination of and which was intended to increase the slope and the curvature while preserving the S-shape of the sigmoid. The second transformation was intended to attain maximum resemblance of its shape to the Brysbaert et al. curve. We used grid search with different combinations of coefficients and to maximize the correlation between the two curves. During 2 The KonKretiKa ranking is available at: https://github.com/yubadryzlova/CONcreTEXT-2020 this fitting, only the values of the indexes are adjusted, while their initial ranks remain intact – thus, there is no data leakage from the psycholinguistic ranking. 2 2.3
Since the CONcreTEXT shared task requires that the concreteness indexes of target words be dynamically adjusted to their sentential context, the following heuristic was applied in the submitted KonKretiKa models. We computed the mean concreteness of all content words in the sentence (with the target word excluded) and adjusted the concreteness value of the target word accordingly. The adjusted index was computed as follows: = − ( ∗ ) where is the target word, is the raw index from the KonKretiKa ranking, is the mean concreteness of the sentence, and is the adjustment coefficient. In the models submitted to the CONcreTEXT shared task, we applied two heuristically defined coefficients: = 0.5 and = 0.8.
Thus, the four modifications of KonKretiKa submitted to the shared task were differentiated by the two parameters: the type of transformation and the contextual adjustment coefficient. 3
The parameters of the four modifications and their results are presented in Table 2 (along with the Baselines and the Leaders). The results indicate that systems with the lower coefficient of sentential adjustment (0.5) perform better than systems with the higher adjustment coefficient (0.8) irrespective of the type of sigmoid transformation; yet, the system with Type 2 (fitted to the psycholinguistic data) transformation somewhat outperforms the system with Type 1 (S-shaped) transformation.
The best of our modifications, KonKretiKa-3, demonstrated Spearman correlation with the gold standard rs = .6634 and Pearson correlation r = .6685, ranking our system third in the track, yet by a substantial margin behind the two winning system (with rs = .83313 and r = .83406 and rs = .78541 and r = .78682, respectively).
(dynamic)
BRY (static) rs =.743 r =.751 We carried out a post hoc analysis of the contextal adjustment coefficient (c) by using grid search to maximize the correlation between KonKretiKa (Type 2 transformation) and the gold standard. Moreover, we altered the scope of the context words for which the mean sentential concreteness (M) was computed – by taking 2-3 nearest semantic neighbors (either of any part of speech, or only nouns, or only verbs); this was done in order to reduce the possible noise from the words that are not semantically related to the target in the sentence. The change of the contextual scope did not lead to a substantial difference in the result. As for the contextual adjustment coefficient, the grid search showed that c = 0.32 – which is lower than the most efficient coefficient from our earlier submissions (c = 0.5 in KonKretiKa-3) – results in a slight increase of correlations: rs = .678 and r = .688.
A closer analysis of the test sentences suggests that contribution of contextual adjustment presumably may be increased by considering a broader context of a sentence – for instance, spanning over 1-3 adjacent sentences from the left and the right contexts; this option constitutes a possible direction for future work. 3.1
Pairwise correlations between the computational (KonKretiKa, KKK) and the psycholinguistic rankings (Brysbaert et al., BRY and the gold standard) are shown in Table 3. It can be seen that KKK better correlates with the BRY data than with the gold standard (rs = .743, r = .751 vs. rs = .663, r = .669, respectively). Presumably, such difference in the two correlations is due to the much larger size of the BRY lexicon. The correlation between the two psycholinguistic datasets word handmaiden (N) tire (V) bedrock (N) alarm (N) text (N) nonreactive (ADJ) temptingly (ADV) hail (V) stance (N) nudge (N) chasm (N)
7 6.18 (BRY vs. Gold) is rs = .755, r = .761, which is close to the correlation between KKK and BRY.
We undertook closer pairwise comparative analysis between two pairs of rankings: 1.
Static KonKretiKa indexes (the indexes after Type 2 sigmoid transformation, without contextual adjustment) vs. the Brysbaert et al. ranking (which is also static): approximately 23,000 words – nouns, verbs, adjectives, and adverbs (the two wordlists are identical).
Indexes of the target words from the CONcreTEXT test data as presented in the dynamic version of KonKretiKa (the sigmoid-transformed Type 2 indexes with contextual adjustment coefficient c = 0.32) vs. the Gold standard (where the target words are also ranked dynamically in context): 436 words – verbs and nouns.
The top residuals between the KonKretiKa and the Brysbaert et al. indexes are presented in Table 4. Analysis of these discrepancies suggests that most of them stem from polysemy and the differences between its representation in distributional semantic models and in psycholinguistic reality. Thus, distributional semantic models do not discriminate between various meanings of words; if occurrences of one of the meanings substantially outnumber the other meanings in discourse and, as a consequence, in the training corpus, the resulting vector reflects the more frequent meaning. e c n e t n e S t e d g r r o a T w d l o G
K K K f f i D 399 vision (N)
Check your < vision > to see if you are seeing blurry or double. 353 vision (N)
With retinal migraine, you may experience loss of < vision > in one eye and a headache that starts behind your eyes. 61 385 81 155 spirit (N) 324 pain (N) 6
Gin is an alcoholic < spirit > made from distilled grain or malt.
See your doctor if you are experiencing < pain > or discomfort. answer (N) war (N)
Be sure to write your final < answer > without the negative sign.
They have escaped from civil < war > in Liberia or Zimbabwe. answer (N)
Final < answers > for equations are considered wrong unless you have broken them down to their simplest form. 237 heart (N) 163 pain (N)
The < heart > pumps blood due to an internal electrical system. Take your medications to ease your physical < pain >. 176 agreement (N) 5.16 1.85 3.31 After signing the indemnification < agreement >, you can sign the legally binding bond agreement.
For example, the nearest semantic neighbors of the noun handmaiden in the distributional semantic model3 are: embodiment, personification, epitome, and paragon – associating this word with its abstract, metaphoric meaning ‘something that supports something else that is more important’4, whereas for speakers of English the other, concrete meaning ‘a woman who is someone’s servant’ apparently stands out as being more salient.
Similarly, among the nearest semantic neighbors of the noun chasm in the distributional semantic model are: disparity, schism, rich-poor divide, mistrust, (the) haves, divergence, antagonism, and inequality – indicating that the distributional vector of chasm is biased towards the abstract meaning of this word (‘a very big difference that separates one person or group from another’) rather than the concrete one (‘a very deep crack in rock or ice’), while human subjects see the latter meaning as more salient or prevalent.
As for nonreactive and temptingly, which are more concrete in the computational data, this could be explained by their perceived vagueness to human subjects, since these words do not have meanings that would be markedly juxtaposed to each other in terms of concreteness-abstractness – thus ranking them rather low in the psycholinguistic data. Meanwhile, the nearest semantic neighbors of temptingly in the distributional semantic model are: strappy sandal, capelet, knee-length skirt, enticingly, floral-print, high-heeled sandal, lace-trimmed, harem pants, and puffed sleeve – all rather concrete objects (or the properties of such objects).
The top residuals between KonKretiKa and the gold standard are shown in Table 5. The discrepancy between the abstract meaning of vision (‘the ability to think about and plan for the future, using intelligence and imagination, especially in politics and business’) and its concrete meaning (‘the ability to see’) can also be attributed to the differences between representation of meanings in distributional semantic models and in psycholinguistic reality – the reason already discussed above. Thus, the nearest distributional semantic neighbors of vision are: worldview, ideal, visionary, thinking, perspective, idea, dream, and blueprint – rather than terms related to eyesight.
The noun spirit in Table 5 (Sentence 155) is
used in the sense of ‘strong alcoholic drink’.
However, its nearest neighbors in the distributional
semantic model are ethos, ideal, idealism, tradition,
essence, enthusiasm, passion, faith, chivalric,
zeal, credo, and compassion – indicating that the
meaning ‘your attitude to life or to other people’
3 Continuous Skip-Gram model
As for the noun war, its nearest neighbors in the distributional semantic model are conflict, warfare, invasion, 1991-95 Serbo-Croatian, IsraelHezbollah, genocide, Bosnia war, Jehad, civilwar, Croatia war, Cold War, Iran-Iraq, wartime, Vietnam-like, etc. – that is, rather abstract concepts. The only more concrete words referring to physical combat action that occur in the distributional semantic neighborhood of war are battlefield and bloodshed, but this is not enough to outweigh the abstract terms. Thus, the distributional semantic model models warfare in terms of abstract rather than concrete (such as names of weapons, military equipment, military personnel, etc.) concepts. As a result, military action is not sufficiently juxtaposed to the metaphoric meaning of war as ‘a situation in which two people or groups of people fight, argue, or are extremely unpleasant to each other’.
In the case of answer and agreement, their nearest distributional semantic neighbors in the model are fairly abstract concepts: explanation, answer, reply, solution, unanswerable, query, TV-talkback answer, question, and yes (for answer), and accord, pact, deal, treaty, initial, negotiation, memorandum, compromise, and negotiate (for agreement). Meanwhile, human subjects rank answer and agreement rather high in concreteness; presumably, this is a consequence of conflating the mental representations of the action of answering / reaching an agreement with their two modes – the spoken and the written, i.e. with the physical actions of speaking and writing. This conflation is not reflected in discourse – it largely exists in the mental representations of answer and agreement and, therefore, is not very distinguishable on the level of linguistic representation.
Of interest are the cases of heart and pain, which have much lower concreteness in KonKretiKa than in the gold standard sentences where these words are used in their physical, concrete meanings. The nearest distributional semantic neighbors of heart are heart-related, coronary artery, kidney, liver, lung, arrhythmia, cardiac, angina, and aneurism. The nearest neighbors of pain are discomfort, ache, agony, tingling sensation, numbness, soreness, menstrual cramp, lightheadedness, stiffness, nausea, and arthritis. It would be quite expected for such semantic neighborhood to entitle heart and pain to higher concreteness values than what they receive in KonKretiKa. A more in-depth analysis into this contradiction revealed that it stems from the vulnerability in the semantic composition of the concrete paradigm which was used to compute the raw indexes (see Table 1). The words of this paradigm belong to the two major semantic classes – living organisms (animals and plants) and manmade artifacts. The class of words denoting human beings was intentionally excluded when the paradigm was compiled on the grounds that such nouns tend to indicate abstract social roles rather than physical humans. As a consequence, physical organic objects such as body parts and organs, or physical sensations and physiological conditions received non-uniform indexes in KonKretiKa: those that refer to humans as well as to animals (e.g. in veterinary or gastronomic discourse) ranked rather high in concreteness: e.g. liver (6.6), pancreas (6.4), foot (6.3), encephalitis (6.25), kidney (6.25), entrails (6.05), tummy (5.92), womb (5.6) – whereas those that tend to be primarily associated with humans received lower indexes, e.g. heart (2.63), heartburn (2.57), scar (2.53), nausea (2.5), headache (1.61), distress (1.5), pain (1.21), queasiness (1.12), etc.
Thus, comparison of the KonKretiKa
computational indexes with the psycholinguistic data of
CONcreTEXT allowed us to detect a potential
shortcoming in our approach to the design of the
concrete paradigm. As was noted in previous
study
We presented KonKretiKa system for computing concreteness indexes of English words in context; the system was submitted to the English track of the CONcreTEXT shared task. The best modification of KonKretiKa ranked third in the task, with rs = .6634 and r = .6685 against the gold standard. We treat concreteness as a bimodal problem and use paradigm lists of concrete and abstract words to compute two indexes for each word, that of concreteness and of abstractness. The single aggregate index indicative of both the word’s concreteness and abstractness is computed as the function of the two respective indexes. The set of raw aggregate indexes is transformed using sigmoid transformation to increase the variance and to attain greater similarity to the psycholinguistic data. To dynamically adjust the concreteness indexes to the context, we apply an adjustment coefficient. Post hoc analysis of the adjustment coefficient indicates that lower coefficients lead to better performance. We hypothesize that the contribution of the adjustment coefficient could be increased by expanding the scope of the context, for example, by considering one or more sentences from the left and the right contexts of the target sentence. According to our analysis, the main source of divergence between the computational and the psycholinguistic indexes lies in the different representation, or salience, of word meanings in distributional semantic models and in psycholinguistic reality. Besides, analysis of divergences between the computational and the psycholinguistic rankings prompted us a potential direction for reducing the bias in composition of the concreteness paradigm, which can be overcome by diversifying the paradigm.