In this paper, we present a novel application of a computational model of word meaning to capture human judgments of the linguistic properties of metaphoricity, familiarity, and meaningfulness. We present data gathered from human subjects regarding their ratings of these properties over a set of word pairs speci cally designed to exhibit varying degrees of metaphoricity. We then investigate whether these properties can be measured in terms of geometric features of a model of distributional lexical semantics. We compare the performance of two models, our own Concept Discovery Model which dynamically constructs context-sensitive subspaces, and a state-of-the-art static distributional semantic model, and nd that our dynamic model performs signi cantly better in its measurement of metaphoricity.
In this study, we investigate whether computational models of lexical meaning might help explain human comprehension of metaphors. We examine several alternative models, based on language statistics in a large general collection of English, to see if they capture relations between words which correlate with human judgments of metaphoricity.
Psycholinguistic studies Research on metaphor in human participants has attempted to clarify the mechanisms underlying understanding metaphoric language. One of the earliest approaches, the standard pragmatic model, stipulates that the literal meaning of a metaphoric sentence needs to be rejected before the
gurative meaning is generated [
Recently, a number of electrophysiological (EEG) studies investigating
metaphor comprehension have been reported in which the N400 component, a
negativegoing wave observed between 300 and 500ms after the presentation of the critical
stimulus, has received considerable attention. Larger N400 amplitudes have been
observed in the processing of metaphoric as compared to literal sentences, with
no di erences in component latency (i.e., the time window within which the
effect is observed) or scalp distribution (i.e., the sites on the scalp over which the
e ect is present) (e.g., [
At the same time, comparable latency and scalp distribution of the
component might be indicative of the involvement of similar mechanisms in literal and
metaphoric language comprehension. Interestingly, di erences have been found
between conventional and novel metaphors, with the N400 amplitudes for
conventional metaphors falling in-between those for novel metaphoric and literal
utterances [
One of the approaches that has attempted to elucidate metaphor
comprehension mechanisms is the structure mapping model and its descendant, the career
of metaphor model, which stipulate that the same mechanisms are involved in
the comprehension of literal comparisons, similes, and metaphors [
Computational studies Computational approaches to metaphor have generally
focused on a combination of pattern matching and hand coded information
processing [
Moving towards data-driven approaches, a model for metaphor
comprehension has been described that employs latent semantic analysis, a statistical
technique for building spaces of word similarity, to the selection of the salient
features tra cked between a metaphoric source and target [
The computational component of the work presented here broadly falls within
the paradigm of the distributional hypothesis, which holds that \words that occur
in similar contexts have similar meanings," [26, p. 148]. The general
methodology of distributional semantics involves the traversal of large scale textual
corpora in order to build spaces of word-vectors where the proximity of two
vectors re ects the tendency for those two words to be observed co-occurring
with similar terms [
In the study presented here, we will in particular be comparing Word2Vec
[
Our objective in this study is to explore the ways in which geometric models of word meaning can capture the perception of metaphor, and in particular can measure the degree to which two-word phrases are perceived as being metaphorical. To do so, we compare words' relations in the geometric model with human judgments of metaphoricity, via a set of empirically-derived normative data. Note that while data were collected for three types of norming measures (metaphoricity, meaningfulness, and familiarity), the principal aim of the computational work is to model the perception of metaphoricity, that is, to discover meaningful subspaces that re ect the extent to which a two-term expression is perceived as being metaphorical. 2.1
The materials were collected for an ERP study, which investigated metaphor
comprehension in bilinguals [
Prior to the ERP experiment, ve normative studies were carried out to ensure the word pairs fell within the following three categories: novel metaphors (e.g., to harvest courage), conventional metaphors (e.g., to gather courage), and literal expressions (e.g., to experience courage). Based on the results of the normative studies, the nal set of 228 English verb-noun word dyads (76 in each category) was selected for the purpose of the current study. Out of the ve normative studies, four will be reported here. The statistical analyses consisted of mixed-design analyses of variance (ANOVAs), with utterance type as a within-subject factor and survey block as a between-subject factor. No main e ect of survey block was observed. Signi cance values for the pairwise comparisons were corrected for multiple comparisons using the Bonferroni correction. When Mauchlys tests showed that the assumption of sphericity was violated, the Greenhouse-Geisser correction was applied. In such cases, the original degrees of freedom are reported with the corrected p value. The demographic data for the participants of the four normative studies are presented in Table 1. Cloze probability Because reduced N400 amplitudes have been observed in relation to expected as compared to unexpected words, a cloze probability test was performed prior to the ERP study to ensure the second word in a given word dyad was not highly anticipated by the participants of the ERP experiment. Each participant of the cloze probability test received the rst word of a given word pair, and was asked to provide the second word, so that the two words would make a meanigful expression. Due to the length of the test, all word pairs were divided into four blocks, so that each word was completed by 35 participants. If a given word pair was observed in the cloze probability test more than 3 times, the word pair was excluded from the nal set and replaced with a new one. This procedure was repeated until the cloze probability for word pairs in all categories did not exceed 8%.
Meaningfulness In order to assess the meaningfulness of the stimuli, participants were asked to rate how meaningful a given word pair was on a scale from 1 (totally meaningless) to 7 (totally meaningful). The set of 228 word dyads was divided into four survey blocks in order to avoid the repetition of the target word within the same survey. Additionally, 76 meaningless word pairs were included in this normative study. The results revealed a main e ect of utterance type, [F(3; 387) = 1611:54; p < :001; = :799; p2 = :93]. Pairwise comparisons revealed that literal word pairs were assessed as more meaningful (M = 5:99; SE = :05) than conventional metaphors (M = 5:17; SE = :06) (p < :001), and conventional metaphors were assessed as more meaningful than novel metaphors (M = 4:09; SE = :08)(p < :001).
Familiarity Familiarity of each word pair was assessed in another normative study. Participants were asked to decide how often they had encountered the presented word pairs on a scale from 1 (very rarely) to 7 (very frequently). The set of 228 word dyads was divided into three survey blocks in order to avoid the repetition of the target word within the same survey. Again, a main e ect of utterance type was found, [F (2; 296) = 470:97; p < :001; = :801; p2 = :83]. Pairwise comparisons showed that novel metaphors (M = 2:15; SE = :07) were rated as less familiar than conventional metaphors (M = 2:97; SE = :08); (p < :001), with literal expressions being most familiar (M = 3:85; SE = :09); (p < :001). Furthermore, conventional metaphors were less familiar than literal word dyads, (p < :001). It is crucial to note that although di erences were observed between categories, all word pairs were relatively unfamiliar. This is visible in the mean score for literal word pairs, which are most familiar of all three categories, but at the same time relatively low in familiarity (below 4 on a scale where 6 and 7 represent very familiar items). The reason why familiarity was low in all three categories is the same as for the cloze probability test, i.e., that we intentionally excluded highly probable combinations.
Metaphoricity In order to assess the metaphoricity of the word pairs, participants were asked to decide how metaphoric a given word dyad was on a scale from 1 (very literal) to 7 (very metaphoric). The set of 228 word dyads was again divided into three survey blocks in order to avoid the repetition of the target word within the same survey. The results revealed a main e ect of utterance type, [F (2; 198) = 588:82; p < :001; = :738; p2 = :86]. Pairwise comparisons con rmed that novel metaphors (M = 5.00, SE = .06) were rated as more metaphoric than conventional metaphors (M = 3.98, SE = .06), (p < :001), and conventional metaphors were rated as more metaphoric than literal utterances (M = 2:74; SE = :07); (p < :001). 3
In order to computationally model human judgment of the conceptual features of word dyads, we construct distributional semantic spaces where the proximity of word-vectors relates to their semantic similarity, and then explore the geometry of these spaces for ways of mapping relationships between words that are productive with regard to such conceptual, cognitive phenomena as metaphor. Speci cally, we compare two di erent distributional semantic models to assess the di erence in performance between a model that might be described as static, such as the one outlined in Section 3.1, versus one that is contextually dynamic, as is the intent with our own model as explained in Section 3.2.
For both our static and dynamic models, we train vectors on the English language version of Wikipedia. For the purpose of capturing word co-occurrences, we focus only on the descriptive content of Wikipedia pages, ignoring headers, lists, captions, and the like. Considering only sentences at least ve words in length, we strip the corpus of punctuation, remove articles (the, a, and and ), and remove parenthetical phrases, resulting in an overall corpus of approximately 7.5 million word types and 1.1 billion word tokens. For the construction of both models, we consider context windows of ve words on either side of a target word, treating sentence endings as contextual boundaries as well. We take the 200,000 most frequently occurring words in the corpus as the vocabulary for both models, constructing one word-vector for each word in the vocabulary.
As our measure of semantic relatedness between two words, we take the cosine
similarity between their corresponding word-vectors, in line with a number of
other contemporary distributional semantic models [
As our primary point of comparison in this study, we use the Word2Vec
distributional semantic model [
Two di erent network architectures have been reported in the literature; here,
we employ the Skip-gram architecture, consisting of a two-layer neural network
which learns to predict context terms based on an input word, as this approach
has been reported as performing particularly well on semantically oriented tasks
[
We compare the performance of the established Word2Vec semantic model with
the output of a distributional semantic model which dynamically interprets
input text to project context-sensitive subspaces from a sparse, high-dimensional
base space. As described in detail elsewhere [
We build a base space of roughly 7.5 million dimensions, corresponding to the number of word types in Wikipedia. From this base space, we dynamically pick 200-dimensional subspaces, speci c to each word-pair in the study. Three di erent methodologies for projecting subspaces will be discussed below, but in each case, the input used to determine the projection is simply the pair of words involved in a potentially metaphoric dyad, and the projection is based on an analysis of the respective values of these inputs along any given dimension. The intuition behind this methodology is that subspaces consisting of dimensions which are mutually salient for both components of a dyad will capture something of the semantic context in which the candidate metaphor might be meaningful. Within these subspaces, as with Word2Vec, we assess the relationship between the words in a word pair in terms of the cosine similarity between their two corresponding word-vectors. One of the primary considerations in the application of this model is therefore the method for selecting these subspaces. Discovering conceptually relevant spaces We experimented with three different techniques for choosing subspaces from our base space, in each case focusing on the relationship between the target and source word in each dyad. cut pierce cause pollution 60 40 20 0 pollution pollution pollution
pollution pollution pierce cut cause 60 40 20 0 cause cut pierce 0 10 20 30 40 50 60 (a) Noun-Only Subspaces 0 10 20 30 40 50 (b) Joint Subspaces
0 20 40 60 (c) Independent Subspaces Fig. 1. Here three di erent types of subspaces are presented, with three expressions involving the word \pollution" superimposed on each two-dimensional projection: the literal phrase \cause pollution", the conventional metaphor \cut pollution", and the novel metaphor \pierce pollution". Angles between the vectors for both words in each pair are measured for correlation with human judgments of the metaphoricity of each expression. Angles and vector lengths from the 200 dimensional subspaces we analysed are preserved in these projections. The word-vector for pollution is the same for all three versions of the noun-only space, since the other terms have no in uence on the selection of dimensions here.
{ Noun-Only Subspaces: the subspace is selected based only on associations with the target term: we take the 200 dimensions with the highest PMI value (as expressed in Equation 1) for the target (i.e., noun) in a given dyad. { Joint Subspaces: selection is based on associations shared by the source and target terms: we select the 200 dimensions with the highest average PMI for both target and source terms in each dyad. { Independent Subspaces: selection is based on independent associations with the source term and the target term, such that we select the 100 terms with the highest PMI values for the source term and the target term independently, and then merge these two sets of dimensions into a single 200 dimensional space.
An example of how a target and source dyads manifest in these three subspaces is shown in Fig. 1.
For both Word2Vec and CDM, cosine similarity values were computed for each word pair used in the behavioral study described above. Because the human ratings are the ground truth in this instance, Cosine Similarity is the dependent variable in each of the multiple regressions reported below. The three measures provided by human raters { Metaphoricity, Meaningfulness, and Familiarity { are the independent variables used in the analyses. The general aim is to identify which aspects of the word pairs (in terms of perceived metaphoricity, etc) are captured by cosine similarity in a given space. We also explore which type of subspace is best able to capture metaphor alone (that is, which space accounts for the most variability in human responses for metaphoricity).
We rst report the results for Word2Vec, which are then used as a baseline against which to compare the results of our CDM model. The results for CDM are broken down by the type of underlying subspace. 4.1
The results of the multiple regression analysis for Word2Vec indicated that the predictors accounted for a signi cant proportion of the variance in Cosine Similarity scores [R2 = :249, F (3; 224) = 24:81, p < :001]. Metaphoricity signi cantly predicted Cosine Similarity scores, [ = 0:25, t(224) = 3:09, p < :01], as did Familiarity [ = 0:22, t(224) = 2:65, p < :01]. Low values of Metaphoricity tend to yield high values of Cosine Similarity, and low values of Familiarity tend to yield low values of Cosine Similarity. Meaningfulness was not a signi cant predictor in the regression.
First, these results con rm that Cosine Similarity does, in fact, capture more information than simply similarity about a given pair of terms: both Metaphoricity and Familiarity help to account for the variance in Cosine Similarity values for Word2Vec. Second, these results provide a standard by which we are able to compare the Conceptual Discovery Model's performance. 4.2
Unlike Word2Vec, the CDM model a ords the discovery of di erent kinds of geometrically-de ned subspaces. The crucial advantage of the CDM model is its ability to project a context-speci c subspace geared towards capturing the semantics of situations in which a metaphor can be meaningfully applied. As such, our objective is to compare the performance of Cosine Similarity scores for detecting properties of metaphors using di erent techniques for constructing context-speci c subspaces, in particular, the Noun-only, Joint, and Independent methods described in Section 3.2. The same multiple regression analysis as above was performed for these three CDM model con gurations. Finally, the relationship between Cosine Similarity and Metaphoricity ratings is explored in more depth for the best performing model.
Noun-only subspaces indicates that the predictors account for a limited proportion of the variance of Cosine Similarity scores, [R2 = :108, F (3; 224) = 9:04, p < :01]. Metaphoricity signi cantly predicts Cosine Similarity scores, [ = 0:30, t(224) = 3:48, p < :01], where higher Metaphoricity ratings are associated with lower values of cosine similarity. The regression analysis for the Joint subspace does not yield any signi cant results, with R2 = :016, F (3; 224) = 1:19, p = n:s:
Given the poor performance of the above models (which is quanti ed in
the low R2 for all four model con gurations) compared with Word2Vec, we
conclude that neither the Noun-only subspace nor the Joint subspace are suitable
for capturing perceived metaphoricity of word pairs. Previous computational
approaches to metaphor generation and interpretation [
Of the three multiple regressions reported here, this analysis accounts for the most variability in cosine similarity values, with an R2 of :271, which is signi cantly higher than the regression for Word2Vec (where the R2 was :249). Note that, interestingly, Familiarity is not signi cant in this analysis. The interpretation of this nding is discussed below in the General Discussion.
To visualize how the relationship between Cosine Similarity and Metaphoricity varies by utterance type (Conventional metaphor, Novel metaphor, and Literal word pair), a correlation analysis is shown in Fig. 2 for this best-performing model, with utterance types demarcated. Metaphoricity is inversely correlated with Cosine Similarity [r = :50; t = 32:49; p < :001], such that word pairs rated as highly metaphorical tend to have low Cosine Similarity values. Novel metaphor word pairs, which participants rated highest for Metaphoricity, generally have low Cosine Similarity scores. This trend is shared by the Conventional metaphor word pairs, although the Metaphoricity scores tend to be slightly lower (this is con rmed by examining the averages of these two utterance types). Finally, Literal word pairs, which garnered the lowest ratings for Metaphoricity, tend to have slightly higher Cosine Similarity values overall.
In sum, whereas Cosine Similarity in Word2Vec is correlated with both Metaphoricity and Familiarity, the exibility of our CDM model (speci cally, the ability of our model to discover speci c, conceptually-relevant spaces) allows us to discover a space in which Cosine Similarity re ects only the metaphorical aspects of word pairs. 5
The set of results for Word2Vec and CDM o ers important insight for the computational simulation of metaphoric language use. Firstly, for both Word2Vec and the Independent subspaces version of the CDM model, human ratings of Metaphoricity were able to account for a signi cant proportion of the variability in Cosine Similarity scores. Although only 24-27% of the variance was explained, it is important to consider that utterance type (which signi cantly in uenced ratings) was not included in the statistical analyses above, because 1) the present research investigates the extent to which cosine similarity (alone) accounts for perceived metaphoricity between two terms, and 2) information about utterance type would not be available when applying our model in other contexts.
The performance of Word2Vec was used as a standard with which to compare the three variants of our CDM model. Both Familiarity and Metaphoricity were signi cant predictors of Cosine Similarity for Word2Vec, but for CDM, we were able to nd a subspace that captures solely the perceived Metaphoricity of word pairs (because Metaphoricity was the only signi cant e ect in the regression analysis). Furthermore, this Independent subspaces model performed best out of all of the models tested here, with an R2 of .271 (while Word2Vec had an R2 of only .249). Although this is only a modest improvement over Word2Vec, the di erence does suggest that our CDM has a greater capacity to capture perceived metaphoricity. We therefore conclude that the Independent subspace method o ers both the most accurate and the most direct model of Metaphoricity (without confounding e ects from Familiarity or Meaningfulness).
It is interesting to consider the comparative performance of the di erentlycon gured CDM models, and explore why the Independent subspaces technique results in by far the best subspaces for mapping human judgments of metaphoricity to cosine similarity between word-vectors. In as much as a \property theoretic view of metaphor" [28, p. 56] has been laid out in computational terms, the expectation is that a successful computational model of metaphor will capture the way in which salient properties of a source are mapped to speci c instances of a target. The subspaces generated by our model are intended to represent a conceptually relevant contextualisation of a lexical space: the dimensions of these subspaces consist of sets of words which taken independently o er only anecdotal glimpses into the way language is used, but which collectively can be understood as a certain way of speaking about a conceptually coherent topic.
So in a metaphorically relevant subspace, we hope to discover a de facto overlapping of some but not all of the properties of source and target. Rather than discover spaces where the mutual properties of two conceptual domains are already to some extent emphasised { as is the case with our Joint subspaces { we seek spaces where only a degree of overlap between the salient properties of each conceptual domain can be found, and where precisely this feature of a space is signi cant. This explains the e cacy of our Noun-only and, moreover, Independent subspaces in mapping human judgments of metaphor. In the case of the Noun-only subspace, we establish a context emphasizing the salient properties of the noun; to the extent that a verb expresses a conceptually paradigmatic action in this context, the cosine similarity between noun and verb word-vectors will be high, becoming lower as the noun-verb relationship becomes more metaphorical in nature. This phenomenon is considerably more evident in our Independent subspaces, where cosine similarity shows an inverse correspondence to the salient properties of each component of the dyad which have been merged into a single hybrid context.
It is worth noting that distributional semantic models have typically been applied to tasks involving the identi cation of word similarity, with the underlying intuition regarding these spaces being that similar words occur in similar contexts. Similarity here must be understood in a di erent light than the familiarity inherent in a word pairing: we might expect familiarity to correlate roughly with a tendency towards juxtaposition, and so a statistical measure of familiarity might emerge simply from calculating the PMI between two cooccurring words. Nonetheless, we must also note that words that tend to occur together will necessarily also tend to occur together in the same context, and so we might expect familiarity to emerge as a kind of artifact of this tendency in spaces geared towards similarity. It is therefore not surprising that a fairly standard distributional semantic model such as Word2Vec captures a degree of familiarity in measures of cosine similarity.
With this in mind, we might imagine a way forward towards building more nuanced subspaces particularly geared to prise apart judgments of metaphoricity. We could, for instance, investigate techniques for building subspaces that focus primarily on the source component of a word pair { the verb, in the cases studied here { in order to draw out the salient properties of the source and then measure the degree to which these properties are typically transferable to a target. Finally, in future work we hope to use our ndings regarding the geometric properties of subspaces to discover how people are likely to interpret new word pairs. In contrast to the research presented above, where human ratings were used to explain the variance in cosine similarity scores, this future direction will use cosine similarity scores for novel dyads to predict the degree to which human participants will perceive a given dyad as being metaphorical.
This research is supported by the project ConCreTe, which acknowledges the nancial support of the Future and Emerging Technologies (FET) programme within the Seventh Framework Programme for Research of the European Commission, under FET grant number 611733. This research has also been supported by EPSRC grant EP/L50483X/1.