=Paper=
{{Paper
|id=Vol-2347/1-Bolognesi
|storemode=property
|title=How Do Words vs. Images Construct and Represent Metaphor
|pdfUrl=https://ceur-ws.org/Vol-2347/1-Bolognesi.pdf
|volume=Vol-2347
|authors=Marianna Bolognesi
|dblpUrl=https://dblp.org/rec/conf/c3gi/Bolognesi18
}}
==How Do Words vs. Images Construct and Represent Metaphor==
https://ceur-ws.org/Vol-2347/1-Bolognesi.pdf
How do words vs. images
construct and represent metaphor
Marianna BOLOGNESI1
University of Oxford
Abstract Metaphorical similarity is a peculiar type of semantic relation, based on a
very limited number of features that are shared by the two metaphor terms. The
nature of these shared features is still largely unknown. Similarly, we know little
about whether different modes of metaphor expression (e.g., images, language) use
the same types of features to construct metaphors. I hereby report a series of
distributional analyses based on a representative sample of pictorial and linguistic
metaphors. Three different types of similarity are operationalized through three
different distributional methods that are based on the same underlying principle (the
distributional hypothesis) but model semantic representations based on different
information. Based on such analyses I show that the pictorial and the linguistic
modes of expression afford different ways to construct metaphors, because they tend
to exploit different types of features that are shared by the metaphor terms. The
results are discussed within a cognitive linguistic framework, in which I defend a
multi-layered view of conceptual metaphor, in which image schemas might
constitute the most generic layer of representation, at which the difference between
pictorial and linguistic metaphors may disappear.
Keywords. Semantic similarity, distributional semantics, conceptual richness,
metaphors, multimodality.
1. Introduction
The growing interest in multimodal communication and in how meaning is constructed
and expressed within the pictorial mode (Jewitt 2009; Kress 2010; Bateman 2014;
Bateman et al. 2017) has attracted also scholars working on pictorial and multimodal
metaphor (e.g., Forceville & Urios-Aparisi 2009), who conducted analyses across
different genres, including advertising (e.g., Forceville 1996) and political cartoons (e.g.,
El Refaie 2009).
However, there are virtually no previous quantitative studies that aim
specifically at comparing the structure and functioning of the pictorial and the linguistic
modes of expression in relation to metaphor construction.
The cognitive view of metaphor, fathered by Lakoff and Johnson (1980)
suggests that metaphors are matters of thought. In this view, (linguistic) metaphoric
expressions are distinguished from conceptual structures in a binary manner: there are
(linguistic) expressions on one hand, and there are conceptual structures on the other
hand. Because conceptual metaphors are considered as ‘supra-modality’, one might
expect to find the same conceptual structures expressed in different modes (e.g., in
images and in language). However, such binary opposition between metaphoric
1
Marianna Bolognesi, Faculty of Medieval and Modern Languages, University of Oxford, UK.
Email: marianna.bolognesi@mod-langs.ox.ac.uk.
expression and conceptual metaphor neglects the variety of different sub-levels that can
be distinguished within the conceptual dimension. As Kovecses (2017) acknowledges,
this is an issue commonly raised in relation to conceptual metaphors: at which level of
generality should we formulate conceptual metaphors? Kovecses distinguishes between
four levels - image-schemas, domains, frames and mental spaces - and claims that all
these levels contribute to structure our conceptual system and the conceptual metaphors
therein. The advantages of adopting a fine-grained and multi-layered view of conceptual
metaphor enable researchers to investigate in a bottom-up and data-driven manner how
different modes (e.g., images and language) construct metaphors, without imposing the
straitjacket of a necessarily unique supra-modality conceptual structure onto which
metaphoric expressions in all modalities shall converge. The analyses presented here
show that, at some level of conceptualization, typical metaphors found in images and
typical metaphors found in language differ quite consistently and do so in several ways 2.
2. Theoretical Background
Functional neuroimaging evidence show different patterns of neural activation during
matched word and picture recognition tasks (Bright et al. 2004; Gates & Yoon 2005),
suggesting that processing the pictorial and the linguistic expression of a concept
activates different brain areas. Moreover, a variety of clinical studies report about
patients with profound visual object recognition disorders, but relatively intact word
comprehension (Binder et al. 2009). These findings suggest that pictorial ad linguistic
stimuli afford different types of cognitive processing routes and tap into different
conceptual representations, at least at some level of abstraction (see also Dual Coding
Theory, Paivio 1971; 2010). Given the same concept, images and words, respectively,
seem to favor the encoding of different types of information about such concept. For
example, images trigger a deeper emotional response, compared to words (e.g. Kensinger
& Schacter 2006).
Taking such findings in the field of metaphor, it can be argued that typical
metaphors expressed through words and typical metaphors expressed through images
might be constructed on the basis of different types of features, which are shared by the
metaphor terms. This does not imply however that one of the two modalities constructs
‘more conceptual’ or ‘more embodied’ metaphors than the other.
Based on a literature review on conceptual richness in cognitive psychology
(Recchia & Jones 2012; Kounios et al. 2009; Pexman et al. 2008) I argue that a fairly
rich approximation of our general knowledge about a concept can be obtained by
observing:
1) Its entity-related, attributive properties;
2) Its experience-based relational properties;
3) Its language-based contexts.
For example a fairly rich approximation of what MARGARITAS are, is given by:
2
The analyses hereby reported are based on three studies published as outputs of the EU-Marie Curie
awarded project CogViM (Cognitive Grounding of Visual Metaphor, FP7-IEF2013-629076).
• Attributional properties: e.g. (perceptual property),
(an ingredient of margaritas), (perceptual property).
• Relational properties: e.g. (can be typically found in the same contexts
as margaritas, on the glass rim), (season in which margaritas
are typically consumed).
• Linguistic contexts: e.g. “blending margaritas”, “making smooth frozen
margaritas”, “drinking too many margaritas”, which are exemplary sentences in
which the word margarita can be used.
Although these three streams of semantic knowledge may contain overlapping
information, they are theoretically distinct. Based on these different streams of semantic
information, different types of semantic similarity between two concepts can be
constructed. For example, MARGARITAS share attributional properties with LONG
ISLANDS (they both contain tequila), but not as many relational properties (the latter
cocktails are not served with salt and are less typical for summer beach parties). It follows
that MARGARITAS and LONG ISLANDS are similar in their attributional structures, but not
so much in their relational and linguistic structures. Contrariwise, based once again on
the few properties mentioned above, the reader might argue that MARGARITAS share
relational properties with, for example, SALADS (which are also served with salt and
consumed especially in the summer), while these two concepts do not share attributive
or linguistic properties. Finally, it could be argued that MARGARITAS share linguistic
properties with MILKSHAKES (both are blended, smooth and frozen), but fewer attributive
and relational properties.
In metaphor studies, the classic comparison view (e.g., Ortony 1979) defines
similarity on the same lines as Tversky’s similarity definition (1977), that is, as a feature-
matching process. Other views suggest that the similarity between two metaphor terms
emerges specifically from their interaction (e.g.: Black 1979), or from the interaction of
complex analogical structures (e.g., Bowdle & Gentner 2005). The approach used here
leans toward the classic comparison view, that is, metaphorical similarity is
operationalized as a function of shared properties across three streams of semantic
information. The analyses discussed here are therefore ‘limited’ to the metaphorical
similarity modelled as a feature-matching process. Nonetheless, such process includes
the matching of entity-related, as well as relational properties, and the syntactic patterns
plus lexical collocates in text corpora.
A growing body of scientific literature has previously tackled aspects of
metaphor comprehension by means of distributional semantics. For example, a
pioneering study conducted by Kintsch (2000) showed in a qualitative fashion how
Latent Semantic Analysis (Landauer & Dumais 1997) can be used to model metaphor
comprehension. In a more recent and extensive project (Utsumi 2011), categorization
and comparison processes involved in metaphor comprehension were compared and
modelled through distributional semantics. Within the nlp and machine learning
communities the interest in statistical modelling of metaphor has also been growing
recently (Veale et al. 2016). These studies typically aim at modelling metaphor structure
(rather than the cognitive processes that lead to metaphor comprehension), and tackle
problems such as metaphor detection in text corpora, or address specific types of
metaphor, such as verb metaphoricity (Del Tredici & Bel 2016).
3. Method and Materials
A sample of 50 pictorial metaphors retrieved from the VisMet corpus (Bolognesi et al.
2018) and 50 linguistic metaphors retrieved from the Metaphor Corpus (Steen et al.
2010) were used for the analyses 3. In order to compare the pictorial and the linguistic
stimuli, the 100 metaphors were all formalized into A-IS-B statements by applying
established procedures in formal content analyses featuring independent annotators and
calculations of interrater reliability scores (MIPVU for the identification of linguistic
metaphors, Steen et al. 2010; VISMIP for the identification of visual metaphors, Šorm &
Steen 2018). Details about these procedures are reported in Bolognesi (2017).
The three distributional analyses are described in detail in the dedicated articles
(Bolognesi 2016a; Bolognesi 2017; Bolognesi & Aina 2017). To summarize,
attributional properties are operationalized as semantic features attributed to the concepts,
collected in property generation tasks (as in McRae et al. 2005) (e.g. CAR: , ). The similarity between each pair of metaphor terms is computed in
terms of the amount of shared semantic features. For example: consider an advertisement
where a car is represented as a rearing horse. The visual metaphor is formalized through
the VISMIP procedure as CAR-IS-HORSE. Metaphorical similarity is quantified here as the
cosine between the vectors of CAR and HORSE, whose dimensions are the semantic
features of the two concepts. Relational properties are operationalized through Flickr
Distributional Tagspace (Bolognesi 2014; Bolognesi 2016b): a corpus of roughly
100,000 tagsets for each metaphor term was created. The similarity between two
metaphor terms is quantified here as the cosine between the vectors of CAR and HORSE,
whose dimensions are the tags with which these concepts appear across tagsets.
Language-based contexts are operationalized through typedm, (Distributional Memory,
Baroni & Lenci 2010), a multi-purpose structured distributional model that encompasses
syntactic as well as semantic information about words. The similarity between two
metaphor terms is quantified here as the cosine between the vectors of CAR and HORSE,
whose dimensions are the linguistic contexts of the two words in type-DM.
4. Analysis
The analyses of metaphorical similarity across the three different distributional spaces
show different patterns for pictorial and linguistic metaphors 4 . The results are
summarized in Table 1.
Table 1. Attributional, relational and linguistic similarity between pairs of metaphor terms.
Attrib.Sim Relat.Sim Ling.Sim
Linguistic metaphors M=0.012, SD=0.047 M=0.096, SD=0.057 M=0.192, SD=0.087
Pictorial metaphors M=0.050, SD=0.067 M=0.156, SD=0.061 M=0.121, SD=0.089
t-test t= 3.282, p < 0.05 t=5.169, p<0.05 t= -4.194, p<0.05
Table 1 shows that the three patterns of metaphorical similarity differ for the two
samples of metaphors: pictorial and linguistic metaphors are constructed and represented
3
The lists of metaphors can be found in the appendix to Bolognesi (2017:550).
4
In the studies reporting these specific analyses, the measures of metaphorical similarity are also compared to
the similarity emerging from to randomly paired concepts.
on the basis of different types of semantic information shared between metaphor terms.
Moreover, the manual annotation of the features shared by the metaphor terms (based on
the Wu and Barsalou 2009 taxonomy of feature types 5) shows that pictorial metaphors
are typically constructed on shared features that express entity-related properties
(typically perceptual features and components of the predicated concept), and
experience-based relational properties (typically locations in which the concepts appear
and objects/participants that populate these environments). Conversely, linguistic
metaphors appear to be typically constructed on features that are mainly taxonomic (such
as, for example, hypernyms that are shared by the two terms of the linguistic metaphor).
Taxonomic information is well-captured and represented in language use, and this is
probably why a language-based distributional model (like DM) is more suitable for
capturing metaphorical similarity for linguistic metaphors, as opposed to distributional
models based on entity-related and experience-based relational properties.
5. Discussion and Conclusion
Linguistic categories (e.g. the word car) and visual categories (e.g. a pictorial
representation of a car) classify perceptual experiences in different ways. These two
semiotic systems have different ‘preferences’ in the type of information that can be more
easily expressed. It is therefore to be expected that, at some level of abstraction, typical
pictorial and linguistic metaphors behave in different ways, and construct comparisons
on the basis of different types of features, which are shared by the metaphor terms. This
is understandable only when we adopt a multi-layered view of conceptual metaphors,
such as that offered by Kovecses (2017): when we talk about conceptual metaphors we
need to take into account a variety of levels that constitute the so-called conceptual
system. These levels of conceptual representations range from levels that contain more
conceptually rich information (e.g., mental spaces) to highly schematic ones (e.g., image
schemas). The first levels involve richer representations that are arguably more deeply
influenced by modality-specific information, and therefore by the metaphoric
expressions that can be typically found in specific semiotic systems. Contrariwise,
deeper and more schematic levels of metaphor analysis, such as those based on image
schemas, may see mode-specific differences disappear, and common embodied (but
semantically impoverished) patterns based on image schemas emerge. The studies here
reported tackle a level of metaphor analysis at which significant differences between
pictorial and linguistic metaphors can still be operationalized and measured. It might be
interesting to investigate, in further research, at what level of abstraction the conceptual
metaphors extracted respectively from linguistic and from pictorial expressions become
really independent from their semiotic manifestations, and therefore completely supra-
modality. I believe that such equipollence can be established only at the image schematic
level.
References
[1] Baroni, M., & Lenci, A. Distributional Memory: A general framework for corpus-based semantics.
Computational Linguistics, 36, 4, (2010), 673-721.
5
These analyses are reported in the three related studies, in which the supplementary materials are provided.
[2] Bateman, J. Text and Image, Routledge, London, 2014.
[3] Bateman, J., Wildfeuer, J., & Hiippala, T., Multimodality: Foundations, Research, Analysis – A Problem-
Oriented Introduction, Berlin, De Gruyter, 2017.
[4] Binder, J.R., Desai, R.H., Graves, W.W., & Conant, L.L., Where is the semantic system? A critical review
and meta-analysis of 120 functional neuroimaging studies, Cerebral Cortex, 19, 12, (2009), 2767 – 2796.
[5] Black, M., More about metaphor, in A. Ortony (ed.), Metaphor and Thought, Cambridge University Press,
Cambridge, 19-43, 1979.
[6] Bolognesi M., Distributional Semantics meets Embodied Cognition, Selected Papers from the 4th UK
Cognitive Linguistics Conference, (2014), 18-35.
[7] Bolognesi M., Modeling semantic similarity between metaphor terms of visual vs linguistic metaphors
through Flickr tag distributions, Frontiers in Communication, 1(9), (2016a).
[8] Bolognesi M., Flickr Distributional Tagspace: Evaluating the Semantic Spaces emerging from Flickr Tags
Distributions, in M. Jones (ed.), Big Data in Cognitive Science, Routledge, London, 144-173, 2016b.
[9] Bolognesi M., Using semantic features norms to investigate how the visual and verbal modes afford
metaphor construction and expression, Language and Cognition, 9(3), (2017), 525-552.
[10] Bolognesi M., & Aina L., Similarity is Closeness: Using Distributional Semantic Spaces to model
Similarity in Visual and Linguistic Metaphors, Corpus Linguistics and Linguistic Theory (2017).
[11] Bolognesi, M. Van den Heerik R., & Van den Berg E., VisMet: a corpus of visual metaphors, in G. Steen
(ed.) Visual Metaphor: structure and process. Amsterdam: Benjamins Publishers, 89-114, 2018.
[12] Bright, P., Moss, H., & Tyler, L., Unitary vs. multiple semantics: PET studies of word and picture
processing, Brain and Language, 89, (2004), 417 – 432.
[13] Del Tredici, M. & Bel, N., Assessing the Potential of Metaphoricity of verbs using corpus data, in
Proceedings of LREC 2016, 4573-4577, 2016.
[14] El Refaie, E., Metaphor in political cartoons: Exploring audience responses, in: C. Forceville and E.
Urios-Aparisi (eds), Multimodal Metaphor, 173-196, Mouton de Gruyter, Berlin, 2009.
[15] Forceville, C. Pictorial metaphors in advertising, Routledge, London, 1996.
[16] Forceville, C., & Urios-Aparisi, E. (eds.), Multimodal metaphor, Mouton de Gruyter, Berlin, 2009.
[17] Gates, L., & Yoon, M., Distinct and shared cortical regions of the human brain activated by pictorial
depictions versus verbal descriptions: an fMRI study. Neuroimage, 24, (2005), 473 – 486.
[18] Jewitt, C. (ed.), The Routledge Handbook of Multimodal Analysis, Routledge, London, 2009.
[19] Kensinger, E.A. & Schacter, D.L., Processing emotional pictures and words: Effects of valence and
arousal, Cognitive, Affective, and Behavioral Neuroscience, 6, (2006), 110-127.
[20] Kintsch, W., Metaphor comprehension: A computational theory, Psychonomic Bulletin & Review, 7,
(2000), 257-266.
[21] Kounios, J., Green, D.L., Payne, L., Fleck, J.I., Grondin, R., & McRae, K., Semantic richness and the
activation of concepts in semantic memory: Evidence from Event-Related Potentials. Brain Research,
1282, (2009), 95-102.
[22] Kovecses, Z., Levels of metaphor, Cognitive Linguistics, 28, 2 (2017), 321-347.
[23] Kress, G. Multimodality: a social semiotic approach to contemporary communication, Routledge,
London, 2010.
[24] Lakoff, G., & Johnson, M., Metaphors we live by, University Press, Chicago, 1980.
[25] Landauer, T. & Dumais, S., A solution to Plato's problem: The latent semantic analysis theory of
acquisition, induction, and representation of knowledge. Psychological review, 104, 2, (1997), 211-240.
[26] Ortony, A., Beyond literal similarity. Psychological Review, 86, (1979), 161-180.
[27] Paivio, A., Dual coding theory and the mental lexicon. The Mental Lexicon 5 (2010), 205–230.
[28] Paivio, A., Imagery and verbal processes, Holt, Rinehart, and Winston, New York, 1971.
[29] Pexman, P.M., Hargreaves, I.S., Siakaluk, P.D., Bodner, G. E., & Pope, J., There are many ways to be
rich: Effects of three measures of semantic richness on visual word recognition. Psychonomic Bulletin
and Review, 15, (2008), 161-167.
[30] Recchia, G., & Jones, M., The semantic richness of abstract concepts. Frontiers in Human Neuroscience,
6, 315, (2012).
[31] Šorm, E., & Steen, G., VISMIP: Towards a method for visual metaphor identification. In G.J. Steen (ed.),
Visual metaphor: Structure and Process, John Benjamins, Amsterdam, 2018.
[32] Steen, G., Dorst, A.G., Herrmann, J.B., Kaal, A.A., Krennmayr, T. & Pasma, T., A method for linguistic
metaphor identification. From MIP to MIPVU, John Benjamins, Amsterdam, 2010.
[33] Tversky, A. (1977). Features of similarity. Psychological Review, 84, 327-352.
[34] Utsumi, A. (2011). Computational exploration of metaphor comprehension processes using a semantic
space model. Cognitive Science, 35, 2, 251-296.
[35] Veale, T., Shutova E. & Klebanov, B., Metaphor: A Computational Perspective. Synthesis Lectures on
Human Language Technologies. Morgan and Claypool Publishers, 2016.