This paper describes a model of synaesthetic metaphor in non-poetic and art-related texts, whose ultimate goal is to suggest sensory alternatives for contents accessed mainly by sight in museums and art galleries. We created and applied a multi-level annotation scheme to create a manually annotated resource of synaesthetic metaphors extracted from museum catalogues and designed a pipeline for the automatic detection and interpretation of synaesthetic metaphors in texts. Finally, we tested a preliminary implementation of this pipeline on real data, shedding light on the relevance and complexity of this phenomenon and the possible improvement areas.
This paper is structured as follows. After surveying the related work in Section 2, we describe the methodology behind the manual annotation and design of the pipeline (Section 3). Section 4 illustrates the evaluation of the proposed pipeline. Discussion and Conclusion end the paper.
Synaesthesia in language Synesthetic metaphors consist of two subjects: the tenor and
the vehicle, so the first can be economically described by a transfer of the implicit and explicit
attributes of the second [
The terms corresponding to tenor and vehicle in Lakof’s theory are target and source, two
structures (or domains) underlying cognitive processes detectable in language [
Interpretation and detection tasks Su et al. [
Tekiroglu et al. [
Lievers [
Our approach rely on a more complete, explicit model of synaesthetic metaphor and difers from previous approaches in the use of annotated resources (lexica and datasets). Moreover, the model is tailored to synaesthetic metaphors in art descriptions.
In order to create resources and tools for analyzing synaesthetic metaphors in artwork descriptions, we combined manual annotation and automatic tools to design a pipeline for detecting and analysing synaesthetic metaphors. After identifying the relevant syntactic, lexical and semantic features of synaesthetic metaphor from the literature, we manually identified and analysed its occurrences in the description of artworks in a set of museum catalogues. Secondly, we designed a pipeline which relies on a combination of supervised and unsupervised classification methods to detect and analyse the occurrences of this specific metaphor type. The ultimate goal is two fold: on the one side, leveraging manual annotation to collect ground truth data; on the other side, testing tools for the detection and study of synesthetic metaphors.
Annotation Scheme Based on the syntactic, lexical and semantic features emerged from the
literature on synaesthetic metaphor, we created a multi-level annotation scheme.
One of the authors manually annotated a corpus of 633 artwork descriptions (in English) taken
from the catalogues of the Turin Gallery of Modern Art (Galleria d’Arte Moderna, GAM), the
Irish Museum of Modern Art (IMMA) of Dublin, and the Hong Kong National Museum, with
the goal of collecting ground truth data on synaesthetic metaphors in our domain.
The text was annotated by using GATE, a general architecture for text engineering [
The annotation scheme includes: • sentence-level contextual information: whether the excerpt includes visual information (e.g., painted or sculpted figures’ position in space, argumentative description about visual features such as shapes, colors, brightness), abstract information (e.g., feelings and concepts), other sensory details (e.g., materials, textures), and historical data (e.g., biographies, historical events) • multi-word synaesthetic metaphors, annotated for: – scheme in Ullmann’s hierarchy – syntactic pattern – word-level information: ∗ source and target that can be experienced through the sensory modalities ∗ source and target that are associated with concrete
Contextual annotation: <Collins’ style developed out of a fascination and connection with the Irish landscape. His practice would often involve detailed studies of twigs and moss><biographical information>. <Misty hues> <synaesthetic metaphor> dominate his palette, contributing to the atmospheric mood. <Although Collins was primarily a landscape artist it was the concept of the land that concerned him – his work rarely related to a particular place but captures the romantic notion of a poetic Ireland><biographical information>. ‘The Wood Pigeon’s Nest’ perfectly encapsulates this atmospheric style. The image of nest and egg is central <visual description><sensory cue:position> The vulnerability and fragility of the nest is evoked<abstract concept>, the subject hew out of an abstract background <visual description>.
Annotation of synesthetic metaphors: misty<haptic><hues><visual> The annotated dataset As it can be seen in Table 1, there are significant diferences in the synaesthetic metaphors found in the various museums. Some museums catalogues, in fact, seem to be more productive in terms of synaesthetic metaphors: for IMMA, in particular, only 13 synaesthetic metaphors were found for overall 493 artworks; on the other extreme, the GAM museum yielded 20 metaphors for only 38 artworks. Despite these huge diferences, which may be motivated by cultural and linguistic diferences in the tradition art writing and deserve further investigation, the role of synaesthetic metaphor proves to be relevant in artwork description.
From the GAM’s catalog, 24 synaesthetic metaphors were identified for 12 diferent artworks. Most identified pairs belong to the adjective-name pattern, while rarely to the verb-noun pattern. For the IMMA (Irish Museum of Modern Art) catalog, 14 synaesthetic metaphors were identified for 13 diferent artworks. As well for the GAM’s catalog, most pairs belong to the adjective-name pattern, while infrequently to the verb-noun pattern.
From the point of view of detection and interpretation, then, it is also worth considering the frequency of the various schemes. In the IMMA and GAM’s catalogues, we have identified 40 synaesthetic metaphors: over 80% reflect the mapping between touch and vision (e.g., cold picture,warm colour, sensuous tone). In parallel, all metaphors identified in the IMMA catalogue represent the same directionality. For the analyzed catalogues, synaesthetic metaphors rarely involve the gustatory, olfactory and hearing modalities. For the GAM catalogue, a few samples of this type were identified, with a frequency of less than 12% (e.g., erotic flavour , fresh tone); the synaesthetic metaphors collected from the IMMA’s catalogue also feature a shallow range of modalities, excluding the gustatory and olfactory ones. Nevertheless, for this catalogue, a rarefied tactile dimension emerges without reference to taste or smell. The preference for visual, haptic mapping is confirmed for the Hong Kong museum. Compared to the IMMA and GAM catalogues, 25% of the collected metaphors from the Hong Kong Museum are descriptive of the brushstrokes on the canvas, e.g. described as delicate, crisp, and rigorous.
Notice that this ordering confirms the findings are also in line with the hierarchy of modalities, according to which synaesthetic metaphors proceed from the higher to lower modalities. Collecting art-related descriptions In order to gather further text data for testing the manual and automatic annotation tools, we created a corpus of artwork descriptions (Corpus A) via web scraping from Google Arts and Culture. Launched in February 2011 by Google, it hosts around six million high-resolution images of works of art worldwide, sometimes complemented by a textual description and metadata such as title, author, and date. This corpus includes artworks descriptions from a set of ten diferent western and eastern countries (Table 2), selected on the basis of the diversity of collections, cultures and geographical area.
The annotation pipeline we designed and tested on the annotated data encompasses three main steps.
1. Syntactic annotation. Artwork descriptions from the museum catalogues are parsed and the word pairs which match the syntactic patterns of synaesthetic metaphor are extracted, yielding a set of candidate synaesthetic metaphors. 2. Identification of sensory modalities . In this phase, the candidate word pairs are automatically annotated for the sensory modalities using a multi-classifier. This step relies on lexical resources for the identification of the sensory modalities of words. 3. Filtering. In this phase, the candidate word pairs, enriched with the sensory domains in the previous step, are matched against the synaesthetic metaphor schemes provided by Ullmann (Section 2). Only the pairs which realize one of the possible schemes are kept, while the others are discarded: for example, word pairs where source and target belong to the same sensory modality cannot qualify as synaesthetic metaphors. 4. Classification . In this phase, a binary classifier is run on the obtained word pairs to identify the actual synaesthetic metaphors.
In the following, we describe the experiments carried out to assess the feasibility of the pipeline.
Syntactic annotation In order to extract from art descriptions the syntactic patterns which
characterize synaesthetic metaphor, we used a well-established, standard format and pipeline for
syntactic annotation (Universal Dependencies, UD) [
After parsing, we extracted four dependency patterns, each composed of a word pair:
• adjective (adj) – noun (nn) where nn is parent of adj & adj precedes nn by exactly one
position,
• adjective (adj) – noun (nn) where nn is parent of adj & adj precedes nn by exactly two
positions,
• adjective (adj) – noun (nn) where nn is parent of adj & adj precedes nn by exactly three
positions,
• verb (vrb) – noun (nn) where nn in parent of vrb & vrb precedes nn by exactly one position.
Identification of sensory modalities We applied a multi-class classification with logistic
regression to map the words involved in synaesthetic metaphors onto seven classes [
A relevant issue is given by the fact that the datasets employed to map words to the five senses contain considerable noise. For example, in Lynott and Connell’s dataset of nouns paired with sensory domains, many nouns are highly abstract, and only some are directly related to perception [13].
Filtering Since the model encompasses constraints on the cross-modal directionality, we applied the scheme provided by Ullmann (1957) for the directionality of the sensory mapping to the extracted word pairs Classification We run the Hugging-Face zero-shot classification pipeline to classify the resulting word pairs into metaphorical and non-metaphorical classes.
Yin et al.[18] proposed a method for using pre-trained NLI models as classifiers of zero-shot sequences. Thus, we used Facebook’s bart-largemnli model [19], which is a checkpoint model further trained on the MNLI (Multi-Natural Language Inference) dataset, as basic model for the zero-shot classification model. The used candidate labels were “metaphorical” and “nonmetaphorical”.
The use of this pipeline, not been previously tested for synaesthetic metaphor, represents a first attempt to automatically identify the occurrences of this specific metaphor type. In addition, it can provide insight on the complex relationship between metaphor and synaesthetic metaphor. The Vrije Universiteit Amsterdam Amsterdam Metaphor Corpus is the largest available corpus hand-annotated for all metaphorical language use, regardless of lexical field or source domain. The VUA corpus was annotated to detect indirect, direct, and implicit metaphors, personification, metaphor signal and borderline cases. It does not include synthetic metaphors [20].
The pipeline has been tested on the dataset of the Hong Kong museum, which yielded the highest number of occurrences of synaesthetic metaphor in the manual annotation phase. In particular, starting from the results of the syntactic analysis (see Table 3), we focused on the steps which rely on less established tools, namely step 2 (Automatic annotation of sensory modalities) and 4 (Zero-shot classification). The output of these two steps has been compared with the manually annotated data to assess the performance of the automatic tools, and gain insight from discrepancies.
Automatic annotation of sensory modalities The multi-class classification was used to annotate the 133 word pairs, extracted from the catalogue of the museum, which belonged to Ullmann schemes (see Table 4). By doing so, we can evaluate the obtained classification, we considered not only the pairs corresponding to the actual metaphors found by the human annotator, but the overall set of pairs identified by filtering the candidate word issued from the syntactic analysis with the schemes identified by Ullmann. By doing, so we can assess the performance of the classifier within the context of the real pipeline. The accuracy of the multiclass sensory modality classification was measured by comparing the manual classification of sensory modalities with the classes returned by the automatic annotation.
The number of correctly assigned sensory modality schemes was 54 out of 133 (40,60%). Given the relevance of the syntactic patterns in the detection pipeline, we also report their distribution according to the syntactic scheme (the number of - indicate the distance between the nodes): • Adjective - Noun: haptic → visual: 41 pairs (of which 5 synaesthetic metaphors, 13%) • Adjective - Noun: haptic → taste: 1 pair (of which, 1 synaesthetic metaphor, 100%) • Adjective - Noun: haptic → auditory: 1 pair (of which, 1 synaesthetic metaphor, 100%) • Adjective - - Noun: haptic → visual: 3 pairs (of which, 3 synaesthetic metaphors, 100%) • Adjective - - - Noun: haptic → visual: 8 pairs (of which, 2 synaesthetic metaphors, 25%) The multi-class classification model sufers from the noise of the lexical resources employed, many related to the documented cross-modality of adjectives and nouns, errors in vocabulary sampling or annotator misunderstandings [17]. Despite this, sensory features remain essential for recognition.
Zero-shot classification The zero-shot model was tested on metaphorical and nonmetaphorical pairs.
We tested the accuracy of the zero-shot classification model on a set of 40 metaphorical (20) and non-metaphorical (20) pairs mined from the Hong Kong museum’s catalogue (issued from manual annotation). The results can be found in Table 5.
Moreover, the zero-shot classification model was evaluated also on a set of 63 metaphorical
(34) and non-metaphorical (29) pairs provided by Su et. al [
For the Honk Kong museum, Precision (P) and recall (R) of the metaphorical pairs are: • P = 100% • R = 5,8%
• P = 100% • R = 25% For the Su et. all dataset (2019), Precision (P) and recall (R) of the non-metaphorical pairs are the following: • P = 74% • R = 100% • P = 79% • R = 100% Precision (P) and recall (R) of the metaphorical pairs are: As it can be observer from these data, the zero-shot classification model sufers from a very low recall score regarding the Hong Kong’s pairs rather while it works well for the Chinese pairs. The recall percentage for the metaphorical pairs from the Hong Kong’ dataset is 5,8% and for the Su et al. one is 100%. This disparity in scoring may be related with the presence of iconic metaphors in the Su et al.’s dataset (2019).
Discussion Considering the dificulties emerged in the application of the pipeline to real data, the preliminary results reported above suggest diferent research lines.
First of all, the quality of resources and their suitability for this task require further investigation.
As acknowledged by the literature, in fact, the mapping of words onto sensory modality is not
fully reliable, and is afected by the method by which the mapping has been obtained. Only
in half of the cases, in fact, the sensory modality scheme has been correctly assigned by the
multi-classifier, partly due to wrong mappings in the datasets employed for the task.
Secondly, the relationship with the models and resources for the detection and interpretation
metaphors appears intricate. Metaphors are usually characterized by abstract concepts, while
synaesthetic metaphors are intrinsically rooted in concreteness, being related with perception
of the physical word through senses. The significance of the better performance of the zero-shot
classification on the dataset of synaesthetic metaphors by [
Finally, this preliminary investigation points out the importance of specific modalities. For example, if we observe the data reported in Table 4, which reports the sensory modality schemes of the candidate word pairs, the primary role of the Hearing-Sight scheme, and of Sight in general, clearly emerges. This prevalence is confirmed also by the manual annotation. On the one side, this represents an opportunity for creating alternative sensory experiences of art by replacing sight with hearing; other examples associated with the tactility of painting surfaces, provide a basis for enhancing the experience of art by touch (see the discussion in Section 3.1). On the other side, these data orient the research towards specific, more frequent patterns and schemes, suggesting that the creation and enhancement of resources should address these sensory modalities to improve the classification tasks.
In this paper, we presented a preliminary model and pipeline for the detection and interpretation of synaesthetic metaphors in artwork description. In order to explore the occurrence and types of these metaphors in real data, we annotated a corpus of texts extracted from museum catalogues. Moreover, we designed a pipeline which leverages automatic methods for identifying synaesthetic metaphors based on syntactic, lexical and semantic features. This pipeline has been implemented with state of the art tools and evaluated on a set of real data. Although preliminary, this experiment confirms the relevance of this phenomenon and its potential for implementing alternative ways for experiencing art, in a universal access perspective. For future research, we intend to improve the quality and coverage of the resources used for the classification of sensory modalities by crowdsourcing annotations on museum data. Also, we will investigate the emotional valence and range conveyed by synaesthetic metaphors, in order to explore their potential in a more comprehensive way. [12] D. Lynott, L. Connell, Modality exclusivity norms for 423 object properties, Behavior
Research Methods 41 (2009) 558–564. [13] D. Lynott, L. Connell, Modality exclusivity norms for 400 nouns: The relationship between perceptual experience and surface word form, Behavior research methods 45 (2013) 516–526. [14] B. Winter, Taste and smell words form an afectively loaded and emotionally flexible part of the english lexicon, Language, Cognition and Neuroscience 31 (2016) 975–988. [15] F. Conca, E. Catricalà, M. Canini, A. Petrini, G. Vigliocco, S. F. Cappa, P. A. Della Rosa, In search of diferent categories of abstract concepts: a fmri adaptation study, Scientific reports 11 (2021) 1–11. [16] F. S. Lievers, B. Winter, Sensory language across lexical categories, Lingua 204 (2018) 45–61. [17] B. Winter, The sensory structure of the English lexicon, University of California, Merced, 2016. [18] W. Yin, J. Hay, D. Roth, Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach, arXiv preprint arXiv:1909.00161 (2019). [19] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019). [20] T. Krennmayr, G. Steen, Vu amsterdam metaphor corpus, in: Handbook of linguistic annotation, Springer, 2017, pp. 1053–1071.