Metaphors are everywhere. They appear extensively across all domains of natural language, from the most sophisticated poetry to seemingly dry academic prose. A significant body of research in the cognitive science of language argues for the existencocnecoepftual metaphors, the systematic structuring of one domain of experience in the language of another. Conceptual metaphors are not simply rhetorical lfourishes but are crucial evidence of the role of analogical reasoning in human cognition. In this paper, we ask whether Large Language Models (LLMs) can accurately identify and explain the presence of such conceptual metaphors in natural language data. Using a novel prompting technique based on metaphor annotation guidelines, we demonstrate that LLMs are a promising tool for large-scale computational research on conceptual metaphors. Further, we show that LLMs are able to apply procedural guidelines designed for human annotators, displaying a surprising depth of linguistic knowledge.
Conceptual metaphors are so deeply ingrained in our everyday language that the most frequent usage of a word or phrase may be metaphorical. This can make it difÏcult to recognize the presence of metaphorical language, even for linguists and other sp1e4c]i.aMlisotrse[over, processing and understanding conceptual metaphors necessarily involves encyclopedic general knowledge of the world, as well as shared cultural norms and stereotypes between speaker/hearer or writer/reader. Identifying conceptual metaphors has therefore consistently proven to be a challenge for both natural language processing (NLP) and artificial intelligence (AI). Given that contemporary transformer-based models ‘learn’ by consuming huge amounts of natural language and build knowledge by looking at how words are commonly used, it seems that they should similarly struggle to recognize these metaphors.
Nevertheless, being able to consistently identify the presence of conceptual metaphors in discourse would be greatly beneficial for fields such as computational literary studies and cultural analytics more broadly. In this paper, we therefore set out to test empirically whether LLMs can be used productively in the context of conceptual metaphor theory (CMT). If the answer is no, it may be evidence that developing computational approaches to conceptual metaphor remains an intractable problem. If, on the other handc,aLnLcMosnsistently identify the presence of conceptual metaphors, the question then arises of exactly how and where LLMs encode metaphoricity.
The study of CMT was inaugurated in the 1980s in Lakof and Johns1o6n]. [Central to the
argument in 1[
The extensive role and function of metaphor in natural language has not escaped researchers
in NLP. Indeed, a great deal of early work in AI concerned exactly how a computer could make
sense of metaphorical langua5g,e23[
understanding of the meaning. 2. Determine the lexical units in the text-discourse. 3.
a. For each lexical unit in the text, establish its meaning in context, that is, how it applies to an entity, relation, or attribute in the situation evoked by the text (contextual meaning). Take into account what comes before and after the lexical unit. b. For each lexical unit, determine if it has a more basic contemporary meaning in other contexts than the one in the given context. For our purposes, basic meanings tend to be i. More concrete; what they evoke is easier to
imagine, see, hear, feel, smell, and taste. ii. Related to bodily action. iii. More precise (as opposed to vague). iv. Historically older.
Basic meanings are not necessarily the most frequent meanings of the lexical unit. c. If the lexical unit has a more basic currentcontemporary meaning in other contexts than the given context, decide whether the contextual meaning contrasts with the basic meaning but can be understood in comparison with it.
4. If yes, mark the lexical unit as metaphorical. However, the encyclopedic world knowledge required in CMT is challenging to encode since the domain is essentially unbounded. Moreover, a core issue at the heart of computational metaphor analysis has historically been the lack of a shared definition of what is being studied, along with the scarcity of robust data and evaluation st2r6a]t.egies [
However, recent years have again seen a growing interest in quantitative and computational
approaches to metaphor, primarily in more applied fields such as corpus linguistics and
stylistics [
here, feel, smell, and taste - related to bodily action - more precise (as opposed to vague) - historically older Basic meanings are not necessarily the most frequent meanings of the word. Remember to only report YES for words whose use in this context is not their most basic meaning.
Examples: “I demolished his argument.” - I: NO (pronoun, basic meaning is the same) - demolished: YES (verb, more basic meaning refers to
destroying a building) - his: NO (possessive pronoun, basic meaning is the same) - argument: NO (noun, basic meaning is the same) “They write about their family.” - They: NO (pronoun, basic meaning is the same) - write: NO (verb, basic meaning is the same) - about: NO (preposition, basic meaning is the same) - their: NO (possessive pronoun, basic meaning is the same) - family: NO (noun, basic meaning is the same)
We are interested in studying whether LLMs are capable of leveraging the contextual and cultural knowledge necessary to perform difÏcult metaphor identification tasks, such as recognizing conceptual metaphors. To do this, we operationalize the Metaphor Identification Procedure (MIP), a set of annotation guidelines introduced by the Pragglejaz Group1i4n].2T00h7e [steps of MIP are detailed in Figu1r. eMIP and its variants are among the most commonly used procedures for metaphor annotation in work in corpus linguistics and stylistics; we therefore take it as a reasonable proxy for the metaphor identification process in humans.
Examining MIP, the first few steps have clear parallels in transformer-based LLMs. Step 1 is performed by the attention mechanism. Creating the representation of each word, in traditional attention mechanisms, requires ‘looking’ at every other word in the text, thus allowing the model to “establish a general understanding of [its] meaning.” Step 2, determining the lexical units in a text, is directly paralleled by tokenization. Finally, Step 3a is achieved via contextual embeddings, whose explicit purpose is to represent the specific meaning of a word in context. However, Step 3b of MIP has no clear counterpart in transformer-based models. This is because the ‘basic meaning’ of a lexical unit, as defined by MIP, is not obviously stored or represented within these models.
We want to know if state-of-the-art LLMs are capable of replicating Step 3b of the MIP
procedure despite these barriers, so we transform it into a prompt for instruction-tuned models
(Figure2). In creating the prompt, we leave the text of Step 3b almost entirely intact. “Lexical
units” are replaced by “words” in order to encourage consistent outputs. Emphasis is added in
several locations to only annotating words whichmhoarveebaasic meaning than their usage
in the input text. Finally, two examples are appended to the prompt. These examples are meant
to facilitate in-context learning and provide a structure for the models to mimic in their outputs.
They instruct the model to provide a parenthetical expression next to each word consisting of
the word’s part of speech and its more basic meaning, if one exists. These expressions were
introduced by thgept-4o model itself during initial exploration and were then included in the
prompt because they force the model to provide explanations for its annotations. This allows
for further evaluation of the outputs and ultimately means they play a similar role to Chain of
Thought prompts, which have been shown to improve model performance across a variety of
tasks [
We use this prompt to query three models from OpenAI’s GPT family, chosen because of their size, popularity, and high performance on a wide range of tasks. Specifically, we look at gpt-3.5-turbo,1 gpt-4-turbo,2 andgpt-4o.3 We access each model using OpenAI’s Chat Completion API. A system prompt“Y(ou are a helpful assistant. You have extensive linguistic knowledge.”) is provided each time the model is queried. Then, the main prompt (F2i)giusre passed as a user message, with the current text of interest appended in quotations at the end. All parameters except nuclear samplitnogp_(p) are left at their default values during prompting for all models. We steotp_p to 0.1, which means that the models only select from the tokens which make up the top 10% of the probability mass when choosing each next token. We adjust the top_p value because we are not prompting for diverse or creative text, but instead for reliable and accurate outputs, which we hypothesize will be more likely produced from highprobability tokens. The cost of all experiments using the OpenAI A∼PI$1w0a0s.
We evaluate each model on two datasets. First, we wish to study the models’ basic ability to use MIP to correctly identify metaphorical words. To this end, we use the Trope Finder (TroFi) dataset3][, which consists of sentences drawn from Wall Street Journal issues published between 1987 and 1989. Each sentence contains one of a list of fity words whose usage is annotated as literal or non-literal. We only consider sentences from the dataset that have been labeled by humans and take their annotations as ground truth, excluding all sentences labeled only by the TroFi clustering system. This leaves us with 3,736 sentences. Each model is then prompted as described above for each sentence and evaluated only on the words annotated in TroFi. A response is marked as correct if the model says “YES” there is a more basic meaning for the word of interest and the label from TroFi is “literal” and vice versa. Occasionally, the models provide labels for a phrase in the input sentence instead of individual words; the label provided for the entire phrase is then applied to the word of interest if necessary. If a model 1gpt-3.5-turbo-0125 2gpt-4-turbo-2024-04-09 3gpt-4o-2024-05-13 does not provide a label for a word, we act as if the incorrect label was given.
However, the TroFi dataset does not permit for a very fine-grained analysis of the models’ capabilities. In order to facilitate such an analysis, we create a new dataset consisting of example sentences pulled from Lakof and Johnson’s original work on C1M6]T. W[e collect all full sentence examples from the text that we believe can be understood as metaphorical without context, leaving us with 544 sentences, which we refer toMasWtLhBedataset. The dataset is made publicly available, along with all model outputs and human ann4otations.
Each model is again prompted to provide word-level annotations for every sentence. Then, we perform detailed qualitative analysis on each model’s output for a subset of 100 sentences. Specifically, we evaluate each output on five binary categories: • L&J Metaphor(s) – Identified : Whether the model has correctly identiafielld metaphors highlighted by Lakof and Johnson. For metaphorical phrases, the model’s response is considered correct if it has identified at at least one key word as having a more basic meaning. • L&J Metaphor(s) – Correct Basic Meanings: Whether the model has provided a correct, or plausible, more basic meaningafllomretaphorical words identified in the category above. A label of 1 is applied only if all metaphors from Lakof and Johnson have been correctly identified. • Additional Annotations: Whether the model has labeled any words not highlighted by Lakof and Johnson as having a more basic meaning. • Additional Annotations – Metaphorical: Whetherall the additional words labeled by the model are plausibly metaphorical or may have a more basic meaning. • Additional Annotations – Correct Basic Meanings: Whether the model has provided a correct, or plausible, basic meaningallfoardditional labeled words. It is possible for models to give the correct basic meaning for a word even if it is not plausibly metaphorical.
All annotations are performed by one of the authors, using MIP as a guideline. Confusing or difÏcult annotations are then discussed between both authors till a consensus is reached. These annotations are necessarily subjective, and we do not claim that they represent a ground truth. Instead, since these evaluations are informed by the same procedure used to create the 4https://github.com/rmatouschekh/science-is-exploration
TroFi Dataset Model Confusion Matrices 4-turbo outputs, we claim that they allow us to studpylatuhsiebility of the models’ performance, if not strictly thaecircuracy. In future work, further evaluation of model outputs with multiple trained annotators may allow for a more concrete analysis of model performance.
On the TroFi dataset, all models demonstrate an ability to distinguish between literal and nonliteral word usage with generally better than chance performan1c).eT(Thaubs,ltehey seem to be able to apply the MIP procedure for identifying metaphors and determine computationally whether a word has a more basic meaning. This is surprising since, as discussed above, basic word meanings are not necessarily the most frequent. Nothing about the language modeling approach to pre-training models guarantees that a model would be able to diferentiate between the most frequent and most basic meaning of a word.
We also find, however, that the models are prone to over-labeling words as metaphorical (Figure3). Whereas all three models achieve high recall for metaphor identification, they frequently label words used literally as having a more basic meaning, leading to low recall for the ‘literal’ category and low precision for the ‘metaphorical’ c3a.t5e-gtourryb.o particularly struggles with accurately identifying literal word usage. It is worth noting that, although there are clear cases where the models have failed to properly apply MIP, there may also be a disconnect between annotations in the TroFi corpus and what is considered metaphorical by the MIP procedure, as MIP was not used for annotation in TroFi.
For the annotated subset of the MWLB dataset, all the models found all the Lakof and Johnson metaphors in over 60% of sentences and further provided a correct basic meaning for the metaphorical words in over 50% of sentences (F4)i.gu4oreperformed the best on this task, achieving a remarkable performance given the subtlety and complexity of many of the metaphors and the cultural knowledge required to identify both their metaphoricity and their basic meanings.4o’s high performance may be due in part to the fact that it was most likely to label words as metaphorical overall, having annotated additional words in the largest percent4o age of sentences (Figur5e). In only 48% of sentences did these additional annotations identify only plausibly metaphorical words. Nonetheless, the inclusion of correct basic meanings for such a large proportion of the Lakof and Johnson metaphors suggests that MIP was overall being correctly applied and true ‘knowledge’ was being demonstrated, not just random chance.
3.5-turbo struggled the most at replicating the desired output structure, occasionally excluding words or truncating sentences, inconsistently formatting lists, and sometimes dropping the parenthetical explanations. This afected its accuracy, as the model sometimes failed to provide annotations for words central to the metaphors of interest. A4d-dtiutriobnoally, was least likely to annotate a word as having a more basic meaning5)(.FiTguhrise meant it identified fewer Lakof and Johnson metaphors, but a greater proportion of the additional metaphors it annotated were plausible. The basic meanings provi4d-etdurbbyo were usually of high quality, and tended to be more accurate and specific than those prod3u.c5e-dtbuyrbo. Despite the benefits, however, this model’s ‘caution’ led it to perform wor4seo tohvaernall.
Each model struggled to correctly identify when smaller function words were used metaphorically, particularly prepositions like ‘in,’ . This made the models worse at identifying so-called container metaphors, such as life is a container and activities are containers. For example, in the sentence “Thati n’sthecenter of myfield of vision.”, labeled by Lakof and Johnson as visual fields are containers, the word ‘in’ was overlooked as metaphorical by all three models.
The models were also challenged by metaphors in which an entity is being treated as a diferent kind of object, like instances of personification, place for institution, or producer for product metaphors. However, the GPT-4 models had a much greater ability to detect and provide accurate basic meanings for these. For example, in “Let’s not let Thailand become another Vietnam.”, both4-turbo and4o correctly identified ‘Vietnam’ as a metaph4oore.xplained that the more basic meaning of the word “refers to the country, whereas in this context it refers to a situation similar to the Vietnam War4”-atnudrbo provides a similar response. Likewise, in“I hate to reaHdeidegger.”, 4o recognized that ‘Heidegger’ is being used metaphorically and stated that the “more basic meaning refers to the person Martin Heidegger, a German philosopher, rather than his works.” Identifying and explaining both of these metaphors requires a nuanced understanding of both the semantics of the sentence and the cultural context surrounding them. % of Samples with Additional Metaphors Labeled 4o The models cannot always perform this analysis (all three miss that ‘the Alamo’ is metaphorical in “Rememberthe Alamo!”), but it is impressive that they are ever able to do so.
In addition, forcing the models to annotate word-by-word makes it challenging for them to identify metaphors comprising multi-word units. For example, it is difÏcult to determine which words should be marked as having more basic meanings in the sentGeentctehe“most out of life.” Analyzing texts word-by-word can also make identifying and providing the basic meanings of metaphorical compound words more complex. For example, both ‘underage’ and ‘brainchild’ cause problems f3o.r5-turbo; it does not recognize that utnhdeer of underage is metaphorical and it says that the more basic meaning of brainchild is “a child conceived in the mind.” In contras4t-,turbo says that the “more basic meanings of ‘brain’ and ‘child’ are more concrete and related to physical objects or beings” and notes that the “more basic meaning of ‘under’ is physically beneath something” for unde4roargeec. ognizes both words as metaphorical, but provides worse basic meanings.
While the models are clearly fallible, they nevertheless demonstrate an impressive ability to synthesize semantic, syntactic, and cultural information. They are frequently able to recognize when a word is being used metaphorically and often provide a correct basic meaning for the word. This ability sometimes holds even for nuanced and complex metaphors.
We find that large, generative LMs are capable of applying the classic metaphor annotation procedure, MIP. In doing so, they demonstrate an ability to discern the “basic meaning” of words and thus a depth of linguistic understanding that is not obviously gleaned from language modeling pre-training. Notably, they also demonstrate that LLMs are able to execute linguistic procedures designed for human annotators. This capacity means that generative LLMs are a promising tool for large-scale computational research on conceptual metaphors, which has previously been largely infeasible. In addition, it suggests that further research on how metaphoricity is learned by models may provide insight into their ability to acquire complex linguistic knowledge that often relies on people’s embodied experiences. These findings also suggest several avenues for future research, including studies into where information about words’ basic meanings or metaphoricity is stored by models, an exploration of open source models ability to annotate for conceptual metaphor, and research on operationalizing other linguistic procedures for execution by LLMs.
We would like to thank Lavinia Cerioni for her help in brainstorming this project and our colleagues at the Center for Humanities Computing for their support and advice. Additionally, part of the computation done for this project was performed on the UCloud interactive HPC system, which is managed by the eScience Center at the University of Southern Denmark.