Introduction

The question of whether computational methods can be used as creative devices can be traced back to the beginning of computers when Ada Lovelace wondered about the endless possibilities of automatic calculators [2]. Even though Artificial Intelligence techniques have largely been used in creative applications [3], the evaluation of such creative outputs remains problematic [4]. In this work, we propose the generation of Vossian Antonomasias (VAs) as a benchmark for exploring the creativity of AI methods.

VAs are a popular stylistic device for describing one entity by referring to another, typically in a witty and resourceful manner. A VA consists of three parts: a target entity 𝐴, a source entity 𝐵, and a modifier 𝐶, and is generally expressed as 𝐴 is the 𝐵 of 𝐶.

A meaningful VA requires a non-trivial degree of creativity and extensive knowledge of the specifics of the target entity. One has to identify a set of salient characteristics of 𝐴 that is similarly, or even more prominently realised by 𝐵. It is fundamental, however, that 𝐴 and 𝐵 differ when compared using the modifier 𝐶. For instance, in the sentence "Nacho Figueras is the Brad Pitt of polo players", Ignacio Figueras, among the most famous polo players in the world, is compared to the actor Brad Pitt due to his appearance. 1 VAs are often used in many journalistic genres and frequently appear in headings, as they can be both informative, enigmatic, and entertaining. In general, 𝐵 is a well-known, widely recognized entity. Through its popularity, the writer encourages readers to classify 𝐴 as similar to 𝐵, despite the difference 𝐶.

In this work, we present a method to automatically generate VAs by exploiting the latent semantic capabilities of vector space embeddings. We extract potential candidates for 𝐵 by using SPARQL queries over Wikidata. We rely on a heuristic method to select entities that can be classified as popular. By relying on publicly available Knowledge Graph Embeddings (KGE) trained on Wikidata, we compute the vector representations for an arbitrary 𝐴, the identified set of 𝐵 and a restricted set of 𝐶. We experiment with different operations between vectors to select the best 𝐵 candidate.

In order to investigate the efficacy of each experiment, we compare the use of KGE with word embeddings obtained from large corpora of text [5]. Additionally, given the recent surge of Large Language Models (LLM) to mine creative analogies [6], we rely on ChatGPT [7] as an additional baseline. We evaluate each method through a user-evaluation study.

The paper can be summarised as:

1. Identification of a suitable pool of candidates that can serve as 𝐵 elements; 2. Proposal of a novel method to automatically generate Vossian Antonomasias; 3. Evaluation of the proposed method using a user evaluation study;

The paper is organised as follows: in Section 2 we describe related work, which is followed by the presentation of the implemented method in Section 3. In Section 4, we discuss the results produced by the methods of Section 3 and present the outcomes of the user evaluation in Section 5. We finish by drawing conclusions and providing an outlook in Section 6.

Related Work

There has been limited research on automatically detecting Vossian Antonomasias in written text. The authors of [1] demonstrated that by using Wikidata, they were able to overcome the shortcomings of available Named Entity Recognition (NER) tools and confirmed that VA is a linguistic and cultural phenomenon. Through quantitative VA explorations, they were able to capture the phenomenon as a whole, encompassing the source, target, and, when available, modifier. Their approach involves searching for a network of individuals interconnected by diverse modifiers, where the nodes can function as either sources or targets. This network aids in understanding hidden patterns of role models, revealing how they vary across countries and languages. However, a limitation they acknowledged is their reliance on the most prevalent pattern of VA, namely, "the...of" which resulted in the omission of numerous expressions extracted from the New York Times (NYT) corpus. For instance, notable phrases such as "the American Oscar Wilde" and "Harlem's Mozart" were overlooked despite their significance. Another approach for the extraction of VAs is presented by [8]. The focus is on the extraction of the target by using coreference resolution and visualising the connections between the source and target entities extracted in the VAs in form of a web demo. The authors of [9] use neural networks for the end-to-end detection of VAs resulting in two models: one for binary sentence classification and another for sequence tagging of all parts of the VA on the word level.

As opposed to the work described above, our approach focuses on the generation of VAs rather than their detection. Similarly to the approach of [1], our method only focuses on the pattern "the...of" by exploiting the latent semantic space of Knowledge Graph Embeddings and word embedding methods. Knowledge Graph Embedding compute a vectorial approximation of the originating Knowledge Graph through the use of various geometrical intuitions. In TransE [10] predicates and entities are modelled as translations in the vector space. Given a triple (ℎ, 𝑟, 𝑡), the vector embedding of head entity ℎ, predicate or relation 𝑟, and tail entity 𝑡 are computed to minimise the quantity |ℎ + 𝑟 − 𝑡| -i.e. 𝑡 should be close to the ℎ + 𝑟. Increasingly complex methods have been presented in literature [11].

Based on the distributional hypothesis, word embeddings [12] are used to compute vectors based on the distribution of words in large corpora of text, such as GloVe [13], where the representation is obtained using word co-occurrence statistics or word2Vec [14], where words with similar contextual distribution are approximated as similar vectors.

Methodology

This section details our proposed approach to automatically generate VAs, as outlined in Section 1. Figure 1 provides a high-level summary of our approach.

Wikidata as a Knowledge Resource

We rely on Wikidata to identify a set of candidates that can serve as 𝐵 entities. This allows us to benefit from a large number of triples 2 which enables us to select a sufficient sample of candidates for every component of the Vossian Antonomasias. Additionally, Wikidata provides a more structured and consistent data model when compared to similar resources, such as DBPedia [16]. Moreover, we can leverage the language-independent design of Wikidata as opposed to DBPedia [16] to ensure that the targets are widely popular.

We first extract the entities that will be used as 𝐵 candidates by means of SPARQL queries. We retrieve popular fictional characters and popular humans with the query of Listing 1. Given the large number of triples retrieved by both queries, they are executed on the Semantic Builders3 SPARQL endpoint rather than on the regular Wikidata endpoint 4 . This allows us to overcome the querying timeout imposed by Wikidata and extract 3815 entities. The extracted entities are used as the set of candidates for 𝐵 in a VA. We rely on a heuristic method to compute the popularity of an entity: the number of worldwide available Wikipedia articles in distinct languages for an entity as a proxy for its popularity. Given 𝑡 the number of translations of one entity, we found 𝑡 ≥ 70 for real-world individuals and 𝑡 > 30 for fictional characters to be a good estimate.

VA Generation using Vector Representations

We generate VA sentences by using geometrical transformations on the latent space provided by vector embeddings. Given an arbitrary 𝐴, we constrain the modifier 𝐶 to be the occupation of the entity 𝐴. The underlying assumption of the proposed model is that, despite their different occupations, 𝐴 and 𝐵 need to be similar with respect to their salient features. For this reason, given a particular 𝐴, all those entities 𝑏 ∈ 𝐵 that share the same modifier (i.e. the same occupation) are excluded from the pool of candidates. This brings us closer to ensuring the accurate selection of 'B' in accordance with the conditions specified in section 1. ) the embedding vector of the predicate 𝐶 that denotes the occupation of an entity (i.e. P106 in Wikidata), the translationbased method first disregards (i.e., "subtracts") A's occupation C obtaining:

⃗ 𝑎 ′ = ⃗ 𝑎 − ⃗ 𝑐(1)

Then, we define the fitness 𝑓 (where smaller is better) of a candidate ⃗ 𝑏 ′ ∈ ℬ as the Euclidean distance between ⃗ 𝑎 ′ and ⃗ 𝑏 ′ , i.e.,

𝑓 (⃗ 𝑎, ⃗ 𝑏′ ) = | ⃗ 𝑎 ′ − ⃗ 𝑏 ′ | = √ 𝑑 ∑ 𝑖=0 (𝑎 ′ 𝑖 − 𝑏 ′ 𝑖 ) 2 (2)

where 𝑑 is the dimension of the vector embedding space. The projection-based method relies on a different assumption. Informally, we would like to compute the fitness 𝑓 (⃗ 𝑎, ⃗ 𝑏 ′ ) on a subspace of the whole embedding space where every information related to the occupation of the entities is ignored. We compute the projection of ⃗ 𝑎 and ⃗ 𝑏 ′ to this subspace, which is a hyperplane perpendicular to ⃗ 𝑐, using

𝜋 ⃗ 𝑐 ( ⃗ 𝑥) = ⃗ 𝑥 − ( ⃗ 𝑥 ∘ ⃗ 𝑐 ⃗ 𝑐 ∘ ⃗ 𝑐 ) ⃗ 𝑐(3)

where ∘ denotes the inner product and ⃗ 𝑥 is either ⃗ 𝑎 or ⃗ 𝑏 ′ . The fitness function 𝑓 is hence adjusted to the cosine distance between the projections of ⃗ 𝑎 and ⃗ 𝑏 ′ , formally 2a) and the projection-based method (Figure 2b).

𝑓 (⃗ 𝑎, ⃗ 𝑏 ′ ) = 𝜋 ⃗ 𝑐 (⃗ 𝑎) ∘ 𝜋 ⃗ 𝑐 ( ⃗ 𝑏 ′ ) |𝜋 ⃗ 𝑐 (⃗ 𝑎)||𝜋 ⃗ 𝑐 ( ⃗ 𝑏 ′ )|(4)

The fitness functions in Equation ( 2) and Equation ( 4) can be seen as similarity functions between two entities. A suitable 𝐵 can be extracted by taking the entity that minimises such distance. Intuitively, in the translation-based approach, the vector representing 𝐵 is supposed to be similar to the one of 𝐴 after we "translate away" or "subtract" the characteristics pertaining to 𝐶 using Equation (1), while in projection-based method by applying Equation (3), we "project away" such characteristics.

As addressed in Section 1, a good VA needs to be a creative sentence. While it is difficult to assess creativity in an objective manner, it has been argued that among the many characteristics, a creative output needs to display novelty when compared to others [17]. Given a set of entities B that minimises the fitness function 𝑓, we propose to further rank such entities by using their 𝐿 1 norm. The intuition is that among all the candidates in B , the ones with a greater distance from the origin of the vector space are the most "extremal" ones.

Figure 2 depicts a simplified illustration of Equation 2 and Equation 4. Using t-SNE [18], we reduce the dimensionality of embedding vector ⃗ 𝑎 of a sample entity A, a set ℬ of embedding vectors of each candidate entity for 𝐵, and ⃗ 𝑏 ∈ B vectors.

Given a particular entity 𝐴 and the corresponding entity 𝐵, selected with one of the proposed methods, we generate an assertional VA following template: {A} {verb} the {B} of {C}. If the entity 𝐴 has an entry in Wikidata that certifies its death, we set verb to was, else we use is.

Purely LMM-based Baseline via ChatGPT

Finally, we use ChatGPT [7], a Large Language Model, as a baseline for VA generation. Through extensive experimentation, we found the prompt that obtains the best result to be Following the discussion of Section 1, we argue that an effective prompt for VA needs to display the following properties:

• 𝐵 should not share characteristics with 𝐶;

• 𝐴 and 𝐵 should share at least one salient characteristic; • 𝐴 and 𝐵 should be popular enough to draw the analogy in the context of 𝐶 Provide 10 Vossian Antonomasias for <Name of A>, where she is equated with another person. Each of the phrases should have the structure "<Name of A> is the [person name] of [profession]", where [profession] must not characterize [person name]. Provide a very short justification for each example.

Table 1

ChatGPT prompt used to obtain a VA.

Experimental Setting

As briefly addressed in Section 1, we experiment with two different methods to obtain vector representations of an entity: Knowledge Graph Embeddings (KGE) and Word Embeddings (WE).

We employ TransE [10] as KGE method. We directly reuse the publicly available model shared by GraphVite [19] and trained on the Wikidata-5M dataset [20]. For the WE method, we employ word2vec [14] and GloVe [13] provided by gensim [5].

Finally, we leverage the use of meta-embedding techniques, i.e. combining different embedding methods together [21], to exploit the main advantages of both methods. We combine KGE and WE by means of concatenation and averaging. When averaging two vectors with different dimensionality, we apply zero-padding [21]. Note that, even though they are supposed to converge to a similar semantic, the latent space represented by a method might differ drastically from other methods. To prevent a drastically higher influence of one method over the other, we normalise both methods by their 𝐿 2 norm before combining them.

To allow interested readers to try out the different methods themselves, we set up a demonstrator website available at https://antonomasia.informatik.uni-bremen.de/.

Results and Evaluation

Due to the highly diverse nature of VAs, we decided to test the quality of the output with human evaluation. A small selection of examples generated by the methods described in Section 3 is presented in Table 2.

The selection on Table 2 shows that ChatGPT, while being very creative when it comes to the description of the domain, does not perform well in identifying a proper 𝐵 that does not share characteristics with the modifier 𝐶, such as in the sentence "Bill Gates is the Einstein of Societal Transformation". This phenomenon particularly occurs with politicians or writers. Despite the explicit request in the prompt of Table 1, ChatGPT did not manage to adapt the chosen entity. The results generated by the KGE, WE and their combination mostly meet the mentioned criteria, even though some exceptions occur, such as in the sentence "Angela Merkel is the Eva Braun of politics. ".

User Evaluation

The method described in Section 3 allows combining several different techniques to generate a VA. For the user evaluation, after manual experimentation, we restricted the set of techniques to the ones described in Table 3. The presented selection allows us to evaluate the importance of different assumptions, such as whether the methods based on the distributional hypothesis can complement content-based methods. Moreover, we are able to assess whether the presented methods can overcome the issues of ChatGPT, namely the difficulty of selecting 𝐴 and 𝐵 from a different domain dictated by 𝐶. We identify six individuals that will be used as 𝐴: Nelson Mandela, Angela Merkel, Mark Twain, Albert Einstein, Bill Gates and Ronald Reagan. Those individuals are both part of the entities extracted using the query in Listing 1 and the real-world samples from the New York Times [1] and Der Umblätterer5 corpora. As mentioned in Section 3, we contain 𝐶 to the profession of the entity.

To recruit participants for our study, we distributed a flyer as advertisement and shared it with colleagues and friends who themselves distributed this further. The study can be done online without any supervision. Due to the specificity of Vossian Antonomasia, we present a definition on the front page of the study:

Vossian antonomasias refer to someone by a special characteristic instead of their name.For example, calling Bill Gates "the Henry Ford of the computer age" highlights his influence as entrepeneur and his effect on the development of technology. It is a way to describe someone by an important quality they possess.

The study was open for one week. We provide each participant with 21 sentences. We randomly sample 3 𝐴s from the set described above and provide 7 VAs, one for every generation method of Table 3. The selected VAs are randomly sampled from the top 10 sentences identified by the method used. The participant is asked to judge each VA on three aspects: how well the description fits, how understandable it is and how original the VA is. These three aspects can be ranked on a Likert Scale ranging form 1 to 5. After that, the knowledge about the source and the target will be inquired in the form of questions: we ask how well the participant knows the individuals. The possible answers are I know who that person is, I have heard of the name but I cannot relate it to anything, and I have never heard of that name before. This ensures a distinction between a negative rating caused by ignorance of the output's components and the lack of a proper connection between the source, the target and the modifier.

Results

Through the user-evaluation test, we obtain a set of 207 human evaluations on automatically generated VAs, provided by 29 unique annotators. The sentences presented to the participants 3 with 1 being the best and 5 being the worst rating.

are repeated to avoid a random evaluation. The inter-annotator agreement, computed using Cohen's Kappa score [22], is 0.0491 on average. Such a low score highlights the difficulty in evaluating VAs since they greatly depend on the reader's knowledge, cultural reference and degree of familiarity with the selected subject.

In Figure 5 the distribution of ratings among the methods reported in Table 5 is reported. Intuitively, whose distribution is skewed towards low ratings (represented in green) should be considered the best-performing method. Using the translation-based meta-embedding method outperforms all the other methods. Interestingly, the baseline provided by ChatGPT performs worse than any other method. While this might be reconducted to the lack of explicit VA-related knowledge that the underlying LLM is based on, it can also be argued that the prompt that we propose to use is not perfectly suited for this task. We will further address this aspect in Section 6. At first glance, the best method turns out to be word2vec using the translation technique, which results in 74 VAs rated with a score of 1. It needs to be argued, however, that using this method results also in a high variance in the results. Indeed, the same method is also the one that obtains the highest amount of low-rated VAs. When taking into account the whole distribution, the projection-based method TransE achieved the best performance: the lowest number of "bad" VAs is obtained with such a method. Indeed, this is the method that achieves the best overall performances, as can be seen in Table 4.

Table 5

Overview of the mean rating for each user question. The mean rating for each confidence rating (𝜇 Table 5 provides an overview of the mean rating 𝜇 𝑖 for each method, where 𝑖 represents the set of sentences for which an evaluator expressed specific confidence in the knowledge of 𝐴 and 𝐵. Those results are complementary to the ones of Figure 5. Interestingly, the best overall method, projection-based TransE, does not classify as the best in any specific user question. Depending on the target task, one model can be considered better than the other, even though projection-based TransE guarantees the most consistent results.

Conclusions and Discussion

We looked at generating Vossian Antononmasias by using embeddings and LLM as a way to characterize the creativity of AI. Our approach has resulted in creative examples of VAs, which proves that both methods are suitable for solving such tasks. Since the lack of a clear definition of creativity prevents a quantitative evaluation of the results, we conducted a manual qualitative analysis of the results which highlighted several different weaknesses. The information bias that is inherent to Wikidata results in VAs that are mostly focused on Western culture. While this might be tampered by penalising some entities, we argue it would only partially solve the issue. A different approach, which takes into account the semantic representation of each entity, can help overcome such issues, making the creation process transparent and explainable.

The human evaluation described in Section 5 meaningful insights into the effectiveness of our methods. Firstly, they generally perform better than the ChatGPT baseline, which fails in the generation of original and understandable VAs. Moreover, the results of Figure 5 and Table 5 show how the use of Knowledge Graph Embeddings is generally to be preferred over the methods, such as word embeddings and meta-embeddings. However, the low inter-annotator agreement shows that rating Vossian Antonomasias is highly subjective and most probably depends on not only the knowledge of an entity but also knowledge of the domain the entity refers to. This could be addressed by filtering the human annotators into groups according to their domain knowledge before rating the sentences.

The focus on the occupation as a similarity measure resulted in several complications, such as the use of semantically similar occupations like television actor and actor in the generated sentences. Additionally, since some entities hold multiple occupations, a more accurate estimation of their primary occupation needs to be investigated. A possible solution is to aggregate the representation vectors of all their occupations instead of selecting a single occupation. Similarly, fictional characters are sometimes compared to their real-life actors. A possible solution is to impose a minimum distance between vectors that are too close. An orthogonal solution is to consider other criteria when comparing entities, such as achievement or awards. Apart from famous people, famous locations or events along with their appropriate modifiers could be added to increase the sample size and achieve a greater variability in the results.

The mentioned limitations of the evaluation and sampling of entities show that the generation of Vossian Antonomasia with an open-domain approach proves to be rather difficult. Instead, we suggest focusing on specific domains, thereby using fine-tuning of the embeddings or different embedding methods to overcome the mentioned shortcomings.

Additionally, we envision integrating an LLM in the pre-evaluation step by evaluating the VAs that have been generated by combining our proposed methods. The idea is to list the salient similarities and differences between 𝐴 and 𝐵 for a given VA by looking at their characteristic properties. Following recent LLM-prompting studies [23,24], we plan on performing additional manual or automatic prompt engineering [25,26]. This can lead to more effective results, since any change in the prompts may affect significantly the quality of the output.

Moreover, an interesting approach is to perform knowledge injection [27,28] into a LLM, following a neuro-symbolic approach to use the language model's potential and control for the criteria defining Vossian Antonomasia. The knowledge available to an LLM like ChatGPT is currently limited regarding whether an entity has recently died or is a fictional character based on the time it was trained. The injection of structured knowledge can help overcoming this issue.