A Vossian Antonomasia (VA) is a stylistic device used to describe a person (or, more generally, an entity) in terms of a well-known person and a modifying context. For instance, the Norwegian chess world champion Magnus Carlsen was described as “the Mozart of chess” [1]. All VAs follow the pattern where a source (e.g., “Mozart”), is used to describe a target, (e.g., “Magnus Carlsen”), and the transfer of meaning is “channeled” through the use of the modifier “of chess”. Although this rhetorical figure is well-known, there has not yet been a dedicated study of targeted automatic or semi-automatic methods to generate and judge the appropriateness of VAs using large Knowledge Graphs (KGs) such as Wikidata. In our work, we propose the use of vector space embeddings - both KG-based and text-based - for producing VAs. For comparison, we contrast our findings with a purely LLM-based approach, wherein VAs are obtained from ChatGPT using a reasonably engineered prompt. We provide a publicly available GitHub repository1 for the implementation of our method and a website2 that allows testing the proposed methods.
CEUR
ceur-ws.org (Semi)-Automatic
The question of whether computational methods can be used as creative devices can be traced
back to the beginning of computers when Ada Lovelace wondered about the endless possibilities
of automatic calculators [
VAs are a popular stylistic device for describing one entity by referring to another, typically in a witty and resourceful manner. A VA consists of three parts: a target entity , a source entity , and a modifier , and is generally expressed as
is the of .
A meaningful VA requires a non-trivial degree of creativity and extensive knowledge of the specifics of the target entity. One has to identify a set of salient characteristics of that is similarly, or even more prominently realised by . It is fundamental, however, that and difer when compared using the modifier . For instance, in the sentence “Nacho Figueras is the Brad Pitt of polo players”, Ignacio Figueras, among the most famous polo players in the world, is compared to the actor Brad Pitt due to his appearance.1
VAs are often used in many journalistic genres and frequently appear in headings, as they can be both informative, enigmatic, and entertaining. In general, is a well-known, widely recognized entity. Through its popularity, the writer encourages readers to classify as similar to , despite the diference .
In this work, we present a method to automatically generate VAs by exploiting the latent semantic capabilities of vector space embeddings. We extract potential candidates for by using SPARQL queries over Wikidata. We rely on a heuristic method to select entities that can be classified as popular. By relying on publicly available Knowledge Graph Embeddings (KGE) trained on Wikidata, we compute the vector representations for an arbitrary , the identified set of and a restricted set of . We experiment with diferent operations between vectors to select the best candidate.
In order to investigate the eficacy of each experiment, we compare the use of KGE with word
embeddings obtained from large corpora of text [
The paper can be summarised as: 1. Identification of a suitable pool of candidates that can serve as elements; 2. Proposal of a novel method to automatically generate Vossian Antonomasias; 3. Evaluation of the proposed method using a user evaluation study;
The paper is organised as follows: in Section 2 we describe related work, which is followed by the presentation of the implemented method in Section 3. In Section 4, we discuss the results produced by the methods of Section 3 and present the outcomes of the user evaluation in Section 5. We finish by drawing conclusions and providing an outlook in Section 6. 1This and many more examples of VAs extracted from a newspaper corpus can be found at https://vossanto.weltliteratur.net/emnlp-ijcnlp2019/vossantos.html
There has been limited research on automatically detecting Vossian Antonomasias in written
text. The authors of [
As opposed to the work described above, our approach focuses on the generation of VAs
rather than their detection. Similarly to the approach of [
Based on the distributional hypothesis, word embeddings [12] are used to compute vectors based on the distribution of words in large corpora of text, such as GloVe [13], where the representation is obtained using word co-occurrence statistics or word2Vec [14], where words with similar contextual distribution are approximated as similar vectors.
This section details our proposed approach to automatically generate VAs, as outlined in Section 1. Figure 1 provides a high-level summary of our approach.
We rely on Wikidata to identify a set of candidates that can serve as entities. This allows us to benefit from a large number of triples 2 with broad coverage of encyclopedic knowledge, 2According to Wikidata’s statistics the knowledge graph currently contains 104,204,236 items. [15] Target (A) is the source (B) of modifier (C)
Extract Extracting B’s from
Wikidata using SPARQL queries
Exploration Knowledge Graph Embeddings (KGE)
Text-based
Embeddings Use Pre-Trained
Models Mapping B’s into embedding space
Evaluate Applying similarity measures to identify appropriate B’s which enables us to select a suficient sample of candidates for every component of the Vossian Antonomasias. Additionally, Wikidata provides a more structured and consistent data model when compared to similar resources, such as DBPedia[16]. Moreover, we can leverage the language-independent design of Wikidata as opposed to DBPedia [16] to ensure that the targets are widely popular.
We first extract the entities that will be used as candidates by means of SPARQL queries.
We retrieve popular fictional characters and popular humans with the query of Listing 1. Given the large number of triples retrieved by both queries, they are executed on the Semantic Builders3 SPARQL endpoint rather than on the regular Wikidata endpoint4. This allows us to overcome the querying timeout imposed by Wikidata and extract 3815 entities. The extracted entities are used as the set of candidates for in a VA. We rely on a heuristic method to compute the popularity of an entity: the number of worldwide available Wikipedia articles in distinct languages for an entity as a proxy for its popularity. Given the number of translations of one entity, we found ≥ 70 for real-world individuals and > 30 for fictional characters to be a good estimate.
We generate VA sentences by using geometrical transformations on the latent space provided by vector embeddings. Given an arbitrary , we constrain the modifier to be the occupation of the entity . The underlying assumption of the proposed model is that, despite their diferent occupations, and need to be similar with respect to their salient features. For this reason, given a particular , all those entities ∈ that share the same modifier (i.e. the same occupation) are excluded from the pool of candidates. This brings us closer to ensuring the accurate selection of ’B’ in accordance with the conditions specified in section 1. 3https://semantic.builders/ 4https://query.wikidata.org/ PREFIX wdt: <http://www.wikidata.org/prop/direct/> PREFIX wd: <http://www.wikidata.org/entity/> SELECT ?item ?itemLabel ?occupation ?sitelinks WHERE { ?item wdt:P31 <type>; wdt:P106 ?occupation; wikibase:sitelinks ?sitelinks .
FILTER(<threshold> < ?sitelinks).
SERVICE wikibase:label {bd:serviceParam wikibase:language ↪ "[AUTO_LANGUAGE],en".}} Listing 1: SPARQL Query to extract candidate entities for . <type> and <threhsold> are replaced by wd:Q15632617 and 30 for fictional characters and wd:Q5 and 70 for humans.
We propose two diferent methods for the selection of the best candidate: a translation-based approach and a projection-based approach. The translation-based approach follows the intuition of TransE and word2vec: given ⃗, ⃗ the vector representations of two arbitrary entities and given ⃗ the vector that represents a predicate holding between ⃗ and ⃗, it has been observed that ⃗+ ⃗≈ ⃗ .
Given ⃗= ( 1, … , ) the embedding vector of an arbitrary target entity A, ℬ the set of embedding vectors of each candidate entity for , and ⃗= ( 1, … , ) the embedding vector of the predicate that denotes the occupation of an entity (i.e. P106 in Wikidata), the translationbased method first disregards (i.e., “subtracts”) A’s occupation C obtaining: Then, we define the fitness distance between ⃗′ and ⃗′, i.e.,
(where smaller is better) of a candidate ⃗′ ∈ ℬ as the Euclidean where is the dimension of the vector embedding space.
The projection-based method relies on a diferent assumption. Informally, we would like to compute the fitness (,⃗ ⃗′) on a subspace of the whole embedding space where every information related to the occupation of the entities is ignored. We compute the projection of ⃗ and ⃗′ to this subspace, which is a hyperplane perpendicular to ⃗, using where ∘ denotes the inner product and ⃗ is either ⃗ or ⃗′. The fitness function is hence adjusted to the cosine distance between the projections of ⃗ and ⃗′, formally ⃗ ( )⃗ = ⃗− ( ⃗∘ ⃗ ⃗∘ ⃗
) ⃗ (,⃗ ⃗′) = ⃗ ()⃗ ∘ ⃗ ( ⃗′)
| ⃗ ()⃗|| ⃗ ( ⃗′)| ⃗′ = ⃗− ⃗ (,⃗ ⃗′) = |⃗′ − ⃗′| = ∑( ′ − ′)2 √=0 (1) (2) (3) (4) (a) Equation 2 (translation-(b) Equation 4 (projection
based) illustrated. based) illustrated.
The fitness functions in Equation (2) and Equation (4) can be seen as similarity functions between two entities. A suitable can be extracted by taking the entity that minimises such distance. Intuitively, in the translation-based approach, the vector representing is supposed to be similar to the one of after we “translate away” or “subtract” the characteristics pertaining to using Equation (1), while in projection-based method by applying Equation (3), we “project away” such characteristics.
As addressed in Section 1, a good VA needs to be a creative sentence. While it is dificult to assess creativity in an objective manner, it has been argued that among the many characteristics, a creative output needs to display novelty when compared to others [17]. Given a set of entities ̂ that minimises the fitness function , we propose to further rank such entities by using their 1 norm. The intuition is that among all the candidates in ̂, the ones with a greater distance from the origin of the vector space are the most “extremal” ones.
Figure 2 depicts a simplified illustration of Equation 2 and Equation 4. Using t-SNE [18], we reduce the dimensionality of embedding vector ⃗ of a sample entity A, a set ℬ of embedding vectors of each candidate entity for , and ⃗∈ ̂ vectors.
Given a particular entity and the corresponding entity , selected with one of the proposed methods, we generate an assertional VA following template: {A} {verb} the {B} of {C}. If the entity has an entry in Wikidata that certifies its death, we set verb to was, else we use is.
Finally, we use ChatGPT [
Following the discussion of Section 1, we argue that an efective prompt for VA needs to display the following properties: • should not share characteristics with ; • and should share at least one salient characteristic; • and should be popular enough to draw the analogy in the context of Provide 10 Vossian Antonomasias for <Name of A>, where she is equated with another person. Each of the phrases should have the structure ”<Name of A> is the [person name] of [profession]”, where [profession] must not characterize [person name]. Provide a very short justification for each example.
As briefly addressed in Section 1, we experiment with two diferent methods to obtain vector representations of an entity: Knowledge Graph Embeddings (KGE) and Word Embeddings (WE).
We employ TransE [
Finally, we leverage the use of meta-embedding techniques, i.e. combining diferent embedding methods together [21], to exploit the main advantages of both methods. We combine KGE and WE by means of concatenation and averaging. When averaging two vectors with diferent dimensionality, we apply zero-padding [21]. Note that, even though they are supposed to converge to a similar semantic, the latent space represented by a method might difer drastically from other methods. To prevent a drastically higher influence of one method over the other, we normalise both methods by their 2 norm before combining them.
To allow interested readers to try out the diferent methods themselves, we set up a demonstrator website available at https://antonomasia.informatik.uni-bremen.de/.
Due to the highly diverse nature of VAs, we decided to test the quality of the output with human evaluation. A small selection of examples generated by the methods described in Section 3 is presented in Table 2.
The selection on Table 2 shows that ChatGPT, while being very creative when it comes to the description of the domain, does not perform well in identifying a proper that does not share characteristics with the modifier , such as in the sentence “Bill Gates is the Einstein of Societal Transformation”. This phenomenon particularly occurs with politicians or writers. Despite the explicit request in the prompt of Table 1, ChatGPT did not manage to adapt the chosen entity. The results generated by the KGE, WE and their combination mostly meet the mentioned criteria, even though some exceptions occur, such as in the sentence “Angela Merkel is the Eva Braun of politics.”.
The method described in Section 3 allows combining several diferent techniques to generate a VA. For the user evaluation, after manual experimentation, we restricted the set of techniques to the ones described in Table 3. The presented selection allows us to evaluate the importance of diferent assumptions, such as whether the methods based on the distributional hypothesis can complement content-based methods. Moreover, we are able to assess whether the presented methods can overcome the issues of ChatGPT, namely the dificulty of selecting and from a diferent domain dictated by .
We identify six individuals that will be used as : Nelson Mandela, Angela Merkel, Mark Twain,
Albert Einstein, Bill Gates and Ronald Reagan. Those individuals are both part of the entities
extracted using the query in Listing 1 and the real-world samples from the New York Times [
To recruit participants for our study, we distributed a flyer as advertisement and shared it with colleagues and friends who themselves distributed this further. The study can be done online without any supervision. Due to the specificity of Vossian Antonomasia, we present a definition on the front page of the study:
Vossian antonomasias refer to someone by a special characteristic instead of their name.For example, calling Bill Gates “the Henry Ford of the computer age” highlights his influence as entrepeneur and his efect on the development of technology.
It is a way to describe someone by an important quality they possess.
The study was open for one week. We provide each participant with 21 sentences. We randomly sample 3 s from the set described above and provide 7 VAs, one for every generation method of Table 3. The selected VAs are randomly sampled from the top 10 sentences identified by the method used. The participant is asked to judge each VA on three aspects: how well the description fits, how understandable it is and how original the VA is. These three aspects can be ranked on a Likert Scale ranging form 1 to 5. After that, the knowledge about the source and the target will be inquired in the form of questions: we ask how well the participant knows the individuals. The possible answers are I know who that person is, I have heard of the name but I cannot relate it to anything, and I have never heard of that name before. This ensures a distinction between a negative rating caused by ignorance of the output’s components and the lack of a proper connection between the source, the target and the modifier.
Through the user-evaluation test, we obtain a set of 207 human evaluations on automatically generated VAs, provided by 29 unique annotators. The sentences presented to the participants 5https://www.umblaetterer.de/datenzentrum/vossianische-antonomasien.html are repeated to avoid a random evaluation. The inter-annotator agreement, computed using Cohen’s Kappa score [22], is 0.0491 on average. Such a low score highlights the dificulty in evaluating VAs since they greatly depend on the reader’s knowledge, cultural reference and degree of familiarity with the selected subject.
In Figure 5 the distribution of ratings among the methods reported in Table 5 is reported. Intuitively, whose distribution is skewed towards low ratings (represented in green) should be considered the best-performing method. Using the translation-based meta-embedding method outperforms all the other methods. Interestingly, the baseline provided by ChatGPT performs worse than any other method. While this might be reconducted to the lack of explicit VA-related knowledge that the underlying LLM is based on, it can also be argued that the prompt that we propose to use is not perfectly suited for this task. We will further address this aspect in Section 6. At first glance, the best method turns out to be word2vec using the translation technique, which results in 74 VAs rated with a score of 1. It needs to be argued, however, that using this method results also in a high variance in the results. Indeed, the same method is also the one that obtains the highest amount of low-rated VAs. Average rating for each method. Lower is better. Best result is represented in bold.
When taking into account the whole distribution, the projection-based method TransE achieved the best performance: the lowest number of ”bad” VAs is obtained with such a method. Indeed, this is the method that achieves the best overall performances, as can be seen in Table 4. Overview of the mean rating for each user question. The mean rating for each confidence rating ( 1, 2, and 3) is reported alongside the overall mean rate 1∪2∪3 for each question. The best results for each criterion are highlighted in bold. Lower values indicate better performance. set of sentences for which an evaluator expressed specific confidence in the knowledge of and . Those results are complementary to the ones of Figure 5. Interestingly, the best overall method, projection-based TransE, does not classify as the best in any specific user question. Depending on the target task, one model can be considered better than the other, even though projection-based TransE guarantees the most consistent results.
We looked at generating Vossian Antononmasias by using embeddings and LLM as a way to characterize the creativity of AI. Our approach has resulted in creative examples of VAs, which proves that both methods are suitable for solving such tasks. Since the lack of a clear definition of creativity prevents a quantitative evaluation of the results, we conducted a manual qualitative analysis of the results which highlighted several diferent weaknesses. The information bias that is inherent to Wikidata results in VAs that are mostly focused on Western culture. While this might be tampered by penalising some entities, we argue it would only partially solve the issue. A diferent approach, which takes into account the semantic representation of each entity, can help overcome such issues, making the creation process transparent and explainable.
The human evaluation described in Section 5 provides meaningful insights into the efectiveness of our methods. Firstly, they generally perform better than the ChatGPT baseline, which fails in the generation of original and understandable VAs. Moreover, the results of Figure 5 and Table 5 show how the use of Knowledge Graph Embeddings is generally to be preferred over the methods, such as word embeddings and meta-embeddings. However, the low inter-annotator agreement shows that rating Vossian Antonomasias is highly subjective and most probably depends on not only the knowledge of an entity but also knowledge of the domain the entity refers to. This could be addressed by filtering the human annotators into groups according to their domain knowledge before rating the sentences.
The focus on the occupation as a similarity measure resulted in several complications, such as the use of semantically similar occupations like television actor and actor in the generated sentences. Additionally, since some entities hold multiple occupations, a more accurate estimation of their primary occupation needs to be investigated. A possible solution is to aggregate the representation vectors of all their occupations instead of selecting a single occupation. Similarly, ifctional characters are sometimes compared to their real-life actors. A possible solution is to impose a minimum distance between vectors that are too close. An orthogonal solution is to consider other criteria when comparing entities, such as achievement or awards. Apart from famous people, famous locations or events along with their appropriate modifiers could be added to increase the sample size and achieve a greater variability in the results.
The mentioned limitations of the evaluation and sampling of entities show that the generation of Vossian Antonomasia with an open-domain approach proves to be rather dificult. Instead, we suggest focusing on specific domains, thereby using fine-tuning of the embeddings or diferent embedding methods to overcome the mentioned shortcomings.
Additionally, we envision integrating an LLM in the pre-evaluation step by evaluating the VAs that have been generated by combining our proposed methods. The idea is to list the salient similarities and diferences between and for a given VA by looking at their characteristic properties. Following recent LLM-prompting studies [23, 24], we plan on performing additional manual or automatic prompt engineering [25, 26]. This can lead to more efective results, since any change in the prompts may afect significantly the quality of the output.
Moreover, an interesting approach is to perform knowledge injection [27, 28] into a LLM, following a neuro-symbolic approach to use the language model’s potential and control for the criteria defining Vossian Antonomasia. The knowledge available to an LLM like ChatGPT is currently limited regarding whether an entity has recently died or is a fictional character based on the time it was trained. The injection of structured knowledge can help overcoming this issue.
The project leading to this application has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101034440. This work was partially funded by the Klaus Tschira Foundation, grant number 40300928 and the French National Research Agency ANR DACE-DL project, grant number ANR-21-CE23-0019. United States, 2013, pp. 2787–2795. URL: https://proceedings.neurips.cc/paper/2013/hash/ 1cecc7a77928ca8133fa24680a88d2f9-Abstract.html. [11] S. Choudhary, T. Luthra, A. Mittal, R. Singh, A Survey of Knowledge Graph Embedding and Their Applications, CoRR abs/2107.07842 (2021). URL: https://arxiv.org/abs/2107.07842. arXiv:2107.07842. [12] F. Almeida, G. Xexéo, Word embeddings: A survey, CoRR abs/1901.09069 (2019). URL: http://arxiv.org/abs/1901.09069. arXiv:1901.09069. [13] J. Pennington, R. Socher, C. D. Manning, Glove: Global Vectors for Word Representation, in: A. Moschitti, B. Pang, W. Daelemans (Eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, ACL, 2014, pp. 1532–1543. URL: https://doi.org/10.3115/v1/d14-1162. doi:10.3115/v1/d14- 1162. [14] T. Mikolov, K. Chen, G. Corrado, J. Dean, Eficient Estimation of Word Representations in Vector Space, in: Y. Bengio, Y. LeCun (Eds.), 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL: http://arxiv.org/abs/1301.3781. [15] Wikidata statistics, https://www.wikidata.org/wiki/Wikidata:Statistics, 2023. Accessed on 2023-06-16. [16] D. Abián, F. Guerra, J. Martínez-Romanos, R. T. Lado, Wikidata and DBpedia: A Comparative Study, in: International KEYSTONE Conference, 2017. [17] G. Ritchie, Assessing Creativity, in: Proc. of AISB’01 Symposium, 2001. [18] L. Van der Maaten, G. Hinton, Visualizing Data using t-SNE., Journal of Machine Learning
Research 9 (2008). [19] Z. Zhu, S. Xu, M. Qu, J. Tang, Graphvite: A High-Performance CPU-GPU Hybrid System for Node Embedding, in: The World Wide Web Conference, ACM, 2019, pp. 2494–2504. [20] X. Wang, T. Gao, Z. Zhu, Z. Zhang, Z. Liu, J. Li, J. Tang, KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation, Trans. Assoc. Comput. Linguistics 9 (2021) 176–194. URL: https://doi.org/10.1162/tacl_a_00360. doi:10.1162/ tacl\_a\_00360. [21] D. Bollegala, J. O’Neill, A Survey on Word Meta-Embedding Learning, in: L. D. Raedt (Ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, ijcai.org, 2022, pp. 5402–5409. URL: https: //doi.org/10.24963/ijcai.2022/758. doi:10.24963/ijcai.2022/758. [22] J. Cohen, A Coeficient of AgI see you’re on the paper, I updated the table and will proofread everything now, do you think that are particulreement for Nominal Scales, Educational and Psychological Measurement 20 (1960) 37–46. [23] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, J. Tang, GPT Understands, Too, arXiv preprint arXiv:2103.10385 (2021). [24] J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, D. C. Schmidt, A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT, arXiv preprint arXiv:2302.11382 (2023). [25] L. Reynolds, K. McDonell, Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm, in: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–7. [26] Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, J. Ba, Large language Models
Are Human-Level Prompt Engineers, arXiv preprint arXiv:2211.01910 (2022). [27] S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, X. Wu, Unifying Large Language Models and Knowledge Graphs: A Roadmap, 2023. URL: https://arxiv.org/abs/2306.08302. doi:10. 48550/ARXIV.2306.08302. [28] L. Yang, H. Chen, Z. Li, X. Ding, X. Wu, ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling, 2023. URL: https://arxiv.org/abs/2306.11489. doi:10.48550/ARXIV.2306.11489.