This study investigates the verbal creativity diferences and similarities between Large Language Models and humans, based on their answers given to the integrated verbal creativity test in [1]. Since this article reported a very small diference of scores in favour of the machines, the aim of the present work is to thoroughly analyse the data through four methods: scoring the uniqueness of the answers of one human or one machine compared to all the others, semantic similarity clustering, binary classification and manual inspection of the data. The results showed that humans and machines are on a par in terms of uniqueness scores, that humans and machines group in two well defined clusters based on semantics similarities between documents comprising all the answers of an individual (human or machine), per tasks and overall, and that the separate answers can be automatically classified in human answers and LLM answers with traditional machine learning methods, with F1 scores ranging from 68 to 74. The manual analysis supported the insight gained from the automated methods in that LLMs behave human-like while performing creativity tasks, but there are still some important distinctive features to tell them apart.
so on. A good survey on LLMs’ verbal creativity is [
Since work on LLMs creativity is just at the beginning,
Creativity has made it possible for humanity to survive there is a need for methods, resources, and evaluation
and develop since prehistoric times. Despite the per- to better understand LLMs’ creative abilities and their
ception that some people are more creative than others, diferences and similarities with human creative traits.
many psychologists argue that everyone has the capacity In a recent article, [
Creativity is inherently interdisciplinary, involving do- it to both humans and LLMs. The scope of this paper
mains like psychology, cognitive sciences, philosophy, is to analyze the answers given by LLMs and human
rearts, engineering, mathematics, or computer science. Re- spondents to this previous study, for a direct comparison
cently, it has become a field of interest in GenerativeAI of human and machine verbal creativity. To this end, we
(GenAI) [
However, much of the current research in genera- classification with classic machine learning methods on
tive models [
CLiC-it 2024: Tenth Italian Conference on Computational Linguistics, and books have been published on diferent aspects of
Dec 04 — 06, 2024, Pisa, Italy creativity [
Regardless of the domain perspective and of the kind
of creativity, a basic idea in defining it, common to most
of the definitions, is that creativity represents the ability
of an individual to come up with something original
or innovative, of good quality, and appropriate, based
on prior knowledge [
Another related aspect of creativity, as stated by [
In [
efects of a specified situation , and 6. Divergent Association (DAT), where the respondent has to produce seven nouns that are maximally semantically diferent, in all their senses and uses.
Creativity assessment is usually performed with human
evaluators who take into account the four creativity
criteria formulated by [
such as Chat GPT1 or Copilot, the interest in
computational creativity peaked, in an attempt to harvest the
creative potential of the machines, in spite of many chal- [
Previous studies on machine creativity are fragmented: tic distance and fine-tuned GPT for scoring the creativity
some are task-specific, like, for instance, using just role- between the prompt and the answer. The results showed
plays[
In this study, we mind this research gap by analyz- with 0.51. Given that the diference is of just 7 decimals,
ing the creative responses to a wide range of tasks, of a one of our goals for this study is to analyze more in-depth
considerable number of LLMs, from [
Character AI (Character Assistant10). 1. Alternative Uses (AUT), where the test taker is asked to come up with uncommon uses for an ordinary object, 2. Instances, for which the aim is to name as many things as one can think of that have a common feature, 3. the Similarities, which consists of stating as many as possible commonalities of two specified objects, 4. the Causes, where the aim is to guess the cause of a given situation, 2https://huggingface.co/chat/models/ 3No longer supported 4https://huggingface.co/spaces 5https://gpt4all.io/index.html 6https://www.bing.com/chat?form=NTPCHB 7https://gemini.google.com/app 8https://auth.arabic-gpt.ai/ 9https://you.com/?chatMode=default 10https://c.ai/c/YntB_ZeqRq2l_aVf2gWDCZl4oBttQzDvhj9cXafWcF8
who responded to the verbal creativity test as volunteers, either in a lab or at their homes by completing a Google Form. Their background was all academic, from students, undergraduates, graduates and professors, the average age being 26.
We implemented all the experiments in Google Colab11 and we have used three LLMs to assist us with the codes: Claude12, Copilot13 and Gemini14, in a setting of mostly zero-shot prompt engineering, with the standard settings and parameters.
For data analysis, we used Python and the following Figure 1: Ranking of uniqueness scores for humans and malibraries: Spacy15, Scikit-learn16, Matplotlib17, Numpy18, chines and Pandas19. 3.1. Data The databases of verbal creativity answers contains 4530 answers, totalling 13714 words. The test was organized in 6 tasks. Five out of the six tasks have five items each and a maximum of 10 answers per item. An answer can have a maximum of 5 words. The sixth task, DAT, consists only of one item of 10 single-words answers, but only the most semantically diferent 7 out of the ten given by the respondents were taken into account by the DAT web page 20. That amounts to 2570 answers for the machines, which responded always with the maximum number of answers, 10, even if the instruction was the same for both humans and machines to give between 1 and 10 answers per task. The human respondents gave any number of answers in the range 1 to 10, obtaining thus 1960 human answers. As such, the database is unbalanced, with with more than a third more machine answers compared to human answers. 3.2. Uniqueness scores for the answers of humans and machines to the verbal creativity test since one of their goals was to evaluate the answers fully automatically. Nevertheless, the uniqueness of the answers of an individual constitutes an important clue to their creativity. Hence, to better understand the uniqueness trait of both humans and machines, we computed uniqueness scores as if follows.
We grouped the creativity test answers of both humans and machines in separate files, each containing all the answers of a particular individual. We thus obtained 20 answer files, 10 for humans and 10 for LLMs. After removing the stop words, we generated embeddings for each file, and then we computed their pairwise semantic similarity, using spaCY library. The uniqueness scores were obtained as the inverse of the average semantic similarity scores between an individual and all the others.
The ranking obtained in the decreasing order of uniqueness is depicted in figure 1, where one can see that the humans (in green) and the machines (in red) are mostly intermingling.
This uniform distribution of humans and machines in terms of uniqueness scores shows that humans and machines are on a par in this respect. 11https://colab.research.google.com/ 12https://claude.ai/chat/ 13https://www.microsoft.com/en-us/microsoft-copilot 14https://gemini.google.com/app/ 15https://spacy.io/ 16https://scikit-learn.org/stable/ 17https://matplotlib.org/ 18https://numpy.org/ 19https://pandas.pydata.org/ 20https://www.datcreativity.com/
is the degree of originality of the answers of one
individual, compared to the answers of all the other individuals. The aim of this experiment was to investigate if individual
The evaluation of this criterion is done manually and humans and individual machines cluster together, based
is time-consuming, since it includes assessing not only on semantic similarity of their answers to the creativity
word similarities, but also similarities between ideas of test. We used the word embedding of the 20 individual
the diferent individuals. [
In figure 2 we can see how the LLMs (dots in red) perfectly cluster together, just as the humans (dots in green) do, considering all responses to the six tasks. This result indicates that from a semantic perspective, humans and LLMs generate creative answers diferently, or at least that there are discriminating features to distinguish
between the two. In this binary classification experiment, we investigated
We also plotted the clusters per answers to a specific if they also have distinctive features at the answer level. task, for all the 6 tasks, in figures 3, 4, 5, 6, 7, and 8. Gen- For this, we trained several traditional machine learning erally, the answers of the humans and of the machines (ML) classifiers to discriminate between the answers of clearly clustered by their kind, with the exception of the humans and of LLMs to the verbal creativity test. The task Instances, where the humans and the LLMs were two classes were represented by all the answers of the interposed, meaning that the semantic content of their humans and, separately, by all the answers of the LLMs, answers was not specific to any of the two classes. A with one answer per line, excluding the DAT task, since bit of mixing appeared also in Divergent Association Task it only required enumerating words. As the LLMs al(DAT). The not so clear separation of humans and ma- ways gave the maximum number of answers required in chines for Instances and DAT tasks might result from the the test, the dataset was unbalanced (2500 answers for fact that the responses to these particular tasks are inher- LLMs and 1890 for humans). To address this problem of ently very short, of just one or two words for Instances unbalanced dataset, we implemented a simple random task and of just one word for the DAT. under-sampling technique, thus obtaining 1890 answers for each class, humans and LLMs. We then employed the 3.4. Binary classification of human and Term Frequency-Inverse Document Frequency (TF-IDF) vectorization technique to convert the text data into numachine creativity answers merical features. The vectorizer used a maximum of 1000 As the clusterization experiment suggested, the answers features, for capturing all important aspects and dealing to the verbal creativity test are almost linearly separable with computational complexity. Stratified sampling was in two classes (humans and machines) at individual level. used to ensure a dataset split for an 80/20 training and
and humans to see what makes their answers so diferent from the others but also investigated the uniqueness scores correlation with the quality and creativity.
Figure 7: Semantic similarity clusters of answers for Conse- The first positioned on the uniqueness ranking, the quences LLM Jais, had the tendency to respond to the Similarity task with word obtained by nominalization (deriving nouns from verbs), like, for instance, "dependency", "cutesting ratio. Thus, training and testing sets contained riosity", "belonging", and "growth", as opposed to all the the same number of samples for each category, e.g. 1512 other LLMs, which responded with regular nouns. It also answers for training, and 378 answers for testing. tended to use answers that started with the same prefix:
In table 1, we give the best three classifier methods, “Unfiltered”, "Unmatched", "Unrestricted", and "Unyieldwith precision, recall, accuracies, and F1 scores. The ing", and to use the same word followed by other words, NaïveBayes classifier obtained the highest accuracy, of like in, for instance, "Thought policing", "Thoughtful 0.74, followed at just three decimals by both the Support shopping", and "Thought clones". In this respect, Jais Vector Machine (SVM) classifier and the Random Forest gave the most unique answers, which, obviously, were classifier, with an accuracy of 0.71. not also the most creative.
This moderate performance of the ML models sug- The second positioned on the uniqueness ranking, Hugests that either the dataset is too small for the models man 3, started the majority of their answers with "use" to perform better, or that there is a fair amount of sim- or "use it as". This respondent also repeated the starting
point of most of their answers, like in "what...", "getting a ...", "where ...", "in a...". These features seem enough to score highly w.r.t. uniqueness, but fail to correlate with We did not use or disclose any personal data from the huthe quality of the creativity. man participants, who remained completely anonymous
This inspection shows that the most unique answers and took part in this research as volunteers. There are no are not necessarily the most creative. If the bulk of the ethical concerns with regard to publishing this research. respondents give good-quality answers, that might result in a high uniqueness score for lower-quality or less 5. Limitations creative responses.
We also checked the appropriateness of the answers The dataset for this research was small and slightly
unbalgiven by both humans and machines, which is an im- anced since the humans answered based on their mood
portant requirement of genuine creativity, as mentioned or capabilities, while the LLMs answered strictly with a
in section 1. Creativity requires divergent thinking, but maximum of ten answers per task.
true creativity emerges when convergent thinking also Also, the sample pool is quite small, as there were only
restricts the divergence to only those responses that are ten humans and ten LLMs involved, so the results might
appropriate for the creative assignment [
In general, humans gave fairly suitable answers. In- Due to lack of space, this study focuses more on austead, not all the LLMs managed to generate all the an- tomated methods of analysis, than on manual analysis, swers in an appropriate manner. For instance, for the thus lacking a more in-depth insight into the patterns of Consequences task, for the item "There is a virus and only the collected answers to the verbal creativity test from children survive", Gemini, although responded creatively, both humans and machines. failed to also respond suitably. This model gave four Finally, this study compares the creativity answers out of the ten answers that are either paradoxical, or of humans and LLMs in English, but the human particinon-sensical, in a situation that clearly implies that only pants to the test were non-native (fluent) English speakchildren are alive, so there are no adults around: "Toy Fac- ers, which can potentially decrease their creativity score, tories booming", "Geriatric Theme Parks", "Grandparents compared to scores they could obtain in their own native raise parents", "Parents taught by Tablets". language.
Another manual scrutiny focused on analyzing the similar or the diferent patterns of LLMs and humans when responding to a particular task. We found that sev- 6. Conclusion and future works eral LLMs answered to the Divergent Association Task with the same word among the seven required ones. This study showed that there are some diferences beFor instance, "Serendipity" was used by three models. tween human and machine answers given to a verbal This phenomenon is not specific only to the machines. creativity test, but also plenty of similarities. For the Guessing Causes Task, Human 3 and Human The LLMs’ answers vary much like the humans an4 produced similar answers, like, for instance, both swers. Individual, unique answers, w.r.t. to the set of gave the answer "earthquake", or produced the same all answers are produced by both humans and machines idea, like "green lights"/"because of green lights", "eating alike, with no noticeable diference. something bad"/"they ate something bad", "St Patrick’s Still, at a semantic level, humans and machines generDay"/"St. Patrick’s day party", "poor construction"/"faulty ally group together as individuals. structural integrity", "looking at screens too much"/"too The performance of automatic classification between much screen time". human and machine answers is moderate and leaves
Also, we noticed some peculiarities of individual LLMs, room for improvement. such as Falcon’s generation of only words starting with The general findings of this study indicate that LLMs’ the letter "a" for DAT, or Cohere’s generation of only op- creative capabilities are comparable with human abilities posite words for this task: "love", "hate", "peace”, "chaos". and, as such, they could be put to good use in the creative
Moreover, humans seem more personally involved in domain. Humans "just" need to adapt to their usage, mind answering than LLMs, which tend to give only general the ethics and safety issues, and discern the information answers to the tasks, with some exceptions. Some LLMs at every step, instead of blindly using them. seem to respond "humanly", even producing humor and In future work, we will focus on expanding the dataset, ifgurative speech, while others only respond quite stan- by adding more LLMs’ and humans’ answers to the test, dard or "robotic". for a better statistical coverage.
Overall, the LLMs’s distribution is similar with the Also, we aim to manually investigate more in-depth humans’ distribution, varying from one individual to the database, to look for more systematic patterns for another. both humans and machines.
ties, we also plan to investigate other aspects of LLMs’ creativity, such as language or image.
Another future approach worthy of pursuing is using Deep Learning approaches instead of traditional Machine Learning approaches for the binary classification task, or using metrics specific to LLM-generated tasks. 6. Divergent Association Task (DAT)
Write ten words that are as diferent from each other as possible, in all meanings and uses of the words.
Rules: 7. Appendix Verbal Creativity Test Only single words in English. Only nouns (e.g., things, objects, concepts). No proper nouns (e.g., no specific There are 6 types of creativity assessments in this test. people or places). No specialized vocabulary (e.g., no Note: Be as creative, original, and innovative as possible. technical terms). Think of the words on your own (e.g., Pay attention to the word and answer limit! Try to think do not just look at objects in your surroundings). of as many answers as possible within the limit!
1. Alternative uses Test Name up to ten unusual uses for the following five items. Use a maximum of five Acknowledgments words. Give one answer per line.
2. Avocado 3. Whistle 4. Chalk 5. Pantyhose
Give one answer per line. Name up to 10 things that:
2. Things that you have control of in your life 3. Situations where it is good to be loud 4. Things that can flow 5. Things that you can mark on a map
Use a maximum of three words to describe a common feature of the following pair of words. Give one answer per line. Give up to ten answers: 1. Prison & School 2. Eyes & Ears 3. House & Den 4. Earthquake & Tornado 5. Baby & Cub 4. Causes 1. Crash of a building 2. Everybody turns green at a party 3. Social media disappears 4. Humanity becomes shortsighted 5. Your hat does not fit you anymore 5. Consequences