In this paper, we apply the ChatGPT Large Language Model (gpt-3.5-turbo) to the 4books dataset, a German language collection of children's and young adult novels comprising a total of 22,860 sentences annotated for valence by 80 human raters. We verify if ChatGPT can (a) compare to the behaviour of human raters and/or (b) outperform state of the art sentiment analysis tools. Results show that, while inter-rater agreement with human readers is low (independently from the inclusion/exclusion of context), e昀케ciency scores are comparable to the most advanced sentiment analysis tools.
then, it could also be used to test LLMs in one of the most common tasks in natural language processing, namely Sentiment Analysis, in a still under-researched language for LLMs (German).
The 4books dataset [
The rating process was as follows: First, the participants evaluated the emotional impact in terms of valence (on a scale from−3 to 3) and arousal (on a scale from 1 to 5) on sentence-level (for an example of the rating interface, see Figur1e). Sentences were shown one by one, without the possibility of reading following sentences or revising previous ratings. This was decided in order to collect the most immediate and direct reactions from readers, by disrupting the least possible their reading experience. To avoid cognitive overload, participants were asked to rate one chapter per day. At the end of each chapter, they answered a series of comprehension questions, aimed at verifying their understanding of the reading material. Finally, they were 1https://chylsa.pages.gitlab.rlp.net/chylsa-websit.e/ 2Due to copyright reasons, the dataset cannot be published. 3Due to the heaviness and complexity of the rating task, it was not possible to match participants’ age with books’ target audience. asked to produce multiple ratings on the chapter and book level (including valence and arousal, but also suspense and transportation).
In this paper, we focus on the sentence ratings for valence, as it is the most frequently studied dimension in Sentiment Analysis research. We had also to exclude working on larger textual segments (like chapters and full books) due to the current limitations of context window in the most widely used LLMs.
In the 昀椀eld of NLP, much attention has been given to the recent LLMs. Multiple studies compare
ChatGPT to the best performing current Transformer models, so-called state-of-the-art (SOTA)
models. Overview studies use ChatGPT for numerous NLP task2s2[
With regards to ChatGPT various ethical considerations have been discussed. Recent research
has approached the question of whether LLMs could be considered as intelligent agents by
using frameworks such as the Turing test4[], theory of mind 1[
However, one strong stance on the subject has been advanced in a few studies which present LLMs as able to substitute human annotators. Such a substitution could constitute one of the main con昀椀rmations of the 昀椀nal “emergence” of the arti昀椀cial intelligence, if we consider that manual annotation is the very groundwork for any machine learning task—where the machine tries to imitate the human. Simultaneously, automatic machine annotation may dramatically decrease the 昀椀nancial resources needed for human annotators.
Gilardi et al.7[] compare the annotations done by ChatGPT and by crowd-workers on MTurk against a gold standard of trained human annotators. They 昀椀nd that ChatGPT outperforms the crowd-sourced annotators while being at the same time much cheaper to fund. Even more, the intercoder agreement exceeds not only that of MTurk annotators but also that of the trained experts, which is especially remarkable due to the frequently reported low agreement rates among human annotators 2[].
In a similar experiment, Törnberg29[] shows that the LLM’s accuracy exceeds all other
human annotations. Intercoder reliability is reported to be much higher when compared to
human annotators. Huang et al. 1[
While these experiments generally show promising results, others 昀椀nd only average
performance: In the study by Ding et al. [
In Sentiment Analysis, dictionary-based approaches are continuously outperformed by Transformer-based models26[]. At the same time, it is conceivable that LLMs may lead to a shi昀琀 in common Sentiment Analysis techniques because promising results have been reported in other NLP applications.
The studies of Wang et al. 3[
Altogether, the main observation—the performance of ChatGPT in Sentiment Analysis,
while generally being high, is lower than that of 昀椀ne-tuned SOTA models—is con昀椀rmed by
most other studies that use Sentiment Analysis next to other NLP tasks22[
Overall, our experimental setup consists of two parallel studies: one focused on verifying if ChatGPT behaves like a human rater (using inter-rater agreement to measure similarity of behaviour); another comparing ChatGPT to Sentiment Analysis tools (de昀椀ning a ground truth to evaluate tool e昀케ciency). The 4books dataset is especially suited for the 昀椀rst study because the ratings are done by readers and not by trained experts—an important distinction as there was neither a guidebook for the raters nor the possibility of discussing possible disagreements and thus increasing inter-rater agreement.
To perform all analyse4sw,e used the OpenAI API. All experiments were carried out with the “gpt-3.5-turbo” model (corresponding to the default version of ChatGPT and therefore referred to as such in the following pages), the most advanced among the ones available for our newlycreated account at the time of writing. Tabl1e presents an overview of the prompts sent to the API.
One possibility for our 昀椀rst study was that of considering ChatGPT as a substitute of multiple
human raters. Such a possibility could be supported by the fact that ChatGPT is able to produce
4All scripts available at the following linhktt:ps://gitlab.rlp.net/srebora/chr23-sentiment-chatg.pt
as new and original in itself), thus not allowing to link a group of them to a single individual. We
its temperature to the lowest level (zero), which makes its behaviour the most deterministic
[31]—and, by consequence, the most consistent [
We veri昀椀ed this assumption by working on a ∼10% sample of the dataset (2,000 sentences,
randomly selected). Each sentence of this sample was processed by sending an API request
with the system prompt5 corresponding to ”P1: Sentence Rating” in Table1. This prompt
was conceived to imitate as closely as possible the guidelines followed by human raters in the
creation of the 4books dataset and taking as an example the prompts described by Rathje et al.
[
For each request, 20 di昀erent completion choices were collected and compared with each other, verifying that in 98% of the cases (1,965 out of 2,000) completions were identical. While a margin of inconsistency remained present, ChatGPT can still be considered as corresponding to a single agent (if we recognize the impossibility also for a human rater to reach a 100% consistency when repeating a task 20 times).
We then extended the analysis to the entire 4books dataset, by collecting only one completion choice with the same setup as described above. Out of 22,860 sentences, ChatGPT refused to produce such a score in 44 cases, requesting instead more context information. A qualitative analysis of these 44 sentences showed that they were mostly brief or broken clauses (a consequence of the automated text segmentation at the beginning of the rating collection). We therefore decided to exclude them from the analysis.
A comparison between ChatGPT and human ratings is shown by Figur2e6. The scores are in line with frequently reported agreement rates in sentiment annotation of literary te1x3t]s. [In this context, ChatGPT always placed itself below the level of agreement between most human raters.
However, one of the main issues in comparing the above-described setup with human
annotation is that ChatGPT was given just a single sentence at each API call, while human raters had
the opportunity to contextualize it while reading the entire text. Thus, to con昀椀rm the validity
6Scores in Figure2 are obtained by 昀椀rst calculating inter-rater agreement scores between each pair of raters using
two di昀erent methods: Cohen’s Kappa and Intraclass Correlation Coe昀케cient of typeICC(3, 1) [
The simplest way to mimic human experience was to add to the prompt the sentences preceding the one to be annotated. We tested 昀椀ve di昀erent con昀椀gurations, with context windows of respectively 1, 2, 4, 8, and 16 preceding sentences. The corresponding system prompt “P2: Context Window” can be found in Table1.
Beforehand, we again performed a consistency check (with the same modality as described above) on 10% of the dataset (200 sentences), obtaining 96.6% of consistent answers.
Results of the full analysis are presented in Figu3r7eand show the substantial ine昀케ciency of context windows in increasing ChatGPT agreement. In most of the cases, agreement does not improve substantially, with just one exception for the book by Me8bsO. verall, increasing the context window does not correlate with higher inter-rater agreement, with worse results produced for longer windows (see ICC scores for 16-sentence windows).
Among the 78 cases when ChatGPT produced an invalid answer (proportionally higher than in the previous experiment), there were 13 cases when ChatGPT misunderstood the task, generating a continuation of the story instead of its sentiment evaluation.
To verify that there were no problems in distinguishing target sentence from context, we repeated the operation on 10% of the dataset (200 sentences) and changed the 昀椀nal part of the previous system prompt (refer to ”P3: Context Window Veri昀椀cation” in Tabl1e). We veri昀椀ed 7Scores are calculated as in Figur2e, by excluding inter-rater agreement between the 昀椀ve di昀erent con昀椀gurations. 8This di昀erence seems to be mainly due to text di昀케culty, being Mebs addressed to 6-8 year olds. that in 72% of the cases the analysed sentence was the same as the target sentence9. A qualitative analysis of the remaining 28% revealed how in most of the cases ChatGPT simply extracted key sections from the target sentence (e.g., direct speech), but never mixed it with the previous ones.
For our second hypothesis (if ChatGPT could substitute state of the art Sentiment Analysis tools) we chose a phrasing to mimic the output of Transformer models for Sentiment Analysis, which generally produce probabilities over three sentiment classes (for the prompt, refer to “P4: Sentiment Analysis” in Table1).
The adopted procedure was the same as described above, starting from the veri昀椀cation of output consistency (96% ) on a subsample of 2,000 sentences.
Table2 presents results of the analysis on the full corpus, compared to three other approaches (pre-trained and 昀椀ne-tuned Transformer models and a dictionary-based approac1h0)T.o get a more 昀椀ne-grained reference point, we took as ground truth the mean of all human ratings per sentence (normalized between−1 and +1). Such a value could be directly compared with the output of the dictionary-based approach (SentiArt, developed by Jacob1s2][ for German language) and was used to 昀椀ne-tune the Transformer model via linear regression. To make the output of ChatGPT and pre-trained Transformer models comparable, we 昀椀rst converted classes to numbers (e.g., “positive”: +1; “neutral”: 0; “negative”−:1), we then multiplied each number by the probability of the corresponding class and 昀椀nally summed the obtained values (this can be synthetically described as a “weighted mean”1)1.
ChatGPT clearly outperforms all approaches, apart from the 昀椀ne-tuned Transformer model. This result is in line with previous studie2s3[, 31]. 9Edit distance was <5% of its total number of characters. 10For the pre-trained Transformer model, we report the resultcsaordfi昀nlp/twitter-xlm-roberta-base-sentiment , which performed the best on our dataset when compared to six other models tagged for German Sentiment Analysis in huggingface.co—see ourGitLab repositoryfor more details. Fine-tuned Transformer modedleiespset/gbertbase, for which we report both means and standard deviations, because r-squared scores were obtained by performing a 10-fold cross validation. 11Note how the scores obtained with this procedure for ChatGPT were strongly correlated with the ones obtained with the prompt P1 in Table1 (Pearson’s correlation = 0.752).
While ChatGPT seems to be not yet fully comparable to the behaviour of human raters, it can already work as a valid substitute to the most widely used Sentiment Analysis tools in German language.
However, due to intrinsic limitations of our dataset and possible improvements of our study design, such conclusions should still be considered provisional. Among the limitations, it should be noted how the human ratings we collected were not curated, so they cannot be considered as professional annotations. There is in fact the possibility that ChatGPT’s disagreement might be due to its choosing the most 昀椀tting answer, while the majority of the raters gave a less 昀椀tting one (a possibility that is suggested by studies like the one by Gilardi et al7.])[. Due to the structure of the 4books dataset, we cannot unfortunately clarify this doubt. However, our 昀椀nal goal was not to check if ChatGPT behaves like a professional annotator (who indeed annotates a document as mechanically as possible), but if its behaviour is comparable to that of humans (which inevitably implies an element of disagreement).
Among the improvements, there is the possibility of repeating our analyses with other and
more advanced LLMs (like GPT-4, but also Llama2, Bard, Claude2, and many others). With
our paper, we just set up the groundwork for such an experimentation. Other possible
improvements relate to the prompting technique, which could have approached the problem from
a few-shot or chain-of-thought perspective (especially when studying the e昀ects of context).
Also, it has been noted how the formulation of the prompt itself can have an impact on LLMs
performance [
One main problem which connects all these issues, still, is that of economic resources. The OpenAI API usage for this paper cost in total $17.52, but such a limited cost was due to the strategies we implemented to reduce API usage (such as random sampling). If we wanted to apply all analyses to the full dataset and test di昀erent prompting techniques, the cost would have easily increased to hundreds of dollars (while still using the much cheaper gpt-3.5-turbo model).
This problem connects to the issue of transparency in LLMs, which are still frequently studied as black boxes, understandable only via direct stimulation/observation, like some kind of biological system. In this regard, our research has not only provided new evidence to such a study, but has also allowed recognizing how the perspective from which LLMs are viewed—being intended just as computational tools or as possible substitutes of human agents—inevitably shapes the way in which they are studied and understood.