Fanfiction -stories inspired by existing works, written by and for an audience of fans, and published for free online -is a burgeoning domain of online cultural activity. Archive of Our Own (AO3), one of the largest English-language websites for publishing and reading fanfiction, recently reported hosting over 13 million works [31]. Fan studies scholars emphasize the need to understand fanfiction in relation to its community contexts, because fanfiction is usually written "within and to the standards of a particular fannish community [9]" and "is often so deeply embedded within a specific community that it is practically incomprehensible to those who don't share exactly the same set of references [35]. " As such, "a fan fiction story cannot be viewed as a wholly self-contained object, a text delimited by the first and last words that appear on the screen, the way that readers of novels and other genres of print culture conventionally read books as bounded by their covers [11]. " Due to the deeply intertwined nature of fanfiction texts and the shared practices and contexts of their online community -which fans call fandom -fanfiction texts must be approached "as both literary and digital objects [4]. " Scholars need methods to study the ways fanfiction is received, embedded and evaluated in online spaces like AO3 that account for its hybrid literary/digital nature.
Additionally, fanfiction often revolves around emotional trajectories. Love, heartbreak, mental illness and other emotional highs and lows are frequently central to fanfiction, which has been described as "emotional landscapes of reading [33]" both because of its intradiegetic emphasis on emotionality and because of its capacity to evoke strong emotions in readers.
The comment-section on AO3 lets readers share their affective responses to fanfiction reading, which we conceptualize as sentiments in the current paper. Sentiments can develop over the course of the reading and may extend or endure into the time after. Through comments, readers express these sentiments in the social context of their fan community. Sentiments are important for understanding which aspects of fanfiction are loved, enjoyed, and appreciated. The hybrid digital/textual nature of fanfiction comment, the enormous scale of fanfiction and comment production and exchange as born-digital objects, and the fanfiction community's emphasis on both intradiegetic and extradiegetic emotion make fanfiction comments suitable for computational approaches such as sentiment analysis (SA).
The application of Natural Language Processing (NLP) methodologies to date in literary studies has been limited [5,23]. Specifically, SA, which categorizes texts as positive, neutral, or negative, is often criticized and considered inadequate for the detail-oriented research needs of literary scholars [20,34]. The sentiments present in literary or narrative text often cannot be reduced to a negative/positive binary. To address some of the limitations of existing SA tools, aspect-based sentiment analysis (ABSA) has gained attention. Unlike conventional SA that labels the sentiment of entire documents, paragraphs, or sentences, ABSA operates at the aspect level, extracting specific aspects and their corresponding sentiment polarities [3,36]. Although ABSA offers a more granular approach to sentiment mining, its application has largely been confined to commercial domains such as customer reviews [36] and applications in (computational) literary studies still need to be explored.
To fill these research gaps -the application of ABSA to literary studies data and the use of NLP to analyze textual/digital fanfiction comments -we employ ABSA to detect the aspects of fanfiction that readers refer to when writing comments and the sentiments commenters attach to these aspects, conceptualized in a binary fashion (positive/negative). We address a methodologically-oriented research question about fanfiction reception on AO3:
• What are the affordances and limitations of aspect-based sentiment analysis (ABSA) for analyzing fanfiction comments?
In other words, we investigate whether ABSA is an effective approach to analyzing the aspect-sentiment combinations in a dataset of fanfiction comments. We limit this research to one platform (AO3) for reasons of scope. We focus on the Greek mythology fandom because Catching Feelings is part of the larger project Anchoring and Innovating Greek Myth in Contemporary Online Fanfiction (2022-2026), which focuses on the way contemporary online fanfiction transforms cultural material about Greek mythology and the way this cultural material resonates with online fan communities. 1Section 2 outlines the processes of data collection and annotation. Section 3 describes our ABSA pipeline. Section 4 reports on the inter-annotator-agreement of the annotated dataset, and presents the results of various ABSA pipelines. Section 5 examines these results. Finally, Sections 6 reflects on the reproducibility, generalizability and limitations of our approach to ABSA for fanfiction comments and outlines some avenues for future research, and in Section 7 we answer our research question.
We based our comment dataset on MythFic Metadata [29], an existing collection of metadata for 5.155 works of fanfiction about Greek mythology that were published on AO3 between 2008 and 2022. 2 We used the list of unique identifiers for the stories in MythFic Metadata as input for the AO3-api [14] to collect comments on those stories. The resulting dataset contains 25.970 comments. Table 1 provides descriptive statistics of the comments in terms of character-and word count.
We must address three limitations of this dataset. First, fanfiction comments, like any type of textual review or comment, do not fully capture the complex mental and emotional processes of reading experience and evaluation. Instead, these comments reflect only the elements of the reading experience that commenters are consciously aware of and chose to comment on, in the way they chose or were led by discursive norms to represent these elements textually. Second, commenters are only part of the users of AO3, and may not be representative of the reader reception happening all over the platform, let alone the fanfiction reception happening on other platforms. In a 2013 census of AO3, 43.6% of respondents reported commenting on stories they had read (n = 4,358) [7]. When MythFic Metadata was collected, 1,584 stories in the fandom Ancient Greek Religion and Lore, or 30.7%, of the fandom's works, had not been commented on at all. Additionally, commenters can be characterized as above-averagely engaged or committed readers, since they invest the time and effort to comment. Finally, the content of fanfiction comments is co-shaped by the culture and affordances of AO3. Like with online book reviews, social factors like "the interactive nature of the reviewing platform or reviewers' desire to cultivate a persona or gain followers [22]" influence these texts. Additionally, fan communities differ between platforms because "the relationship that fans have to technology and online platforms is integral to the culture of these communities [13]. " The exact dynamics of such relationships remain largely unknown and so the generalizability of our findings to other platforms may be limited.
Of the 25,970 comments, 25,240 contain text and 730 contain only emojis. The emoji-only comments were discarded in our subsequent analysis. From the textual comments, we created an annotated subset using the following criteria:
• Length: 100 -4,000 characters. 3• Language: English as predicted by the Python package langid [25] with high confidence (>0.9).
From the remaining 12,866 comments, we randomly sampled 1,000 to create the annotations. Following existing research, we define sentiments as "social constructs of emotions that develop over time and are enduring [10]. " Since we conceptualize aspects here as particular features of a fanfiction story that elicit sentiments, it becomes possible to identify and computationally extract positive and negative sentiments as they relate to particular aspects of fanfiction reception.
To develop an annotation guide we conducted exploratory analysis of the comment data, such as an analysis of the most-frequent noun chunks as detected by SpaCy [17] and a topic model of longer comments (>300 characters) created with Top2Vec [1]. Based on these explorations, we formulated a scheme of 8 aspects of fanfiction that were frequently commented on. Table 2 provides examples of each aspect-category. Following the methodological principle of ethical fabrication [19] these examples were fabricated to protect the privacy of commenters while still mirroring as closely as possible the content of the data. Some aspects were chosen bottom-up, i.e. based on the exploratory analysis, and others were chosen top-down, i.e. based on aspects of comment-data that interest us.
The eight resulting aspect-categories are:
• Canon: reference to how fanfiction transforms, critiques, or engages with canonical material, including references to Greek mythology but also source materials from popular culture. • Character: reference to and assessment of character, characterization, character appearance and relationships between characters from the story. • Emotion: references to the emotions experienced by characters intradiegetically.
• Events and Storyworld: reference to plot events and settings, specific scenes, world building, story content like tropes, and general plot elements like twists or endings. • General: reference to the story as a whole or in general terms.
• Reading Experience: reference to reading experience, such as emotional engagement (of the reader, not the characters), absorption or narrative tension. • Style: reference to how the story was written down or rendered, including writing style, word choices, metaphors, turns of phrase, voice, perspective. • NULL: expressions of sentiment that do not refer to specific words or aspects.
One central aspect of fanfiction comments we exclude is expressions that foster a bond with the author. Because fanfiction exchange relies on a gift economy [16], expressions of gratitude, appreciation and encouragement are an important part of commenting, and phrases like "Thank you for writing this" or "Please write more!" occur very frequently. An existing LDA topic model of fanfiction comments showed that "author encouragements" and "requests for story" were among the most prominent topics in comments on another fanfiction website, Fanfiction.net [26]. However, because these types of expressions do not explicitly evaluate aspects of stories themselves, but rather relate to the status and position of authors and readers in the community, we disregard them here.
We used INCepTION [21] to annotate (see Figure 1 for an example). The first author annotated all 1,000 comments in the annotation-set. One co-author annotated an overlapping 100 comments to determine inter-annotator agreement (Section 4.1).
Aspect-based sentiment analysis (ABSA) is a type of information extraction (IE) task consisting of two subtasks: aspect extraction (AE) and sentiment analysis (SA) of the sentiments associated with these aspects. Approaches to these tasks in and beyond computational humanities include rule-based models [22], machine learning models [3,10,36,30] and, more recently, generative large language models [12,18,8]. We selected a machine learning approach because (a) formulating domain-specific rules is time-consuming and would not generalize well over other datasets, and (b) while approaches using generative Large Language Models (LLMs) are currently in full development, the effects of their opaque nature, tendency to propagate bias and hallucinate are not yet fully understood in IE-applications for the literary domain [12]. However, exploring the use of LLM's for ABSA presents an interesting avenue for future research.
Given the evaluative nature of the comment section on AO3, ABSA could be a good fit for assessing the sentiments of fanfiction readers on specific aspects of these texts. At the same time, the strong influence of subcultural community norms on the description of aspects and the expression of sentiments in these comments may make it challenging to apply an out-ofthe-box method.
Before training a model on our annotations, we conducted a small-scale inter-annotator agreement study of the 100 comments annotated by both annotators. Our inter-annotator agreement was exceptionally high: pairwise Cohen's 𝜅 of 0.86 (aspects) and 0.88 (sentiments). This is likely due to the intensive collaboration and discussion between the annotators on the annotation guidelines and because of our shared familiarity with the discursive and social norms of fanfiction communities. The only categories showing notable confusion were Events and Storyworld and Character, with 9 instances of confusion between annotators. This is somewhat to be expected since characters often participate in story-events and always exist in the storyworld, so the distinction between these aspects can be blurry. Both annotators found a clear prevalence of positive evaluations (n=206 agreed occurrences in the annotation-set) of aspects over negative evaluations (n=4). This aligned with our expectations based on the exploratory data analysis and on a review of the existing literature, where research has shown that fanfiction comments are often written in a "positive style [28]. "
We tested three models for joint aspect extraction and categorization: roBERTa-base [24] (Macro F1 0.25), Twitter-roBERT-sentiment [2] (Macro F1 0.31), and NuNER [6] (Macro F1 0.45). Evaluation metrics for each of these models are listed in Table 5 and per aspect in Table 7. Since NuNER performed best, we tried to improve its results further by splitting the tasks of aspect extraction and categorization, to minimal effect (from 0.45 to 0.50 Macro F1). Note that the numbers for support for split-and joint task approaches are different because in the split task, aspect extraction is a separate first step where some aspects may not have been detected. Additionally, the support-numbers in Table 7 do not perfectly match the number of annotations reported in Table 3, because in a handful of instances the aspect-term in an annotated sentence proved difÏcult to locate due to variations in spelling, punctuation, capitalization or even spaces.
We used the Twitter-roBERTa-sentiment model since it has already been fine-tuned for sentiment detection, so we were expecting it to perform well on a polarity detection task. These results were decent (0.75 macro F1) (Table 6). This may be because sentiment polarity does not deal with as many categories as the aspect-task (two instead of eight), and because the model has been trained on Twitter-data, which is in some ways similar to our dataset of fanfictioncomments. Both types of texts are relatively brief, use internet jargon and often convey strong sentiments or opinions.
Based on these results, the affordances and limitations of ABSA for analyzing fanfiction comments seem strongly differentiated by aspect category. As expected, the models perform best for those aspect categories that have high support in the annotations, especially General and Style. Emotions are particularly problematic, perhaps due to their low support but perhaps also because emotions are not commonly considered entities or aspects in NER-tasks. Figure 2 is a confusion matrix between the best-performing aspect-extraction model (NuNER) and our gold standard annotations. As you can see, the only significant source of confusion was between Character and Event & Storyworld aspects, which makes sense as characters usually participate in events and interact with story worlds. Entities that the model failed to detect were left out of Figure 2 for the sake of interpretability. Manual inspection of these left-out results reveal that many Events were not detected. Emotion was also difÏcult for all models to detect, perhaps because words that refer to the Emotion-aspect also often occur in descriptions of Reading Experience. Detection of Canon was often partial, but detecting these partial matches (often revolving around words like 'retelling, ' 'original' 'allusion' or 'portrayal') is still useful in mapping on a large scale how Canon is discussed in the comments. Characters that were very frequent in the data (such as Achilles) were more often accurately detected than less prominent characters (such as Demeter). General aspects were captured well in most models, with the exception of some relatively long and unique figures of speech, such as 'I take my hat off to you. ' Non-specific expressions of affect were also detected relatively well, including the kinds of borderline-incoherent expressions that are specific to fanfiction communities. Examples of these include 'WAHHH' and 'HNNG'. NuNER even identified 'skdjahskdjhsd' as a NULL-aspect; correctly, in our opinion, as keyboardsmashes often convey strong but non-specific sentiments.
Figure 3 is a confusion matrix between the two annotators, for whom the Character and General aspects were easiest to agree on. With the General aspect, then, the skills of hu- man annotators overlap with those of NuNER. For Style, it seems that NuNER performed well because Style is often described in the comments using a relatively specific and limited vocabulary. In our test set, the Style tokens contained only 30 unique words, compared to 53 for Emotion, 93 for Canon and 287 for Events & Storyworld. The most common words for Style are also quite consistent and content-related, such as 'written', 'writing', 'describe', 'language', and 'words. In comparison, the most common words detected in other aspect-categories are often stop-words. This may make Style relatively easy to detect computationally. With NuNER's help, we also found some small errors in our annotations, such as characternames that were overlooked in the part of the annotation-set that was only annotated by one author. In future work, an annotation effort where each text is annotated by multiple people may provide better results.
The machine-learning approach to ABSA is more generalizable to other domains, especially other fandoms, than a rule-based model would be. Particularly in the relatively successful categories of General and Style, we expect no significant differences between fandoms in how commenters refer to these aspects of fanfiction. We do expect, however, that our model will be difÏcult to generalize across different platforms for fanfiction exchange, as research has shown that "our lack of knowledge about platform influence on norms of commenting makes it difÏcult to generalize about the dynamics of fanfiction commenting beyond AO3 [28]. " The importance of subcultural norms in expressing the sentiments of members of fanfiction communities like AO3 may also make it difÏcult to apply our ABSA pipeline to other types of comments, such as book reviews on Goodreads or comments on Youtube.
We want to highlight five potential directions for future research. Firstly, it may improve results to annotate more data in a targeted way, supplementing those aspects for which support was low. Secondly, it may be of interest to analyze the 730 comments containing only emojis, using, for example, the Multidimensional Lexicon of Emojis [15]. Third, it may be interesting to tackle the task of ABSA with a generative LLM, since these models are gaining ground as information extractors. GoLLie [32] looks especially promising, as this prompting framework would allow us to feed our existing detailed guidelines for annotation directly to a generative model. Fourth, a hybrid approach to ABSA, particularly adding some rule-based elements to our existing ML pipeline, may be a productive way to improve outcomes for some of the aspects that were difÏcult to detect using our current setup. For example, we could use an emotion lexicon like EmoLex [27] to detect the Emotion aspect and create lists of characters and story settings based on existing metadata [29] to detect the Character aspect and the storyworld-dimension of the Events and Storyworld aspect. Finally, a fruitful next step in the field of fandom studies would be to examine the outcomes of our current best model in more detail to generate domain-specific insight into the ways sentiment is expressed and attached to particular aspects of fanfiction in the comment data.
NuNER delivered the best performance at our aspect-extraction task. Although results were poor for some aspects (especially Emotion but also Events & Storyworld), other aspects (especially General and Style) were more accurately detected. Furthermore, since the model detected many partial matches in the aspect extraction for aspects such as Canon, results are still very useful to researchers wanting to know which aspects are mentioned in connection to positive and negative sentiments in the fanfiction comment data. Our approach of using Twitter-roBERTa-sentiment proved sufÏciently suitable to detecting positive and negative sentiment. Overall, fanfiction comments seem suited to ABSA because many of the texts explicitly express sentiments regarding specifically named aspects of fanfiction stories.
To protect the privacy of the fanfiction community, the dataset of fanfiction comments will not be made available for reuse. Derived data and code are available on Github.