This paper presents a new corpus for the Italian language representative of the fanfiction genre. It comprises about 55k usergenerated stories inspired to the original fantasy saga “Harry Potter” and published on a popular website. The corpus is large enough to support data-driven investigations in many directions, from more traditional studies on language variation aimed at characterizing this genre with respect to more traditional ones, to emerging topics in computational social science such as the identification of factors involved in the success of a story. The latter is the focus of the presented case-study, in which a wide set of multi-level linguistic features has been automatically extracted from a subset of the corpus and analysed in order to detect the ones which significantly discriminate successful from unsuccessful stories
Computational Sociolinguistics is an emergent
interdisciplinary field aimed at exploiting
computational approaches to study the relationship
between language and society
Copyright c 2020 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
a characterization of this new genre with respect
to more traditional ones
From a computational linguistics standpoint,
one perspective from which fanfiction has been
investigated aimed to infer the relationship between
user-generated stories and their original source,
e.g. comparing the representation of characters
according to their gender, as well as to model reader
reactions to stories
The corpus comprises texts collected from efpfanfic.net, a portal active since 2001 which allows users to publish stories and to comment on them. The website is made up of two sections: one for original stories and the other for fanfictions. We considered only the latter and we limited the collection to stories based on the fantasy saga by the British writer J.K. Rowling, “Harry Potter”. This choice was motivated by the main purpose of our analysis, i.e. characterizing the success of a novel with respect to its writing style rather than as an effect of the various subject matters it deals with. At the same time, the preference given to a very popular book allowed us to keep a consistent number of potential readers and reviewers across the corpus, still having a large sample of texts to analyze. The data collection was performed through web scraping, with two spiders written in Python using the open-source Scrapy framework. The first spider crawls the list of stories in the category of choice and extracts their first chapters together with some metadata, including the URLs of the subsequent chapters. The second spider takes these addresses as input and downloads texts and additional information about all the chapters after the firsts. In the dataset created this way, the record for each chapter includes: ID and Reference ID, combinations used by the website to identify the webpage of each chapter. We use the ID of the first chapter as a reference to group together records belonging to the same story; Title; Rating, an estimate given by the author about the rawness of themes and scenes contained in his story; Date of posting; Author’s nickname; Number of chapters in the story; Text; Total number of reviews received by the story, divided in positive, negative and neutral; Number of reviews received by the single chapter, as well as the text of the most recent ones. The crawlers downloaded 54,717 stories, for a total of 19,7310 chapters and a mean of approximately 3.6 chapter per story, which is consistent with the one calculated taking into account every entry on the website. The obtained corpus was divided into folders, each containing stories with the same number of chapters.
Based on the newly created dataset, we carried
out a computational stylometric analysis aimed at
studying whether there is a connection between
the success of a fanfic story and its writing style.
Such a connection has been demonstrated for more
canonical literary works covering novel and movie
domains
The methodological framework of our
investigation is linguistic profiling
All texts were pre-processed by means of
regular expressions, with the aim of removing
errors and inconsistencies in the use of
punctuation, capitalization and special characters, in order
to increase the reliability of automatic linguistic
annotation and the process of feature extraction,
which were performed using the Profiling-UD tool
In what follows we first provide an overview of the linguistic features used for our statistical analysis and then we discuss the ones that turned out to be more prominent in successful writing. 3.1
The set of features is based on the one described
in Brunato et al. (2020) and counts more than 150
features, distributed across distinct levels of
linguistic annotation and computed according to the
Universal Dependencies (UD) annotation
framework. These features have be shown to be
effective in a variety of different scenarios, all related to
modeling the ‘form’ of a text, rather than the
content: e.g., from the assessment of sentence
complexity by humans
Raw Text Features: Document length computed as the total number of tokens and of sentences ((#Tokens, #Sentences in Table 2); average sentence length and token length, calculated in tokens and in characters, respectively (Sent length, Word length).
Lexical Richness: Distribution of words and
lemmas belonging to the Basic Italian Vocabulary
of all grammatical categories, with respect to the Universal part-of-speech tagset (UPOS * and the language specific tagset (XPOS *); Distribution of verbs according to tense, mood and person, both for main and auxiliar verbs (aux *; V *)).
tion of verbal roots and of verbal heads for sentences (VerbHead); features related to the arity of verbs (i.e. average number of dependents for verbal head, distribution of verbs by arity).
Global and Local Parsed Tree: Average depth of the syntactic tree (MaxDepth); average depth of embedded complement chains headed by a preposition; average length of dependency links and of the maximum link (Links Len; Max Link Length); relative order of the subject and object with respect to the verb;
Syntactic relations: Distribution of typed UD dependency relations (dep *);
Use of Subordination: Distribution of main and subordinate clauses (Main clause, Subord clause), average length of subordinate chains, distribution of subordinate chains by length. 4
For each considered feature we calculated the average value and the standard deviation in the two classes. We the assessed whether the variation between mean values is significant using the Wilcoxon rank sum test. We found that 57% (i.e. 126 out of the 219) of features are differently distributed in a significant way between successful and unsuccessful stories. In Table 2 we report an extract of the most interesting ones.
As it can be seen, successful stories are on average longer in terms of number of tokens and sentences (1, 2), although these sentences are generally shorter (3), suggesting that readers appreciate more a plain writing style. However, when lexical factors are considered, the preference is given to texts exhibiting less frequent words, as suggested by the slightly lower distribution of words belonging to the Basic Italian Vocabulary (5,6) and especially to the Fundamental one (7). Inflectional morphology also appears as a domain of variation between the two classes. Successful fanfictions employ quite more often verbs in the second person (15), a feature typical of narrative writing related to direct speech. On the contrary, we observe a higher distribution of third person verb, specifically auxiliaries, both singular (14) and plural (13), in less successful texts, which can hint at a preference for reported speech. Unsuccessful
Example (Italian) La citta` di Edimburgo era sommersa da una cascata d’acqua. Pioveva. Pioveva da giorni e giorni, senza sosta. Il cielo era illuminato di lampi e scosso da tuoni.
Le strade erano vuote. Per la prima volta da giorni, allo scoccare della mezzanotte, la pioggia cesso` di colpo. Il silenzio piombo` sui quartieri che sembrarono improvvisamente piu` bui. E in quel silenzio penetrante, l’unico rumore che si riusciva a distinguere era un tactac-tac leggero e discontinuo. Proveniva da una finestra. La finestra di una lussuosa casa in centro, l’unica luce accesa a quell’ora. Joanne era davanti al computer, fonte di quel tremolio e scriveva.
Batteva le dita sulla tastiera per alcuni istanti, poi si fermava, rileggeva, cancellava e riscriveva. Andava avanti cos`ı da giorni. I suoi occhi erano stanchi, ma la sua mente lavorava frenetica. Mancava poco2.
Il cielo era tetro cosparso di nuvole che sembravano volere annunciare un acquazzone, il vento ulula forte facendo sbattere le finestre violentemente, come se volesse gridare, liberarsi da una rabbia repressa. La donna dai lunghi capelli rosso scuro continuava a fissare la devastazione attraverso il vetro che ora si era appannato dal suo stesso respiro. Aveva lo sguardo malinconico non piu` illuminato da quella dolce espressione che il riso le donava. Una mano le si poggio` sulla spalla e giro` pian piano il volto verso la persona amata che con un ritmo lento comincio` ad accarezzarle le gote che assunsero un colorito roseo alla sua pelle pallida. Chiuse gli occhi come per assaporare quel dolce tocco che ora si era spostato nei suoi capelli. “Non guardare piu` oltre il vetro” Mormoro` la voce con una nota di preoccupazione, apparteneva a James, marito di Lily la donna dai lunghi capelli rossi3.
Example (English) The city of Edimburgh was flooded by a cascade of water. It was raining. It had been raining for days and days, relentlessly. The sky was lit by lightning and shaken by thunder. The streets were empty. For the first time in days, at the stroke of midnight, the rain stopped abruptly. Silence fell upon the districts that suddenly seemed darker. And in that piercing silence, the only noise that could be recognized was a faint and irregular tac-tac-tac. It was coming from a window. The window of a luxurious house in the city centre, the only light still on at that time. Joanne was in front of the computer, source of that trembling and was writing. She tapped her fingers on the keyboard for a few moments, then stopped, reread, deleted and rewrote. She had been going on like this for days. Her eyes were tired, but her mind was working frantically. Almost there.
The sky was bleak strewn with clouds that seemed to want to announce a downpour, the wind howls loudly making the windows slam violently, as if it wanted to scream, to free itself from a suppressed anger. The woman with the long dark red hair kept staring the devastation through the glass that was now clouded by her own breath. Her melancholic gaze was no longer lit up by that sweet look that laughter gave her. A hand rested on her shoulder and slowly turned her face towards the loved one who started slowly caressing her cheeks which took on a rosy tone on her pale skin. She closed her eyes, as if to savor that sweet touch that had now moved into her hair. “Don’t look beyond the glass anymore” Whispered the voice with a note of concern, it belonged to James, husband of Lily the woman with long red hair. ditionally we can see that balanced marks (24), i.e. syntactic categories, there is a significant differparenthesis and quotation marks, occur more in ence in the usage of the most common punctuation successful texts, strengthening our previous claim marks, commas (25) and full stops (26), which about a more frequent presence of direct speech are quite more frequent in highly-reviewed fanin this class. At syntactic level, dependency relafictions. These features relate themselves to the tions are slightly shorter in successful texts, both previously observed difference in terms of docuconsidering the average value of all dependenment length, as texts with more sentences necescies (29) and the value of the maximum depensarily use punctuation marks to divide them. Addency link (30). In readability assessment stud2The full story can be found at https://efpfanfic. net/viewstory.php?sid=607026&i=1
3The full story can be found at https://efpfanfic. net/viewstory.php?sid=27412&i=1 ies, longer syntactic dependencies are typically found in complex texts, and the same holds for deeper syntactic trees. Both these features have lower values in highly-reviewed stories, suggestsignificantly between successful and unsuccessful stories. All differences are significant at p < 0.001, except for features marked with an asterisk, which have p < 0.05. ing that the style of successful writing is characterized by a simpler syntactic structure. Interestingly, these results, although preliminary, go in the opposite direction to those reported by Ganjigunte et al. (2013) for successful literary works in English, which where found to be less correlated with text readability scores. Finally, subordinate clauses (33) occur slightly more often than main clauses (32) in unsuccessful texts, while there is a nearly even split between hypotaxis and parataxis in successful ones.
To deepen our analysis, we also computed the coefficient of variation * for all features varying significantly between the two classes, where * is the ratio between the standard deviation mean . This allowed us to evaluate the dispersion of values around the average in a standardized way, and thus to compare the stability of features pertaining to data measured on different scales. A feature that is much scattered in a class of texts and highly stable in the other has a greater chance of being a meaningful representative of the latter.
In Figure 1 we show the average variability in the two classes of the four groups of features distinguished according to the level of annotation they were extracted from. As a whole, we noticed that successful texts display less variability in nearly every considered feature: 117 of them (92%) are more stable in this class. In successful stories, features with greater stability compared to the other class are mainly raw text, e.g. number of sentences, number of tokens and syntactic ones, e.g. verbal heads per sentence and average depth of syntactic trees. Among the few features which are more stable in poorly received texts, we find instead verbal predicate features, such as the distributions of past tenses and of indicative moods, in addition to the frequency of usage of cardinal numbers. The set of lexical features is instead the most stable one for both classes. class of features, both for successful and unsuccessful texts.
In this paper, we presented a NLP-based stylometric analysis on the emerging genre of fanfiction aimed at characterizing the writing style of a successful story. We collected a new large-scale corpus which – to the best of our knowledge – is the first one of this genre for Italian. We showed that successful stories, defined as those receiving a number of reviews higher that the average, are characterized by a variety of linguistic features at different levels of granularity and that these features are more uniformly distributed within them.
In the future, we would like to broad the perspective to other genres in order to study whether there are linguistic predictors of successful writing which are constant across different genres, as well as across concepts somehow similar to success, such as virality and engagement.