Although coreference resolution is a necessary step for a wide range of automated narratological analyses, most of the systems performing this task leave much to be desired in terms of either accuracy or their practical application in literary studies. While there are coreference resolution systems that demonstrate good performance on annotated fragments of novels, evaluations typically do not consider performance on the full texts of novels. In order to optimize its output for concrete use in Dutch literary studies, we are in the process of evaluating and 昀椀netuning Dutchcoref. Dutchcoref is an implementation of the Stanford Multi-Pass Sieve Coreference System for Dutch. Using a “silver standard” of annotated data on 2,137 characters in 170 contemporary Dutch novels, we assess the extent to which Dutchcoref is able to identify the most prominent characters and their gender. Furthermore, we explore the usability of the system by exploring a speci昀椀c narratological question about the gender distribution of the characters. We 昀椀nd that Dutchcoref is highly accurate in detecting noun phrases, proper names, and pronouns referring to characters, and that it is accurate in establishing their gender. However, the ability to cluster co-references together in a character pro昀椀le, which we compare to BookNLP's performance in this respect, is still sub-optimal and deteriorates with text length. We show that, notwithstanding current state of development, Dutchcoref can be applied for meaningful literary analysis, and we outline future prospects.
Characters are one of the primary building blocks of narratives: their subjectivity, complexity, and agency are what sets stories in motion. For that reason, the concept of ‘character’ — “a text- or media-based 昀椀gure in a storyworld, usually human or human-lik9e]” —[ is among the most fundamental in narratology, next to e.g., plot, discourse, narration, focalization and motifs. To a greater or lesser extent, understanding narratives is thus understanding characters. The importance of this unit-of-analysis is exempli昀椀ed by a wide range of narratological studies that rely heavily on a particular understanding of characters (cf. for insta5n]c)e. [
In order to arrive at a broader, empirical understanding of the concept, scholars have tried to
automate the analysis of characters, o昀琀en by doing some form of automatic character detection
in larger-scale corpora (e.g1.7[]). Where and how characters occur in texts is key for any sort of
computational narratology; it underpins a wide range of automated analyses such as character
network analysis (e.g. 1[
Character detection relies on (a variant) of coreference resolution. References and
identity linking in general are one of the major challenges in arti昀椀cial intelligence (AI). The goal
of coreference resolution is to distinguish linguistic entities by disambiguating all individual
references to them. In Tolkien’sThe Lord of the Rings one would for instance want to have
indicated that the references “Frodo” and “The ring-bearer” point to the same character, and
that speci昀椀c instances of “Mr. Baggins”, and “he” do too. Some coreference resolution systems
such as, for instance, BookNLP 2[] and the Stanford CorefAnnotator1[
BookNLP is currently only available in English, and although it has acquired funding to expand the number of supported languages, Dutch is not among the prospective supported languages. The development of a narratological informed and accurately performing BookNLPlike system for the Dutch language thus remains a strong desideratum for Dutch literary research.
Dutchcoref, an implementation of the Stanford Multi-Pass Sieve Coreference System for
Dutch literature 3[
Earlier evaluations of Dutchcoref demonstrated that in a dataset of contemporary Dutch literature, 95.0% of human mention1sare correctly recognized; furthermore, 89.9% of male mentions, and 73.4% of female mentions can be distinguished4[, p.51]. Taking those results into account, in this paper we present an analysis of a larger corpus of contemporary novels, to analyze the distribution of human mentions, as well as their gender (im)balance and dynamics.
For our twofold evaluation of Dutchcoref we use the Libris2013 corpus, consisting of all 170 submissions in one year of the annually awarded Libris Literatuurprijs, one of the most prestigious literary prizes in the Dutch language are1a0[, p. 15]. All novels were published in 2012, were written in the Dutch language, and are considered literary novels (with the corresponding NUR2 code 301.) These 170 books represent 37 percent of all the novels published in that particular year.
Earlier research on this corpu5s,[
In section 3, we use this silver standard to evaluate the accuracy of Dutchcoref. The silver standard allows us to identify all positions in the texts of the novels where a person is referenced by name or name variant. All novels are analyzed with Dutchcoref which results, among other analytical data, in a CoNLL 昀椀le with coreference information and a 昀椀le listing all “mentions”, that is: all one or multiple token fragments that Dutchcoref identi昀椀es as a mention of a person, location, organization, etc. We assess to what extent Dutchcoref is able to identify all silver data 1A mention refers to an instance of a description, name, or pronoun referring to a person or object mentioned in the text. 2NUR is a marketing instrument used by publishers and booksellers to categorize books with an eye on the different sections in bookshops. According to19[, p. 400] it would appear that “NUR can be regarded as a rough approximation of the concept of genre as it is understood by booksellers and readers.” name variants. We then continue to compare its performance to that of BookNLP. In sectio4n we progress to examine how well Ductcoref in its current state can be applied for a meaningful literary analysis of the Libris 2013 corpus.
Dutchcoref was used to analyze the text of all novels. Dutchcoref yields (among other analytical results) a CoNLL 昀椀le providing lexical, syntactic, and coreference information for all tokens in a text. We use the CoNLL 昀椀le to locate all name variants listed by the silver data. Dutchcoref also yields a 昀椀le containing “mentions”, listing all single and multi-token occurrences in a text of which Dutchcoref asserts they are nouns (or noun phrases), proper names, or pronouns referring to persons, locations, organisations, etc. Combined into a match table this information allows us to compute recall. Note that we cannot compute an F-score because there is no reliable way to determine exactly what a false positive is in this case: Dutchcoref generates many more mentions (among other things in the form of pronouns) that might very well be correctly identi昀椀ed co-references to characters, but our silver data will not tell us if they are indeed correct. Therefore, we rather compute two recall scores, a strict and a lenient score. The strict recall only scores exact matches between silver standard and Dutchcoref identi昀椀ed characters, while the lenient score allows for some leeway in the span of the text identi昀椀ed by Dutchcoref. Figure1 provides an example of lenient and and strict matching. In the interest of precision we provide the strict measure, although a human reader would easily con昀椀rm that in most instances by far the lenient matches should indeed be counted as correct.
Across the corpus as a whole we 昀椀nd a strict recall of 0.90, and a lenient recall of 0.97. A density plot of recall results (昀椀gur2e) further corroborates the high accuracy of Dutchcoref in identifying character name variants.
Dutchcoref performs well as a named entity recognition (NER) tool “on steroids”. It does not just yield proper names indicating characters, such as “Böckli” and “Fehmer”. Rather, it also identi昀椀es descriptive phrases that refer to the same characters, such as “de heer Böckli” (en.: “Mr. Böckli”) and “de beroemde architect Fehmer” (en.: “the famous architect Fehmer”), and predicts properties such as gender and number for them. Unfortunately, without a gold standard we cannot determine exactly how accurate Dutchcoref performs on this corpus in this respect, but perusing match tables for di昀erent novels indicates that most “no matches” are actually correct identi昀椀cations of compounded co-references.
Ideally a co-reference resolution tool goes beyond identifying proper names and noun phrases that refer to characters, and in addition also identi昀椀es correctly which pronouns refer to what characters. Dutchcoref tries to construct clusters of mentions that pertain to one character or object. Without a gold standard it is very hard to tell Dutchcoref’s accuracy in this respect, but we can gauge its performance a little from its behavior across shorter and longer text samples. What we observe is that Dutchcoref tends to combine too many references in one cluster if the length of the analyzed text increases. Thus, in short samples of text it seems to be functioning reasonably well, yielding groups or clusters of mentions that pertain to one particular character, and ideally these would represent meaningful characters from the novel. However, when Dutchcoref sees a longer text it starts wildly con昀氀ating references to di昀erent characters. A typical example is given in 昀椀gure3, which shows the relative contribution (y-axis) from di昀erent clusters of co-references (x-axis) to the total amount of references. The changing characteristic between the right and le昀琀 chart shows how Dutchcoref assigns more and more co-references to the same cluster if it shown a larger part of the text. The tabl4e in shows the (昀椀rst) actual references from the largest cluster from both charts. It is easily observed that while the co-reference resolution for the sample (depicted as the bottom table) might still show some coherence, the top one has aggregated far too many mutually exclusive references into one cluster, con昀氀ating several characters in the process.
We compared this progressively worsening accuracy of Dutchcoref to BookNLP’s performance, which is a neural pipeline for English literary texts. For this we selected four books (from the 19th century to current) that are not in BookNLP’s training corpus (LitBan2k]),[ i.e. “The Girl on the Train”, “The Running Man”, “The Grapes of Wrath”, and Dorothy
Wordsworth’s “Journal Volume 1”. In 昀椀gur5e we reproduce the result from Steinbeck’s novel as an illustrative measure. Our results indicate that BookNLP falls victim to the same defect, but to a much lesser extent than Dutchcoref. However, we also surmise that BookNLP’s greater accuracy is actually caused by it only taking into account coreferences for a restricted set of entity categories, rather than all noun phrases, as Dutchcoref does. Therefore, it might be advantageous to re-train the coreference system to consider only mentions referring to persons. Reducing the number of mentions than can be linked also reduces the potential for erroneous links.
Given its current performance in clustering coreferences in long texts Dutchcoref may not be a very useful tool for accurate character identi昀椀cation yet. We hope to improve its accuracy over time, so that clusters will reliably coincide with characters. For the inference of social networks of characters and their interactions this is pivotal, as merely looking at named entities leaves out a lot of information: around 40% of mentions in literature are pronouns, while names are approximately 10% (with the rest being nominal description3s]).[Yet, in its current state Dutchcoref already has potential to assist in narratological analysis. While its clustering mechanism clearly needs work, its ability to indicate names and pronouns is state of the art. Moreover Dutchcoref indicates whether a noun or pronoun refers to a human entity and tries to establish the entity’s gender, and reaches 0.84 recall on proper names given the current silver standard. Lacking a fully annotated corpus there is little sensible evaluation possible with regard to pronouns, but in 昀椀ction most pronouns by far relate to characters, in which case gender identi昀椀cation is trivial in Dutch. Together this allows us substantial insight into the gender dynamics in the Dutch literary corpus we are using.
We know from prior research by, inter alia, Corina Kool1e0n] [that gender balance in Dutch literature is skewed heavily towards the male side of the spectrum in many respects: reader appreciation, critical appraisal, awards, etc. Koolen also looked at di昀erences in vocabulary and topic use between male and female authors, using LIWC, topic models and purpose made extraction algorithms. However, Koolen did not consider how gender representation di昀ers throughout the texts of novels themselves. Using Dutchcoref we can now easily add such an aspect of narratological analysis. Using a rolling window approach we can count and average the dispersion of female and male proper names and pronouns throughout full texts, creating a visual chart of gender balance across each novel. We produced such charts for windows based on 1,000 token windows (i.e. each window is roughly two pages) progressing the window through the text token by token. We also produced charts in the same manner but based on 100 paragraphs each, progressing paragraph by paragraph. These parameters were chosen to gauge if results for a method that respects text structure (i.e. paragraph boundaries) would signi昀椀cantly deviate from a method that does not (i.e. that is token based). Consult 昀椀gures7 through10 for four examples. The chart in the top le昀琀 of each 昀椀gure shows the contribution of female names and of male names to all tokens as percentage of the 1,000 tokens in each window. The top right chart in each 昀椀gure shows the same, but as the percentage of all tokens in each paragraph. The two charts at the middle level of each 昀椀gure show the same measure but for male and female pronouns. The bottom charts in each 昀椀gure give the use of female names (or pronouns) as the ratio of all names (or pronouns) for each window of 1,000 tokens (le昀琀) or paragraphs (right).
To gauge possible skewedness at corpus level we can aggregate the numbers for individual novels. Figure6 shows a density graph of the di昀erence between the use of female proper names and pronouns and the use of male proper names and pronoun in each novel. For each novel we computed the mean percentage of female and the mean percentage of male names and pronouns across all 1,000 token windows. Then we calculated the di昀erence between these (male minus female). These give us a data point for each novel shown in the top bar of 昀椀gur6e. The density plot below that is based on the values for all the novels. If genders were somewhat balanced we would expect a bell curve centered around zero on the x-axis. Instead the center is far more to the male side of the spectrum. An aggregated corpus ratio of the means con昀椀rms that male names and pronouns are used more than twice as many times as female ones. The token-based corpus-wide mean for use of male proper names and pronouns is 0.29, while for female ones it is 0.14. Numbers for the paragraph based approach are comparable and this, together with highly congruent charts, suggests that unit of measure does not play a signi昀椀cant role in these results. Note that the observed clear bias does not fully conclusively show that Dutch literary production is a heavily male focused activity. As Kool1e0n, p[.134-6] argues, there exist known selection biases in bulk, long and short lists for literary prizes, which is the type of corpus we are working with (in this case: a bulk list of all novels submitted for a literary prize in one year). Further analysis should evaluate if the same bias exists in a random sample of contemporary literary production. The numbers we yield give us pause to think in any case.
Qualitatively, gender dispersion plots as shown in 昀椀gur7ethrough10, are useful to add a new
distant reading perspective on gender representation within particular novels. The ways in
which (male and female) pronouns and (male and female) proper names are distributed across
a novel elicits an additional dimension that can be taken into account by studies on the literary
representation of gender. Most obviously, such dispersion plots add to our understanding of
gender representation in terms of visibility. Up until now, studies taking into account visibility
as a factor of gender representation focus on how o昀琀en respectively male and female characters
occur in literary texts (e.g. 1[
As such, these plots yield a more distributed view on the gender dynamics within particular novels than studies that focus on a more restricted de昀椀nition of ‘character’. More generally, density graphs for full corpora, such as 昀椀gur6e can be used to track down individual works that either conform to or deviate from a particular norm, while rolling plots (such as 昀椀gure 7) o昀er more insight in the particular gender dynamics of a single novel. In some novels, for instance, the trend lines do not or only barely cross (e.g. see 昀椀gur8eand 9), which suggests that the over- or under-representation of one particular gender is remarkably constant and stable throughout the narrative. For instance, Herman BrusselmaGnus’ggenheimer in de mode (2012), occurs at the very right part of the plot in 昀椀gure6. From a close reading point of view, that makes sense. This particular novels excels at stereotypical representations of women, which has become one of the trademarks of Brusselmans’ authorship. In 昀椀gur1e0 a sharp distinction between male and female gender is visible, which aligns perfectly with the maledominated, reactionary 昀椀ctional world that Brusselmans has created in this particular novel. In that imaginary universe, women are constantly objecti昀椀ed and sexualized, and their narratological function solely seems to ful昀椀ll the goals and desires of the male protagonist. Such qualitative observations match perfectly with the dispersion of pronouns and proper names as shown in 昀椀gure 10: both qualitatively and quantitatively the under-representation and stereotypical characterization of female characters seems constant. Conversely, if the the trend lines in the dispersion plots of novels do cross at particular moments in the narrative (e.g. see 昀椀gure 7), that might indicate a shi昀琀 in gender representation that might have various narratological explanations in terms of e.g. focalization, narration, or motif structure. Fig8u,froer instance, represents a gender dynamic view of Lidewij Martens’ novDelubbel Rood. The parts of this graph indicating less mentions of female names seem to coincide with perspective changes in the story.
Accurate identi昀椀cation of characters in narratives, including co-reference resolution of pronouns and noun phrases referring to those characters, remains a high value desideratum for computational literary research. Our contribution shows that the current state of the art in NLP for this task is insu昀케cient to clearly, unambiguously, and accurately identify and describe characters in Dutch and English literary materials in some automated fashion. Dutchcoref shows a high recall on proper names and their gender identi昀椀cation, but co-reference resolution performance deteriorates quickly with longer (novel sized) text, causing too many co-references to di昀erent characters to be con昀氀ated. Although BookNLP performs better on English texts, this may be due to the fact that BookNLP ignores many types of referring expressions such as noun phrases describing objects. This behavior may negatively impact BookNLP’s usefulness for narratological analysis. From the current discourse it is not clear whether such noun phrases (e.g. ‘the ring’, ‘photographs’) will be highly contributory to tasks in computational literary research, although it seems plausible that accurate identi昀椀cation and resolution of such noun phrases will be important for (automated) event and plot analysis.
Their current sub-optimal performance does not preclude useful application of BookNLP and Dutchcoref in the domain of (computational) literary analysis. We have shown that with its current abilities Dutchcoref can positively contribute to, for instance, the granularity of our measurement and therefore knowledge about character gender dynamics in novels. This may well impact the current state of the art in social network analysis of novel character, as well as event analysis.
We were able to add our insights by using a “silver standard” listing the proper names and gender properties of 170 Dutch contemporary novels. To improve the performance of Dutchcoref, which we consider as one of our challenges for future work, a gold standard of a fully annotated corpus – including explicit information on the relations between all referents and antecedents – remains as much a desideratum as highly accurate co-reference resolution for literary texts.
The data of the silver standard are available in a GitHub repositohrtytp:s://github.com/roelsme ets/character-network.sData and Jupyter notebooks (Python) for our evaluation and analyses are also available via GitHubh:ttps://github.com/jorisvanzundert/computational_characte.rs