In recent years, computational literary studies have increasingly utilised machine learning (ML) models to analyse and classify literary texts, e.g. to predict reader appreciation [31,34] or literary success [18,45,21,9] with uptake in applications in the publishing industry. 1 Models often rely on text-intrinsic features, contributing to the study of which text characteristics serve as predictors for a given classification. While other studies have shown that literature assessment can be biased by gender [43,29] and ethnicity [15], focusing on text-intrinsic characteristics might seem like a way to avoid such biases as it concentrates solely on the text.
However, seemingly objective features can harbour social biases, reflecting disparities in the underlying data. The present work examines gender biases in ML models that predict literary canonicity, demonstrating how the uncritical use of ML models in humanities research can lead to biased knowledge production, potentially skewing our understanding of literary history and the phenomenon of canonicity. This has implications beyond academic research, as these models could influence real-world applications, including the assessment of new manuscripts by publishers based on predicted success or likeness to existing canon. By integrating insights from algorithmic fairness into our analysis of predictive models, we aim to highlight the potential for hidden biases in seemingly objective computational methods. Our analysis demonstrates how these biases can affect our interpretation of literary history and canon formation, and we emphasise the importance of critical reflection on ML methodologies in DH research.
Our findings underscore that the significance of this work lies not only in the practical application of prediction models but also in exposing the epistemic consequences of using biased ML models to study literary phenomena. This approach invites researchers to consider how computational methods may inadvertently reproduce or amplify existing biases in literary history.
This study builds on prior research demonstrating the potential of ML classifiers to predict various literary attributes, such as whether a book belongs to the literary canon, is written by a Nobel laureate, is a bestseller, is longlisted for given awards, or receives a high rating on GoodReads [8]. To narrow the scope, we will focus on the attempt to predict canonicity. While various studies focus on classifying canonical works and gauging their textual profile [6,12,33], the limited resources in the literary field are rarely openly available. We thus focus on one newly published dataset [10], which served as the foundation for Bizzoni, Feldkamp, Jacobsen, Thomsen, and Nielbo [8] and provides a rich and diverse collection of features of literary works.
In [8], the focus extended beyond classification accuracy to provide insights into the textual features important for the classification models, seeking to understand the characteristics that differentiate canonical from non-canonical books. The study found that "canonical texts have the most distinctive profile across all dimensions and are therefore the easiest to classify in the binary classification task" due to their denser nominal style, lower readability, less predictable sentiment arcs, and higher perplexity.
However, it is well-known that canonical literature -like the literary field more broadlyhas historically been dominated by men [37,30,36]. Still, studies that seek to predict some form of canonicity or perceived literary quality rarely include reflections on how biases in their data inform their results, and the cultural, temporal, or gendered dimensions of texts are rarely mentioned. While Algee-Hewitt and McGurl [1] show how the "canon" significantly changes depending on the approach taken; our study highlights the critical oversight of gender imbalances inherent in literary datasets, which can inadvertently bias model outcomes.
Previous research on gender differences in literary texts highlights key issues to avoid. One concern is treating these differences as fixed and universal markers of men's and women's writing. For example, Burrows [13] shows that gendered patterns in writing styles changed over time, with distinct differences found before 1860 but not after, indicating that gendered styles are historically contingent.
A second concern is the assumption of a binary gender model, where men's writing is seen as the default. Land [26] critiques such approaches for framing women's writing as deviant, as seen in studies like [3,25], which rely on essentialist assumptions and risk reinforcing biased interpretations of literary styles. 2With that being said, studies have found linguistic and stylistic differences between texts by men and women that are independent of topic and genre [41]. In the literary domain, Argamon, Koppel, Fine, and Shimoni [3] shows that a high frequency of pronouns is a "strong female marker", which is supported by Newman, Groom, Handelman, and Pennebaker [35] who also found that women's language more frequently includes pronouns, social words, various psychological process references, and verbs, as well as negations and home-related terms. Men, on the other hand, used longer words, more numbers, articles, and prepositions than women (p. 223).
Hiatt [20] examined contemporary (1978) American prose and found that women use twice as many emotional adverbs compared to men, while men use nearly twice as many pace adverbs. She concludes that while there is a distinct feminine writing style, there is "far less basis for labelling the feminine styles as hyperemotional than for labelling the masculine style hypo-emotional" [20, p. 226].
Hayward [19] tests whether readers can identify an author's gender and concludes that gender differences are subtler than genre differences. Koolen [24] goes deeper into the question of genre and examines the interaction of gender and genre, especially with regard to "false labelling", i.e., that works by women are more often labelled as "women's books" regardless of genre [40]. The findings suggest that while some romantic novels have distinct styles, novels by women are heterogeneous and not distinguishable from those by men. Considering the prevalence of biased mechanisms in the literary field (e.g., false labelling), it is possible that readers focus on similarities among women authors and differences among men authors rather than the reverse.
The literature reviewed here highlights the complexity of considering gender differences in literary texts and not reducing these differences to essentialist notions about "how women write. " In the following, we will use methods from algorithmic fairness to examine biases in models used to predict canonicity. We do not claim to establish definitive conclusions about the general differences between men's and women's writing; rather, we emphasise how modelling a literary phenomenon inevitably mirrors the underlying data and that results could differ if other datasets were used.
Considering that questions about bias and fairness are increasingly discussed in ML development and that ML is increasingly applied in DH, algorithmic fairness insights are rarely integrated into computational literary studies. Although Bagga and Piper [4] explored the impact of bias on predictive accuracy and positive prediction balance in literary data, our study presents a more comprehensive bias analysis informed by the methodologies of algorithmic fairness. We aim to answer the following research questions:
β’ RQ1: To what extent do ML models trained on (imbalanced) literary corpora exhibit biases on author gender in classification tasks, particularly in predicting canonicity? β’ RQ2: Which features in the dataset significantly differ between books by women and men authors, and how do these features impact the bias in classification models?
These questions are, of course, contingent on the data analysed. Therefore, we zoom out and include a question that addresses a broader concern:
β’ RQ3: How does the use of biased ML models affect the knowledge produced in computational literary studies?
This study focuses on binary gender categories, including only men and women authors. We acknowledge that this does not capture the full spectrum of gender identities and that gender is performative and shaped by discursive practices [14]. However, this approach aligns with historical perspectives and addresses existing biases between men and women in literary canonicity.
The dataset used in this work is the Chicago Corpus, which consists of 9,089 novels from diverse genres published in the US between 1880 and 2000. The data is compiled on the number of libraries holding each novel, with a preference for more circulated works. The dataset was made available with a recent paper [10]. 3 The canon category is compiled from books by authors in the Norton Anthology, the Penguin Classics series, and the top 1000 authors mentioned in English syllabi (collected by the OpenSyllabus project), as shown in Table 1.
A diverse set of stylistic, syntactic and narrative features were used in [8], which found that "[t]he highest F1 score was achieved when all proposed features were included". In addition to these features, we have included normalised frequencies of part-of-speech (PoS) features, as they have been highlighted as different in the writings of men and women (see Section 2.2).4
To replicate the experiments in [8], we employed Random Forest (RF) models for the classification task. RF models are known for their robustness to overfitting and ability to handle nonlinear relationships. For fairness analysis, we utilised the Dalex library 5 , which provides tools to explain, explore, and mitigate biases in ML models.
In this work, bias is defined as systematic deviations in predictions that favour or disadvantage one group -here, authors -based on sensitive features (such as gender, ethnicity, religion, etc.). To address this, we incorporate fairness analyses to identify and examine such biases.
Group fairness is particularly relevant in our context as it seeks equitable treatment across different groups of authors. This approach balances the distribution of treatments and resources between groups to ensure that predictions do not disproportionately favour or disadvantage one or multiple social groups [16]. Equality of opportunity, equalised odds, and calibration within groups are metrics used to estimate group fairness in predictive models. Integrating these fairness considerations into DH research is crucial, as biased tools can lead to the misrepresentation of corpora and minority groups, as highlighted in [27].
Equality of opportunity ensures that the opportunity to be classified as a true positive instance is equal for all groups, and for all social groups considered their positive rate (TPR) should be equal:
In relation to the binary classifier for canonicity, equality of opportunity ensures that the likelihood of correctly recognising a canon book is equal regardless of whether the book is written by a man or a woman.
Equalised odds extends beyond equality of opportunity by ensuring equality of the true negative rate (TNR) and the false positives rate (FPR) for all the specified groups:
For the canonicity classifier, equalised odds ensure that the likelihood of incorrectly classifying a non-canon book as canon is equal regardless of whether the book is written by a man or a woman.
Calibration Within Groups ensures that the precision of the classifier is balanced, meaning the proportion of correct positive predictions (true positives) out of all positive predictions is the same for all groups:
For the canonicity classifier, this means that the books classified as 'canon' are actually canon and that the accuracy of these predictions is consistent across books written by both men and women. Dalex reports various classification outcomes and calculates the fairness metrics outlined above. The criteria are evaluated using the following: π β€ metric for non-privileged group metric for privileged group
with π = 0.8, following the four-fifths rule [5]. This threshold is widely used to detect significant disparities in treatment between groups. The benefit of this approach is that it offers a clear and standardised benchmark for assessing fairness, while its limitation is that it may not detect subtle biases and could oversimplify complex fairness issues [39]. The groups considered in our experiments are women and men authors, with men authors being the privileged group. The outlined criteria have been shown to be impossible to satisfy simultaneously, except for trivial cases [32,23]. This is a challenging finding because it is difΓcult to justify sacrificing any of these criteria in a fair classifier. It emphasises the importance of conducting fairness analysis and interpretation within the specific context of use, considering the underlying data foundation. We prioritise equalised odds in the canonicity classifier to ensure fair treatment of men and women authors by balancing FPR and TNR across genders. Without this, one group could disproportionately influence what is deemed canonical. See Section 5 for further discussion.
Dalex was also used to estimate feature importance for the canon classifiers, employing a permutation-based approach to compute feature importance. This assesses the contribution of each feature to classification outcomes by systematically permuting them and calculating their impact on model performance.
In the first round of the experiments, we used the same sampling methods as in [8] to ensure balance between the positive and negative class: All 618 canon books are used with a random sub-sample of 618 non-canon books. This process was repeated 20 times, and the average accuracy was 0.72 -somewhat reproducing the accuracy of 0.75 reported in [8]. However, as the gender distribution is not equal for either the positive or negative class, we cannot rule out the effect of class imbalance when examining the fairness results, as models trained on imbalanced datasets often develop a bias favouring the majority class [16].
Considering this, we bootstrapped a 50-50 gender distribution to conduct a more meaningful bias analysis. Since the canon group contains few books by women authors (166 vs. 452 by men authors), we randomly sampled 166 books by men authors from the canon group to achieve gender balance, alongside 166 books by men and women authors from the non-canon group, resulting in a total of π = 664. Each sampling selects a random subset of canon books by men authors and non-canon books, and the entire process, including model training and fairness analysis, is repeated 20 times. Sampling is conducted with replacement between rounds to ensure variability between iterations. Hence, all 166 canon books by women authors are in each run used together with a random subset of canon books by men authors.
When training on a 50-50 gender distribution, the average accuracy on the 20 runs remains approximately the same, 0.71. One potential reason the accuracy is not affected by a smaller data sample is that the balanced gender distribution may enhance the model's ability to generalise across different author groups, counteracting any potential loss of information from the reduced sample size.
Out of the 20 models trained on a 50-50 gender distribution for both the positive and negative classes, 16 models are unfair according to the fairness criteria. Specifically, for 9 of the models, the FPR is lower for women authors than for men authors, and for 7 models, the TPR is higher for women authors than for men authors. The FPR results indicate that the models have a greater tendency to classify non-canon books by men authors as canon, compared to non-canon books by women authors, violating the equalised odds metric. The higher TPR for women shows that the proportion of correctly recognised canon books is greater for women authors, violating the equality of opportunity metric. To gain insights into these results, in the following section, we summarise the feature distributions in the underlying data and feature importance of the models.
Before examining the predictive models' feature importance, we first tested whether the included features differed between books by women and men authors. To do so, we conducted a Mann-Whitney U test with Bonferroni correction to account for multiple comparisons. This was done for each sample process to ensure that the findings were robust and not related to the random sample. Conducting the test on a 50-50 gender distribution sample rather than the full (imbalanced) dataset minimises the influence of unequal group sizes, providing a clearer understanding of each feature without the confounding effects of gender imbalance. The following features are reported as statistically significant between books by men and women in the canon set in more than half of the sampling rounds:
β’ Narrative features: The mean sentiment of all sentences in the book as well as the mean sentiment of the first and last 10% of the book. β’ The normalised frequencies of negation modifiers, auxiliaries, pronouns, verbs, and nominal subjects. β’ The ratio between verbs and nouns. Thus, at least some of the 36 text-intrinsic characteristics differ between the canon books by men and women authors, suggesting that there may be a distinct profile for women and men canon authors. When performing a Mann-Whitney U test with Bonferroni correction for multiple comparisons for the whole corpus of 9,000 novels, 31 out of 36 features exhibit a statistically significant difference between books by men vs. books by women authors. Hence, there are larger differences between books written by women and men in the whole corpus than there are in the canon set. Next, we examined each model's feature importance to see if the differences in features between men and women drive the observed biases.
For each model, we analysed feature importance and counted the presence of each feature in both fair and unfair models, respectively. Using the Dalex Library, which identifies the top 10 most influential features, we counted the presence of these features across all models. Fig. 2 presents an overview of the important features in both fair and unfair models, as well as the features that are reported as statistically significantly different between author genders within the canon set.
The frequency of negation modifiers, type-token-ratio, perplexity and approximate entropy are often reported among the top ten features regardless of whether the classifier is fair or unfair (w.r.t. the considered fairness criteria). Recalling the findings from [8], our results confirm the discriminating power of the textual metric perplexity.
The frequency of negation modifiers and auxiliaries is statistically significant between canon books by men and women authors in all sample runs and an important feature in all and most fair models, respectively. This might suggest that the canon vs. non-canon signal for these features is stronger than the gender difference. This may also be the case for the type-token ratio, which we report to be different between canon books by men and women in 30% of the sample runs, but important for all models.
Furthermore, the frequency of relative clause modifiers and the compressibility of the text are also important features for distinguishing canon books from non-canon books. Both features are reported more often for the fair models, indicating that despite compressibility being reported as different for men and women authors in the canon group (in 25% of the sample runs), this does not explain the observed bias.
For the unfair models specifically, we find that the stop words and verb frequency are more important than in the fair models. Verb frequency is reported as statistically significant between books by men and women canon authors in 60% of the sample runs. It is reported as important only in unfair models, indicating that relying on this feature might contribute to the observed biases. Similarly, although the frequency of stop words is only reported as statistically different in books by women and men canon authors in 5% of the sample runs, it might still add to the observed biases when combined with other features.
For the mean sentiment of the 10% first and last parts of the books, we see that they are only important for the unfair models, while they are reported to be statistically significantly different for books by men and women canon authors. This indicates that these features might contribute to the observed biases. The mean sentiment of all sentences, which is reported as statistically significantly different between men and women authors in all runs, is an important feature for 20% of both the fair and the unfair models, and we can, therefore, not conclude how it contributes to biases.
Moreover, the frequencies of nominal subjects and the verb-noun ratio are reported as different between canon books by men and women authors. However, these are not important for the classifiers to tell non-canon from canon books. This suggests that while women canon au-thors and men canon authors differ in these features, they are not important predictors for the canon category as such. On the other hand, features such as approximate entropy, perplexity, relative clause modifiers, use of stop words, and type-token ratio appear crucial for determining canonicity. Notably there are no substantial differences between men and women authors regarding these features within the canon group, suggesting a shared canon style among men and women canon writers w.r.t. these features.
The results presented in this paper show that while it is possible to predict canonicity based on text-intrinsic features, it is crucial to consider social biases in these models, such as the effect of author gender. Moreover, our results show that research in DH and computational literary studies can benefit from insights from algorithmic fairness to increase awareness of social biases ingrained into methods and datasets. In the following, we outline our main findings and discuss them in relation to earlier work and fairness considerations.
Regarding the feature importance results, it is important to note that with the 50-50 gender distribution in this work and the inclusion of PoS frequencies, we do not reproduce the same feature importances as reported in [8]. While perplexity is confirmed as having discriminative power in our results, nominal style, readability, and predictability of sentiment arcs do not appear to be significant predictors of canonicity.
As the experiments in [8] did not take gender into account, their models have been exposed to more men authors than women authors -both of the canon and the non-canon group. In contrast, our bootstrapped sampling process ensures that our models are exposed to an equal number of texts by men and women authors. Predictability and nominal style were reported as statistically significantly different in canon books by men and women authors, but these features did not emerge as predictors of canonicity in our experiments. Therefore, it seems plausible that by highlighting these exact features, the models in [8] might have picked up on a style associated with men (canon) authors rather than canonicity itself. However, keep in mind that our inclusion of PoS features in the analysis may also influence which features are reported as most important. It is possible that these features remain important but appear further down the list in our models.
These results underscore the necessity of a careful sampling process when dealing with imbalanced data. Our bootstrapping method, while straightforward, is not without the limitation of reducing datapoints. A more refined approach would involve up-sampling texts by women authors to match the distribution of existing women-authored books.
11 of the 16 unfair models have a higher FPR for men authors than for women authors, showing a tendency to over-include non-canon men in the canon, rather than non-canon women. At the same time, the TPR is higher for women authors, indicating that the models have an easier time recognising canon works by women than by men. Overall, this seems to point to a harder divisibility of the men authors' space between canonical and non-canonical books. One potential explanation for this is that the distance from the canon group might be larger for the non-canon women than for the non-canon men. The hypothesis is sketched in Fig. 3. Further work is needed to test whether this is the case, potentially through techniques like embedding-based clustering of books based on text-intrinsic features used in the present study. A closer examination of how genre plays into the effect observed is also needed, especially as a larger distance between canon and non-canon women authors may be due to other effects related to gender disparity. Women authors are shown to predominantly write in genres such as romance, children's literature, and young adult fiction [42,28]. If genres like romance are dominated by women authors and are less represented in canonical compilations [17], and if genres are closely related to writing style [22], the disparity between canon and non-canon books (such as romance novels) by women authors may be larger.
To avoid naturalising these findings, caution is required when speculating that non-canon women authors align less with the canon style; and our study does not draw definitive conclusions about the intrinsic qualities of men's versus women's writing. Previous studies have identified gender differences in texts (see Section 2.2), but these findings do not always generalise well [11]. Our analysis reflects the underlying data of the Chicago Corpus, which prioritises widely circulated books. If there is a greater disparity between canon and non-canon women authors than men authors, it could result from differential reception [24] and "false labelling" of women's works. This highlights how seemingly objective text-intrinsic features can embody social biases, as extensively discussed in [11].
As discussed in Section 3.3, the impossibility theorem of algorithmic fairness [32] shows that different metrics for group fairness are incompatible with each other if the distribution of pos-itives varies between groups -also known as unequal base rates. In our experiments, we ensured equal base rate through the 50-50 gender distribution for both the positive and negative classes. Despite this, the majority of the models displayed biased predictions based on author gender. Specifically, the FPR is higher for men authors than for women authors in 11 out of the 16 unfair models, leading to a violation of equalised odds.
In real-world scenarios, the base rates are rarely equal. Therefore, addressing such unfairness often involves accepting lower accuracy, a trade-off known as the parity-accuracy tradeoff [23]. To balance accuracy and fairness and to choose which fairness metric to prioritise, it is essential to consider the context of use and the intended goals carefully. For a canonicity classifier aimed at understanding canonical literature, it is arguably important to avoid unequal false positives, as this would result in one social group having disproportionate (false) influence over what represents canonical literature. This consideration supports prioritising equalised odds, which addresses fairness in terms of error rates across groups.
The publishing industry might make up another potential use case for binary classifiers predicting categories such as 'bestseller' or 'quality' [45,2]. If an ML classifier predicts the success of new manuscripts, it is still preferable to avoid favouring one group over another, thus supporting equalised odds. However, if human experts later sort the manuscripts, over-including false positives is not as harmful as violating equal opportunity (where one group's positive instances are more likely to be disregarded). In such a use case, the cost of being falsely disregarded is higher than being falsely recognised. Therefore, equality of opportunity is crucial to ensure manuscripts with high potential are equally likely to be recognised, regardless of the author's group (e.g., gender, ethnicity)
For a binary classifier used in the publishing industry, it is also crucial to consider the fairness criteria calibration within groups, as it ensures consistency between predicted probabilities and actual outcomes within each group. Hence, if the classifier consistently predicts a 10% likelihood of bestseller status for manuscripts written by men, then roughly 10% of those manuscripts should indeed turn out to be bestsellers when checked against the actual data, and similarly for other groups. Lack of calibration within groups could lead to systematically overconfident or underconfident predictions for certain groups.
The sections above show how biases can be embedded within ML models used to predict literary phenomena. While the field of algorithmic fairness can help identify and address such skewness, it is worth asking some more fundamental questions about the existing approaches of using imbalanced literary corpora to classify literary works. One thing to keep in mind is that developing predictive ML models relies on an assumption about the existence of classification schema, which can serve as a ground truth. In other words, justifying a canonicity classifier through its accuracy relies on an acceptance of the distinction between the canon novels and the non-canon novels and the canon's historical profile. Such considerations should not be seen as a dismissal of the idea of a literary canon per se; rather, we aim to encourage reflections about what happens when contested classification schema is operationalised into predictive models. Similar points are addressed by Piper [38]: "[W]hile statistical tests can measure the functioning of the model ("the extent to which what we are observing exceeds the boundaries of chance"), they cannot confirm "whether the model is an appropriate approximation of the phenomenon that one is claiming to observe"" (quoted in [11]).
This study emphasises the critical role of algorithmic fairness in computational literary studies, especially in addressing gender biases in classification models. Despite balanced training data, our findings show that ML models still exhibit significant gender biases, misclassifying noncanon books by men as canon more frequently than those by women, thus violating equalised odds. This suggests that ignoring gender distribution in literary datasets can bias models towards associating men-authored writing styles with canonicity and relatedly can lead to misguided ideas about the textual characteristics of categories like canonic literature.
Our results reveal that seemingly objective text-intrinsic features can harbour social biases, highlighting the need to critically reflect on potential biases in datasets and corpora. By integrating fairness considerations into ML model development and application in computational literary studies, we can not only improve the reliability of the research results but also foster inclusivity by ensuring the representation of all social groups and not just those historically included in established canons. 6Further research is needed to understand feature distributions across author genders and their impact on biases. One approach is to create embedding-based clustering to analyse how different author genders are located and distributed within and outside of the canon category.
As pointed out in section 4.3, some features (approximate entropy, perplexity, relative clause modifiers, use of stop words, and type-token ratio) appear crucial for determining canonicity while showing no substantial differences between men and women authors. This suggests a shared canon style among the canon writers, and future work could examine whether these features are consistent across different genres or literary movements within the canon and how this evolves over time.
A limitation of our experiments is that genres were not considered. Future research should incorporate genre distinctions to ensure significant features of canon literature are not conflated with genre-specific ones. This is particularly crucial in the sampling process to avoid comparing canon books against genre literature.
Another limitation is the influence of pressures from the publishing industry. Research has shown that women writers often face constraints from publishers regarding their writing style and subject matter [44,7]. While this requires further investigation, it highlights the social context shaping how literature is written, published, and distributed -factors that inevitably influence literary data and the resulting predictive models.
See https://zenodo.org/records/12699037 for code.
Number of (sampling) runs where the feature levels are statistically significant between men canon and women canon books. A statistically significant difference is defined as p<0.05, with Bonferroni correction for multiple comparisons. (F) = frequencies, normalised based on the word count. Figure 4: Distribution plots for features reported as statistically significant between men and women canon authors in more than half of the sampling rounds.