Machine classi昀椀cation of historical books into genres is a common task for NLP-based classi昀椀ers and has a number of applications, from literary analysis to information retrieval. However it is not a straightforward task, as genre labels can be ambiguous and subject to temporal change, and moreoever many books consist of mixed or miscellaneous genres. In this paper we describe a work-in-progress method by which genre predictions can be used to determine longer sequences of genre change within books, which we test out with visualisations of some hand-picked texts. We apply state-of-the-art methods to the task, including a BERT-based transformer and character-level Perceiver model, both pre-trained on a large collection of eighteenth century works (ECCO), using a new set of hand-annotated documents created to re昀氀ect historical divisions. Results show that both models perform signi昀椀cantly better than a linear baseline, particularly when ECCO-BERT is combined with t昀椀df features, though for this task the character-level model provides no obvious advantage. Initial evaluation of the genre sequence method shows it may in the future be useful in determining and dividing the multiple genres of miscellaneous and hybrid historical texts.
Thinking about large-scale development of early modern public discourse through the use of structured data is an exciting opportunity as was established by Moretti some time ag1o9. ][ Besides the use of already available bibliographic data for “distant reading”, a useful further element is to use unstructured textual databases as source material for the creation of new structured data on 昀椀elds that are currently poorly availabl1e5.][One such classi昀椀cation 昀椀eld is genre. Readily available genre information is o昀琀en sporadic, but the opportunities to use it – especially when we think that many documents are composed of several sequential genres – can open a new window to the development of public discourse. With better structured data, we will be able to study the systematization of particular genres in a new manner and take a fresh look on authorship and the relevance of publisher networks.
Much work in literary history and the history of the book has relied on the analysis of
generic categories (for examples see2[
This paper describes a number of improvements to existing methods: 昀椀rst, rather than relying on existing modern or broad classi昀椀cation systems, we use a newly-created training set of documents, with a custom-designed, domain-speci昀椀c taxonomy which attempts to balance pragmatism with capturing meaningful and 昀椀ne-grained eighteenth-century organisational categories. Second, we use a BERT transformer model which has been speci昀椀cally trained on eighteenth century texts, which performs signi昀椀cantly better than base BERT, and third, we propose a method by which we hope this 昀椀ne-grained classi昀椀cation can be used to represent books as sequences and combinations of genres.
We report on and compare results from a number of classi昀椀ers: a document-level classi昀椀er
that uses only one BERT input segment for each document (ECCO-BERT-Seq), a classi昀椀er for
text chunks, which can also be aggregated on a document-level (ECCO-BERT-Chunk), and a
character-level Perceiver model using the same input as ECCO-BERT-S1eqT.he BERT model
[
The data used both for model training and for predictions comes from Eighteenth Century
Collections Online (ECCO). ECCO is a set of 180,000 digitised documents published originally in
the eighteenth century, created by the so昀琀ware and education company Gale. 5[] These
digitised images have been converted into readable text data using Optical Character Recognition
(OCR). Despite its size, a recent study comparing ECCO to the English Short Title Catalogue
(ESTC) has highlighted signi昀椀cant gaps and imbalances[32], and the ESTC itself is known to
be incomplete. [
Key to the work leading up to this paper was to create a usable training set of documents annotated with genre labels. We began with a sample set of book records and a set of preliminary genre labels. These books were then labelled by two annotators with domain expertise. At this stage, we revisited the labels, and made some adjustments to those which had particularly low inter-annotator agreement. Once the set of genre labels had been 昀椀nalised, we annotated a large set (5,672 individual works, which correspond to 37,574 known editions, of which 30,119 correspond to ECCO documents) with genre information. A昀琀er this second round, we again checked for inter-annotation agreement, coming to a consensus following a discussion of each disagreement. The eventual 43 昀椀ne-grained categories were then collapsed into main categories for some of the classi昀椀cation tasks. These book labels were then mapped to the equivalent ECCO document IDs. The 昀椀nal set of labels are given in appendix A.
Existing categorical distinctions were either too broad (for example 昀椀ction and non-昀椀ction)
or too 昀椀ne-grained (for example the many historical literary divisions, particularly poetic) for
our needs. Our categories attempt to re昀氀ect the divisions as found in contemporary sources
such as catalogues. [
In this section, we introduce the pre-trained ECCO-BERT model, 昀椀ne-tuning models and baselines.2 We denote the training dataset as{( , )} =1, where is the book, and is the genre of . Our goal is to learn a function( ) to predict the genre for book or the genre of a chunk in book .
ECCO-BERT [
Because we want the training and prediction of the model to take into account the full information of the document, a document is torn into di昀erent chunks of 510 tokens each to train the model and predict results, since the maximum input size of ECCO-BERT is 512 tokens (510 input tokens and 2 special tokens expected by the model). For training the model, we assume that each chunk has the same genre as the document, and the model is trained with the resulting (chunk, label) pairs. During the inference procedure, we 昀椀rst split the document into chunks. The 昀椀ne-tuned model then scores each chunk; the predicted genre probability of the document is the average of all chunks’ probability. The inference process is shown in Figure 1. We call this model ECCO-BERT-Chunk. For comparison, we also train a model conditioned only on the 昀椀rst 510 sub-words of the document as input, which is denoted as ECCO-BERT-Seq.
Although the ECCO-BERT-Chunk model considers all chunks to make the 昀椀nal judgment, its prediction process is very slow since a book o昀琀en contains a lot of chunks. At the same time, the much faster ECCO-BERT-Seq is only conditioned on the 昀椀rst 510 sub-words, so it might lose some important information of other parts in the book. To solve this problem, we trained 2The model implementation is available at https://github.com/HPC-HD/ECCO-genre-classi昀椀cation. The original ECCO-Bert model has been released and is available at https://huggingface.co/TurkuNLP/eccobert-base-cased-v1 a linear model by concatenating the tf-idf features of the full text with the pooling output of the 椀昀ne-tuned ECCO-BERT-Seq. The input can be denoted as [Φ ( ), Φ − ( [∶ 510])], where Φ represents the transformer encoder and the vectorizer of tf-idf. We call the model ECCO-BERT-t昀椀df, all results shows in Table1.
There are two baseline models we adopt for comparison. The input of linear model is tf-idf features of the full document. The model only contains the linear layer, the fan-out of the linear model is the number of main or sub categories. The bert-base-cased is released1b1y], [ which we 昀椀ne-tuned directly with our training data.
There are 30,119 documents annotated by experts. 6,024 documents were randomly selected and split into development and test datasets, with 3,012 documents each. The labels contain 10 main categories and 43 sub-categories. The genre labels are presentedAin.1.
The sequence length of all BERT models is set to be 512. For 昀椀ne-tuning the ECCO-BERT-Seq model and bert-base-cased model, we only adopt the 昀椀rst 510 sub-words of the document as input. These models are trained for 100 epochs on 1 NVIDIA V100. ECCO-BERT-Chunk is 椀昀ne-tuned on 4 NVIDIA A100 GPUs; the main category model and the sub-category model were trained for 21 and 20 epoches respectively, using an early stop strategy.
The loss function of the linear model is cross entropy. We perform training for 200 epochs
with SGD with momentum [
The ECCO-BERT-t昀椀df models are trained for 220 epochs with SGD with momentum. The feature extractors are the encoders of 昀椀ne-tuned ECCO-BERT-Seq and vectorizer of linear base models. In order to make the model make more use of tf-idf features, at the 昀椀rst 200 epoches, we mask the features from ECCO-BERT-Seq. The number of tf-idf features is 500,000, the dimension of features extracted from ECCO-BERT-Seq is 768.
In addition to the primary ECCO-BERT model, we also trained the Perceiver IO mod9e]l [ on the same data as the BERT models. Perceiver is a Transformer model that decouples input size from overall model size and allows the model to scale linearly with the size of the input as well as model depth. Perceiver IO generalizes Perceiver further by allowing for arbitrary outputs. Due to their linear scaling characteristics, the Perceiver models make it practical to use character-level input data which could result in a model that is more robust against characterlevel OCR artefacts in the ECCO dataset. Testing this property is our main motivation for using Perceiver IO on this task. We pre-trained Perceiver on the ECCO data for 1 million steps with an e昀ective batch size of 768. Training is done similarly to ECCO-BERT, except that the next sentence prediction task is not used. Fine-tuning for the genre classi昀椀cation task is also similar to the BERT models, except that un昀椀ltered, byte-level data is used as model inputs.
We report the models’ accuracy for main categories and sub-categories in Ta1b.leThe confusion matrix of ECCO-BERT-t昀椀df is shown in Figure2. There is a signi昀椀cant gap between 椀昀ne-tuned bert-base-cased model and other models based on ECCO-BERT, since the bert-basecased model is pre-trained on modern language corpus, was not exposed to OCR noise during pre-training, and the language has naturally evolved between 18th century and present-day English. Although ECCO-BERT-Seq is only conditioned on the 昀椀rst 510 tokens of the document, its results are also competitive compared to ECCO-BERT-Chunk and ECCO-BERT-t昀椀df which consider the full document. As shown in Tab1le,ECCO-BERT-t昀椀df performs best since it combines the transformer feature and t昀椀df of the full document. ECCO-BERT-t昀椀df is also much faster than ECCO-BERT-Chunk because extracting t昀椀df is much faster than inference of transformer models.
Of particular note is the performance of all ECCO-BERT models over base BERT and the linear model, when looking at the more 昀椀ne-grained categories. Somewhat disappointingly, the 昀椀ne-tuned Perceiver IO models do not perform better than BERT-based models on this task in our evaluation. This would indicate that the OCR noise does not interfere with the genre detection task enough to degrade the performance of BERT-based models.
Here we report on both the evaluation of the document-level results for the main categories. The confusion matrix in2 shows that the precision of the literature category is the highest while education is the lowest. We also use the ECCO-BERT-t昀椀df model to predict unlabeled ECCO data and obtain model-predicted genre distributions. There are 177,494 unlabeled documents in total. The breakdown of predicted categories are shown in Figu3r.eAs our label taxonomy is custom-made, there is no ground truth for the entirety of ECCO to fully evaluate the accuracy of the predictions. However the predictions roughly match up with our expectations: previous analyses of the ESTC, using the existing Dewey Decimal System labels, have found that the most common subject category is religion.4][ Sales Catalogues Philosophy 2.25%
Arts
As well as using the ECCO-BERT-Seq to generate document-level predictions using average
values, we can use the individual chunk predictions directly. Here we propose a method to
use this paragraph-level detection to detect chunks within documents where the change from
one genre to another is signi昀椀cant and sustained. Because the predicted genre generally
oscillates signi昀椀cantly from one individual chunk to the next, we needed a method to capture
only sustained changes, ignoring shorter breaks within a ’run’ of the same genre. To do this,
we used the Kleinberg algorithm for detecting ’bursts’ of activity in time-series da1ta2.] [This
uses a hidden Markov process to probabilistically determine when a subsequent event will
occur. When events occur more rapidly and for sustained periods in comparison to this
determination, these are labelled bursts. The detection of the bursts were computed using R bursts
package [
To adapt this method, the most probable prediction for each chunk within each document was treated as a time series data point for Kleinberg. We have calculated sections for main and subcategories separately. The method allows for ’fuzzy’ and overlapping sections of genres. Additionally we have experimented with only retaining highly-probable classi昀椀cations which helped to further 昀椀lter out noise. There are drawbacks: because the burst method looks for change rather than simply all clusters of events, currently not all sections are detected if most of the text is of a single genre.
To give some examples, we take some exemplary texts and calculate genre bursts. To visualise the changes in genre, top genre predictions (over .5 probability) are charted as a scatterplot in the paragraph sequence, coloured by genre. Burst start and end points are overlaid as coloured areas. As the method looks for periods of change rather than absolute values, it ignores the main category of the book (which is detected by the document-level method successfully anyway) and in most cases highlights sustained excerpts where the detected genre is di昀erent to the dominant one. Here, we see that David Hume’sPolitical Discourses (Figure4, A) contains discrete sections on economics (categorised as scienti昀椀c improvement), philosophy (a section on the balance of power), history (a section on ’ancient nations’) and 昀椀nally law (a chapter on the idea of the commonwealth)W.ealth of Nations (Figure4, B) begins with a section on labour and society categorised here as philosophy and smaller sections on law (a discussion on a speci昀椀c statute), and in the education genre. Most of the book is not classi昀椀ed as its dominant genre (economics and trade, under the higher-level category scienti昀椀c improvement) as it does not involve change. VillierM’siscellaneous Works (Figure4, C) detects a large number of overlapping genre changes. FinallyR,obinson Crusoe (Figure 4, D) is also mostly without detected bursts, but of note is a section of religious genre, corresponding to a section in the plot where Crusoe is ill and has prophetic dreams.
A1.0 0.9 0.8 0.7 0.6 0.5
In this paper we aimed to describe the process to detect sections of fuzzy and overlapping genre excerpts within individual editions. The results show that at the level of 昀椀ne-grained divisions (43 subcategories), a model which combines the t昀椀df feature of the full document and the features of a 昀椀ne-tuned ECCO-BERT model performs signi昀椀cantly better than baselines, suggesting they may be particularly useful for such tasks. That the BERT model performed so well on 昀椀ne-grained categories is signi昀椀cant because existing methods to look at genre have generally used very broad divisions (such as 昀椀ction and non-昀椀ction). The kinds of questions we are interested in use more 昀椀ne-grained categories, for example looking at the rise of medical textbooks in certain publishers. This kind of sequencing also has other potential uses, for example document retrieval. On the present task, we did not observe any improvement o昀ered by the Perceiver model, which we speci昀椀cally included to test a character-level model which is capable of accounting for OCR artefacts. At present, we think this is due to a combination of two factors: Firstly, the base performance on the task is around 95% accuracy, leaving only very little headroom for improvement with more advanced models. And secondly, the task is by its nature a document-level task and the good performance of the linear baseline demonstrates that enough information is present in the data even without explicitly accounting for OCR errors. It is therefore possible that the advantages of character-based models such as the Perceiver will be demonstrated on tasks where the correct modelling of individual word occurrences in their context plays a more signi昀椀cant role. These would include various text tagging and information retrieval tasks.
In our future work we hope to further develop the sequencing method, and investigate the genres in their own right, for instance looking at the sequence patterns of individual authors, the relationship between intra-book diversity and the success of particular authors or publishers, and understanding co-occurrence between genres. [20]
M. Poovey. “Mary Wollstonecra昀琀: The Gender of Genres in Late Eighteenth-Century England”. In:NOVEL: A Forum on Fiction 15.2 (1982), pp. 111–126. url: http://www.jsto r.org/stable/134521.9