Categorizing literary works based on the time period in which the story is set can enhance the accessibility of these works to library patrons through online catalogues. However, this categorization is not frequently implemented, which makes it difficult for patrons to find works based on their historical context. The reason is that assigning time period categories is a tedious and time-consuming task that requires extensive analysis by librarians and cataloguers. To address this issue, this paper suggests using machine learning techniques, Latent Dirichlet Allocation (LDA), Term Frequency-Inverse Document Frequency (TF-IDF), and Word Embedding (WE) with Sentence-BERT (SBERT) to categorize literary works by time periods automatically. LDA identifies underlying topics within the text, TF-IDF measures word importance, and SBERT measures the semantic similarity between sentences, enhancing the precision of categorization. The study evaluates the accuracy of these different techniques using a dataset of Swedish historical fiction pieces from the Swedish Literature Bank while performing quasi-experiments. Based on the F1-score, the study found that TF-IDF and LDA are effective techniques for categorizing text data by time period. On the other hand, the results indicate that WE with SBERT did not perform well for any of the three time periods analyzed.
One way to organize literature is by including metadata about specific time periods. Frommeyer [
The National Library of Sweden and its Metadata Office are responsible for maintaining the
guidelines for categorizing literary works by time periods; educated cataloguers at different libraries
are responsible for this categorization. In Sweden, the joint library catalogue (Libris) which is used by
cataloguers has around nine million printed works of fiction and two million printed works of
nonfiction. A search in Libris database shows that out of the 12 million works, 108,816 works had a
dedicated time period specified. These numbers support the claim made in Dalli's [
Although it is possible to categorize literary works by time periods, it is often not prioritized,
especially in the case of fiction. A possible reason for the lack of time period metadata could be that it
is time-consuming and challenging to decide on what metadata to specify in each case. Cataloguers
categorizing in Libris can obtain time period information from the author, publishers, other cataloguers,
or organizations. In the absence of time period information, cataloguers must read the blurb or a portion
of the text, search the internet or ask colleagues. Understanding text characteristics is a crucial step, and
it is where cataloguers must spend most of their time when cataloguing. A poor understanding of the
work’s characteristics can lead to mis-categorization. Therefore, a commonly used principle for creating
metadata that represents the time periods of a literary work is that the work must contain a minimum of
20% of its content dedicated to describing one or more time periods [
Understanding text characteristics is known as feature engineering, which is a common task in ML
used to identify patterns in data and make predictions [
In this study, the aim is to evaluate and compare different feature engineering techniques to determine which one analyses the data most accurately leading to accurate predictions of time periods for Swedish historical fiction literature from the Swedish Literature Bank. By using ML techniques to aid cataloguers in their decision-making process, this study could improve the accuracy and efficiency of time period categorization. The techniques that are being compared are: Latent Dirichlet Allocation (LDA), Word Embedding with Sentence-BERT (WE SBERT), and Term Frequency-Inverse Document Frequency (TF-IDF), and they are evaluated by F1 score. More information is provided in the Methods section.
The topic of fiction content analysis and retrieval has become increasingly significant in the context
of knowledge management and literature organization, as pointed out by Saarti [
There are numerous ways of categorizing fiction, as evidenced by various attempts. However, one
prominent approach involves analysing the content. ML can simplify this task by enhancing the ease of
conducting content analysis. Manger [
"Time" by Barbara Adam [
There are many different types of categories that can be used to categorize time. The most basic type
of categorization is by identifying periods and events in the order of occurrence in a particular year or
decade [
Because time can be categorized in numerous ways, it can be challenging and contentious for
scientists to agree on how to conceptualize and determine the start and end points of time periods [
Quasi-experiments were used in this study to categorize Swedish historical fiction texts using three ML techniques. The efficiency of these techniques is measured using the F1-score.
According to Cook [
This research utilized historical fiction novels from the Swedish Literature Bank, a nonprofit initiative to offer free access to digital versions of Swedish literature. Initially, 48 novels were searched and retrieved in full text by submitting the phrase "historical novels" to the search interface. Historical fiction novels were chosen due to their basis on historical events, making them easier to categorize into a time period than other novels. Since the novels did not have time period data, they had to be manually categorized, resulting in a dataset of 35 novels, which is considered a small dataset for ML. The novels were also lengthy, ranging from 38,000 to 186,000 words. To address these issues, the novels were sliced into smaller chunks, resulting in 1,055 individual texts with around 3,500 words each. Slicing the novels into smaller pieces helped increase the dataset size and made the texts more manageable for ML. When I mention ‘novels’, I am referring to complete, unaltered literary works. On the other hand, when I use the term ‘text’, it denotes the novels that have been divided into smaller segments or sections. The distinction is important since feature engineering and categorization is done on the smaller segments of text and not the whole novel.
The Metadata Office and Bergsten and Elleström offer suitable time period categories for literature categorization. However, time periods by Bergsten and Elleström and the Metadata Office were slightly revised due to the data used in this study. There was a considerable number of novels with Medieval time periods, therefore this period was added as a supplementary category, as suggested by Bergsten and Elleström. The Vasa period was excluded from the list of time period categories since there was only one novel set in that time. The Karolinska period was merged with the Era of Great Power because subcategories were not allowed. Additionally, different periods ended and started in the same year, creating ambiguity. To avoid this, one period had to end a year before the next period started. After the revisions, the time period categories are as follows: 1200-1520, Medieval period; 1611-1717, Era of Great Power; 1718-1772, Age of Liberty; and 1773-1809, Gustavian Period.
A script was created to scan each novel for any numerical years mentioned. These years were important in determining when in time the story takes place. The years extracted from each novel were individually examined to select a time period. If a novel had no years mentioned, it was excluded from the dataset. However, if a novel had too many different years mentioned (30 or more), and none stood out, then they were also excluded to avoid guesswork. Historical fiction novels usually have titles that mention important details like the name of kings, queens, wars, or years, and as an additional step, the novels titles were compared to the extracted years to increase the accuracy of the time period categorization. After the manual categorization, each time period category had 100 texts each. The steps of manual time period categorization are shown in Figure 2.
In order to make sure that the texts were consistent for analysis, several steps were taken during preprocessing. First, all the texts were converted to lowercase to make sure that words that meant the same thing, like "King" and "king," were treated as identical. Additionally, the texts were broken up into smaller parts, or tokens, by splitting it into individual words. Some words and punctuation marks that didn't add to the interpretation of the text, such as "I," "and," "she," ".", ":", and "?," were removed. The texts for LDA and TF-IDF were cleaned, but for WE with SBERT, the texts remained as they were. This is because SBERT works on the sentence level, and splitting the words or removing punctuation makes it challenging to distinguish sentences apart. 4.3
After the pre-processing stage, the texts were transformed from whole texts into a sequence of chosen words. However, ML algorithms cannot directly use words in this format; thus, the words need to be converted into numerical values. To achieve this, each text is represented as a list of numbers, also known as feature vectors. This study used three different techniques to create these feature vectors: LDA, WE, and TF-IDF.
LDA is a type of generative probabilistic model in the topic modelling family that aims to identify the set of topics present in a document. LDA assumes that a document contains words that correspond to various topics and assigns a set of topic probabilities to each document. These probabilities range from 0 to 1, with 1 indicating a strong probability that the topic exists in the document and 0 indicating a low probability [22]. To find the optimal number of topics, various topic numbers were tested. Using fewer than five topics gave poor results, which caused the algorithm to incorrectly categorize a text in 50% of the time, which was as good as a random guess. However, as the number of topics increased, there was a clear improvement in the results. For example, using ten topics was better than using 5, and using 15 topics was better than using 10. This pattern continued up to 20 topics, after which there were minimal improvements. Beyond 20 topics, the results worsened, dropping from 79% to 78% in correctly predicting a text's time period. This suggests that meaningful topics could be formed with 20 topics, which are used in this study.
TF-IDF is a statistical model that measures a word's significance in a document or a collection of documents. It does this by calculating the frequency of the word in the document and weighting it according to how common or uncommon the word is across all documents [24]. The model uses two measures: term frequency (TF) and inverse document frequency (IDF). TF counts the number of times a word appears in a specific document, while IDF measures the rarity of the word across all documents. A high document frequency means a lower IDF score, while a low document frequency means a higher IDF score. Words that appear frequently in 90% of the documents were removed because they carry little meaning and do not provide information to differentiate one document from another. Similarly, words that are too rare (appear in less than 10% of the documents) or unique were also removed because they are unlikely to be useful in distinguishing between documents. Misspelt words or proper names that appear only in a single document were also unlikely to help identify relevant documents.
In natural language processing, word embedding represents words as numerical vectors in a highdimensional space. SBERT is a type of pre-trained model available for generating sentence embeddings [23]. The training aims to maximize the similarity between the sentence embeddings of semantically similar sentence pairs, and minimize the similarity of semantically different sentence pairs. When given a sentence as input, SBERT generates a fixed-length vector representation of the sentence called sentence embedding, which captures the semantic meaning of the sentence input. This is achieved by encoding the input sentence into a sequence of token embeddings using the pre-trained BERT model. Unlike LDA and TF-IDF, the data were not preprocessed for SBERT because punctuation is needed to distinguish between sentences.
In this feature engineering step, the texts were transformed into feature vectors, which are numerical representations of the text. Each algorithm has produced its representations by analyzing the text from different perspectives. 4.4
Logistic Regression is a model used for categorical dependent variables, specifically for binary categorization problems. In this study, there are four categories of time periods, which creates a multicategory categorization problem. The approach to solving this problem is called "one vs all", where one category is compared to the remaining combined categories. The logistic regression model uses feature vectors to predict whether a text belongs to category A by outputting a 1 or 0. Since Logistic Regression is a supervised algorithm, pre-labelled data is needed to train it. In this case, each text was first manually labelled with its correct time period before the model training. K-fold cross-validation with 10 folds was used to train and test the LDA, TF-IDF, and WE with SBERT models. The output from this step is a categorization model that predicts the probability of a text belonging to a certain category. 4.5
The three models - LDA, TF-IDF, and WE with SBERT - were evaluated separately and compared using the F1-score, a common accuracy measure of categorization models. The F1-score is the harmonic mean of precision and recall, with a maximum value of 1.0 indicating perfect precision and recall. The minimum value of 0 indicates that the categorization models did not identify any true positives correctly. When dealing with multiple categories, an overall F1-score is not computed. Instead, a onevs-all scoring method determines the F1-score for each category. This method assesses the performance of each category individually, using precision and recall. In categorization tasks, precision and recall are two metrics commonly used to evaluate the performance of a model. The F1-score is a metric that combines precision and recall into a single score that reflects the model's overall performance. Precision is a metric that measures the proportion of true positives out of all positive predictions the model makes - see Equation 1. Precision measures how many instances the model predicted as positive are positive. A high precision score means the model makes very few false positive predictions. Recall is a metric that measures the proportion of true positives from all actual positive instances in the dataset - see Equation 2. Recall measures how many positive instances in the dataset are correctly identified by the model. A high recall score means that the model correctly identifies many positive instances. The F1 score is a weighted average of precision and recall, with equal weight given to both metrics - see Equation 3. A high F1 score means the model has high precision and recall, which is desirable in many categorization tasks.
1 = = =
+ + × + (1) (2) (3) Precision Recall F1
1 e r of 0,74, which equals to correct predictions in 74% of the time. Great power and Gustavian period.
As a contrast to TF-IDF, Figure 5 shows that WE SBERT has low score on all metrices. Medieval period and Age of Liberty have scores close to 0,5 which means that only 50% of the texts were categorized correctly. Era of Great power and Gustavian period have even lower F1-score and it means that more text were categorized wrong than correct.
WE SBERT
The results improved as the quasi-experiments proceeded. The study results suggest that TF-IDF and LDA are promising feature engineering techniques for analyzing text data across different time periods, while WE SBERT may be less effective in this context. Both LDA and TF-IDF performed consistently well across all four time periods, with scores ranging from 0.74 to 0.99, while WE SBERT had lower scores across all four periods, about and under 0.5. Several factors may contribute to the differences in performance between these techniques. For example, LDA and TF-IDF are wellestablished techniques widely used in natural language processing and may be better suited to analyzing texts from a range of time periods. Additionally, LDA and TF-IDF rely on statistical methods that can identify patterns and trends in text data, which may be particularly useful for identifying similarities and differences across periods. On the other hand, WE SBERT is a newer technique that uses neural network models to generate vector representations of sentences. However, using too many sentences may result in lower-quality embeddings, which affect the categorization.
As Saarti [
The categorization of fiction has traditionally relied on formal and external aspects such as genre,
literary form, author, place, and language. However, as pointed out by Almeida and Gnoli [
Additionally, as ML techniques getting more available, areas of fiction categorization that were
previously explored manually can now be explored with ML, such as identifying if a text is fiction or
nonfiction, as done by Manger [
The findings in this study have several implications for future research and practical applications. First, researchers and practitioners interested in analyzing text data across different time periods may benefit from using LDA or TF-IDF as feature engineering techniques, as these techniques have shown consistent performance across all time periods. Second, more experiments should be done to optimize WE SBERT for time period categorization, i.e. to vary the number of input sentences to get higher performance. It is important to note that this study had some limitations, including the small sample size, and not enough data to cover the Vasa period. Future research could explore the performance of these techniques on larger or more diverse datasets with more time periods, and investigate different pre-processing and categorization algorithms. Moreover, this study explored one-to-one categorization, meaning that one text could only belong to one time period, whereas future research could explore multiple possible categories for each given text.
This study aimed to evaluate and compare three feature engineering techniques to discover which technique excels in engineering features, with the aim of categorizing historical fiction novels by the correct time period: Medieval Period, Era of Great Power, Age of Liberty and Gustavian Period. This was done by experimenting with the following ML techniques: LDA, TF-IDF, and WE with SBERT. Overall, the results suggest that TF-IDF and LDA are promising techniques for categorizing text data across different time periods, while WE with SBERT produced poor results for all three time periods.
I want to thank Libris for supplying the information used in this study. Moreover, I extend my thanks to my supervisors and colleagues for providing guidance, insights, and valuable comments.
[22] D. Blei, A. Ng, M. Jordan, Latent dirichlet allocation, Journal of Machine Learning Research 3.1 (2003) 993-1022. [23] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084, 2019. URL: https://doi.org/10.48550/arXiv.1908.10084. [24] S.Shalev-Shwartz, S. Ben-David, Understanding machine learning: From theory to algorithms, 1st. ed., Cambridge University Press, 2014.