When coping with literary texts such as novels or short stories, the extraction of structured information in the form of a knowledge graph might be hindered by the huge number of possible relations between the entities corresponding to the characters in the novel and the consequent hurdles in gathering supervised information about them. Such issue is addressed here as an unsupervised task empowered by transformers: relational sentences in the original text are embedded (with SBERT) and clustered in order to merge together semantically similar relations. All the sentences in the same cluster are finally summarized (with BART) and a descriptive label extracted from the summary. Preliminary tests show that such clustering might successfully detect similar relations, and provide a valuable preprocessing for semi-supervised approaches.
Recent applications in the field of Natural Language Processing (NLP) are exploiting data-driven techniques from the general area of Machine Learning (ML). These are typically Deep Learning (DL) systems based on multi-layer neural networks fitted with the input text data, to be converted in numerical objects by some embedding scheme. Such DL-NLP systems are successful in extracting knowledge from natural language and capturing the underlying narratives.
As a matter of fact, most of these NLP efforts are focused on a few mainstream applicative areas, such as biomedical literature [Zhang et al., 2018; Lv et al., 2016] or news and social media [Trieu et al., 2017; Ghosh and Shah, 2018]. Other inputs such as literary text in the form of novels or short stories received less attention [Wohlgenannt et al., 2016; Volpetti et al., 2020]. This is unfortunate as literary texts might exhibit high complexity in the narrative plots, while also lacking explicit annotations, thus making the knowledge extraction process very challenging. Handling such complexities, helps in evaluating the models Natural Language Understanding and creating benchmarks for these low-resource domains. This could in turn be helpful for common sense reasoning, reading comprehensions and enhance NLP applications such as summary generation, machine translations and question answering.
Despite their astonishing applications in NLP, e.g., [Zhu et al., 2019; Paulus et al., 2017], DL models are typically based on discriminative functions with a huge number of parameters, whose interpretation is often problematic. This prevents both the explainability of the results and the possibility of doing reasoning over the model entities. For this reason, alternative approaches to NLP, based on so-called Knowledge Graphs (KGs), i.e., relational ontologies providing interlinked descriptions of the entities involved in a text, are also popular. Despite the existence of techniques for automatic KG extraction acting at the syntactic level [Tang et al., 2016; Ruan et al., 2016], most of the approaches require supervision in the form of manual annotations or access to knowledge bases, such as UMLS1, for higher level descriptions.
Of course these two orthogonal perspectives, say DL and KGs, can be combined. DL models can be trained from KGs [Socher et al., 2013; Li and Mao, 2019] and used for ML, and, vice versa, DL models such as embeddings can be used to predict missing links of the KG, classify relations, or align entities from different KGs [Liu et al., 2019; Lin et al., 2015].
Here, we follow such an integrated point of view, being
motivated by specific features of literary text understanding.
In fact, for this kind of text, the KG entities are typically the
characters in the plot, and no serious alignment issues
appear, while the classification of the relations becomes much
more challenging because of the lack of supervision. In other
domains the number of possible relations is typically limited
descriptions of relations between the characters of a novel. Our preliminary tests seem to be promising with respect to the proper identification of similar relations, while also giving directions about the most suitable clustering strategies as well as further development of semi-supervised tools.
The paper is organized as follows. In Section 2, we summarize the existing literature in the field. Section 3 describes our workflow, which is demonstrated by applicative examples in Section 4. Conclusions and outlooks are in Section 5. 2
As discussed in the previous section, DL tools such as sequence and self-attention models as well as transformers (e.g., BERT, Xlnet, BART) have been widely and successfully used in NLP for word and sentence encoding [Peters et al., 2018; Devlin et al., 2018; Yang et al., 2019; Lewis et al., 2019]. These models can be fine-tuned and used for various tasks such as classification, summarization and sentiment analysis. This also concerns KGs, where DL models are used for embedding the triplet information and used for tasks such as link predictions and KG completion [Lin et al., 2019; Yao et al., 2019], while other researchers worked on the training of embedding from KGs [Ji et al., 2015; Wang et al., 2014; Lin et al., 2015; Bordes et al., 2013].
None of these application was concerned with literary text. Despite some attempts to apply ML and DL models in the field [Worsham and Kalita, 2018; Short, 2019; Labatut and Bost, 2019; K and Antonucci, 2019; Volpetti et al., 2020] to analyze character relations, sentiments and visualizations, a connection with KGs still remains under-explored. The goal of this paper is to fill this gap by providing an unsupervised alternative to the relational classifiers recently developed for supervised tasks in [K et al., 2020]. 3
Figure 1 depicts the workflow of the approach we propose for the unsupervised identification of similar relations in the KGs obtained from literary text. This involves a NLP part for preprocessing (entity recognition, sentence tokenization, detection of relational sentences and triplet generation) corresponding to the red blocks, a DL abstraction level (sentence embedding and summarization) corresponding to the blue blocks, as well as classical ML techniques (characters de-aliasing, sentence clustering, semi-supervised extension) associated with green blocks. These steps in their sequential order are described here below together with the main challenges they present. The tool is available as a free software.2 Named Entity Recognition (NER). The very first step is the identification of the entities to be associated with the KG nodes. These are detected by a custom version of the Stanford NER Tagger3 such that consecutive entities in a sentence (i.e. words tagged as PERSON), are detected as a unique element (e.g., Harry James Potter).
Dealiasing. As a same character can be termed with different aliases in the same novel, a de-aliasing might be required. This issue has been already addressed in [K et al., 2020], where a satisfactory solution based on ML and NLP has been found. Here we adopt a similar strategy based on the classical DBSCAN clustering (✏ = .3 and Levenshtein string distances), together with a number of manual adjustments. In our approach in fact, we first perform separate pre-clustering over entities starting with the same letter (e.g., Hermione and Hermione Granger are identified as a cluster while Harry, Harry Potter and H. Potter as another one), and then adding similar but unassigned names to a cluster (e.g., Granger assigned to Hermione’s cluster and Potter to Harry). All the occurrences of the aliases in the same cluster are finally replaced by identifiers (e.g., CHAR0 replaces Harry, Potter, Harry Potter and so on).
Tokenization. Embeddings based on tranformers are based on contextual information. Since, sentences are considered as the simplest logical and meaningful unit that provides a semantic intuition of the context, we rely on a segmentation at this level.
Relational Sentence Identification. Let us call relational a sentence including two or more characters. We extract relational sentences from the de-aliased and tokenized text, by also evaluating whether or not the text between the two character occurrences is a simple proposition or not (e.g., Harry and Ron were having good time and Harry looked at Ron). If this is the case we call the relation symmetric and we generate two distinct input for the pipeline. Note also we only use sentences containing exactly two characters and excluding self-relations (e.g., Harry, I am Harry Potter). Sentence Embedding. To identify the relations between entities, we embed the relational sentences using Sentence BERT (SBERT) [Reimers and Gurevych, 2019]. SBERT uses a Siamese network structure [Schroff et al., 2015] to reproduce meaningful encodings. The method was specifically modelled for clustering and semantic search. SBERT adds a pooling operation on top of BERT to derive these embeddings. SBERT is fine tuned on SNLI [Bowman et al., 2015] and MNLI [Williams et al., 2018] datasets with a three-way soft-max classifier objective function for one epoch with the default pooling strategy MEAN (computing the mean of all output vectors).
Sentence Clustering. Since, these embeddings encode semantic and contextual information, sentences with similar vector representations are supposed to share similar relations. Hence, we adopt a simple clustering approach to group the sentences with similar relations. The distances between the vectors returned by SBERT are assumed to reflect the semantic similarity between the corresponding sentences and hence the relations included in these sentences. Classical clustering methods such as k-means or DBSCAN can be therefore used to create groups of sentences and hence triplets with the same relation. We considered the Euclidean distance, as well as the classical cosine distance. Even though clustering the entire sentence may not explicitly cluster the relationships, the sentences that fall into similar semantic spaces can provide us a coarse-grained grouping of relations.
CHAR0 and the Philosopher’s Stone... Cluster Summarization. After the clustering of the relational sentences, we might want to represent these relations as a summary of the sentences involved in the cluster. To achieve that we adopt the BERT summarization pipeline based on the BART [Lewis et al., 2019] model. This includes an encoder like BERT and a decoder like GPT [Radford et al., 2019] and it is trained on CNN/Daily Mail dataset with learning rate 3 · 10 5 (Adam optimizer). This performs extractive summarization, giving most suitable representative sentences of each cluster. Although training is not in-domain, as news articles are also narratives, we use this for a preliminary set-up. Triplet Generation. Once the extractive summary is produced, for asymmetric relations we extract the phrase which comes between the two reference characters. We then extract only the verbs from these phrases, which are the considered as part-of-speech tags that could convey some information about the type of relations.
From Unsupervised to Semi-Supervised Learning. The overall procedure described in this section is purely unsupervised. Yet, the clusters of relational sentences are described by the summaries, first, and then labels generated by the system. This might be the basis of a system where, part of those clusters are manually inspected and their summaries/labels validated or fixed by a human annotator. This would turn the system into a semi-supervised one, where the annotated clusters can be used as classifier of the relations.
Book HP HP LW HP
Sentence Dumbledore smiled at the look of amazement on Henry’s face Ron grinned at Henry Brooke smiling at Meg as if everything had become possible him now Henry stared as Dumbledore sidled back into the picture . . . gave him a small smile For a first empirical validation of our pipeline we process, in a single run, two novels, namely Harry Potter and the Philosopher’s Stone (HP) by J. K. Rowling and Little Women (LW) by Louisa May Alcott. 1307 suitable sentences out of 32365 are identified and grouped in 200 clusters (i.e. different relations types). As the characters of the two books are distinct, the system generates a KG with two disconnected components (see Figure 2). Yet, the relation clustering is able to detect similarities between sentences in the two books. E.g., sentences in Table 1 are related to smiling actions. For that cluster the extractive summarization returns the first sentence as a summary and, finally, the triplet generation mechanism return smile as representative label.
Concerning sentence clustering we considered both DBSCAN and k-means algorithms both paired with Euclidean and cosine distance. In the considered setup we did not found significant differences with the two metrics. Regarding the algorithms, an observed issue with DBSCAN was a sudden transition from a huge number of single-sentence clusters to very large clusters. Both these extreme scenarios prevent a meaningful identification of relations. Yet, it was not possible to automatically decide the number of clusters with k-means, as the silhouette analysis returned monotone results. Dumbledore
Harry Amy Hagar
Ron
Snape
Jo Laurence
Meg Hermione
Beth Say Smile Look
Others An unsupervised approach to KG extraction from narrative texts has been proposed. The procedure exploits transformer models to detect similar relations in the triplets, then generates summaries and representative labels for these clusters of similar relations. This represent a pre-processing step for a semi-supervised approach where the representative labels are validated by human annotators and used as a relational classifier. Validated clusters can define relational classifiers, while the automatically generated labels are used for the others. As a future work we want to apply our pipeline to a corpus of literary texts and validate the clusters. This being a starting point for the creation of a knowledge base for literary texts.