We address the challenge of enriching plain text dramas with predicted TEI/XML elements. We use a large corpus of dramas annotated with TEI information about act/scene changes, speaker changes, and stage directions, among others. On this data, we fine-tune a pre-trained BERT transformer model on several subtasks, like predicting stage directions vs. utterances. We show that the used architecture is able to predict the learned structural elements on unseen data for several settings and models.
ACT 1 Scene 1 Enter Barnardo and Francisco, two sentinels.
BARNARDO Who’s there? FRANCISCO Nay, answer me. Stand and unfold yourself.
BARNARDO Long live the King! FRANCISCO Barnardo.
… More generally, automatically predicting text structure from plain texts ofers new ways to use texts, as layout, text formatting or structure play an important role in many historic text types and uses of them in computational humanities research. It is important to note that we do not see this work as editorial work or a form of interpreting the text by inserting markup. Instead, our goal is to create machine readable markup of structures that are already overtly in the text and can later be used for computational analyses.
While there are many attempts of automatically segmenting texts into smaller and cohesive
units, many of these works focus on recognizing discourse units [
Most recently, McConnaughey, Dai, and Bamman [
Pethe, Kim, and Skiena [
Zehe et al. [
To our knowledge, there is no research on the automatic segmentation of dramas.
We make use of a Bidirectional Encoder Representations from Transformers (BERT) architecture as our base model. BERT is a large neural network architecture, with a number of parameters that can range from 100 million to over 300 million.
It is usually preferable to use a pre-trained BERT model that was trained on a huge dataset. For the following experiments, we use models that were trained on English and German data. While the English models are trained on a diferent language than that of the dramatic texts, it allows us to show how much the model relies on structural compared to semantic information.
Our models are based on the pre-trained BERT model. It consists of 12 layers with a deep selfattention mechanism. We feed the BERT model with the tokenized sentences which contain the input ids and attention mask. The output of the BERT model is passed on to a dropout layer (rate: 10 %) so that we can prevent our model from overfitting. The output of the dropout layer is passed on to a rectified linear activation function to overcome the vanishing gradient problem and allow our model to learn faster and perform better.
After that, two dense layers are used. The first layer has 768 states equal to the number of hidden states of the pre-trained BERT model. The layer is followed by a second dense layer with a softmax activation function.
During fine-tuning, we freeze all the layers of the model, to prevent any updating of model weights.
We are using the German Drama Corpus (GerDraCor), a collection of German-language plays
from 1730 to the 1940s [
We use single sentences as input for the BERT model. This is a compromise, as some tags will only cover single words or incomplete sentences. Using whole sentences as input has advantages though, as it gives more context to the model and it is straightforward to later assign XML tags for sentences or groups of sentences instead of arbitrary sub-sequences.
Some tags will also have multiple lines of text stored inside them. We therefore use the NLTK Sentence Tokenizer to split groups of multiple sentences into single sentences, which can be given as input to our model. The extracted plain text is passed to the tokenizer. The text coming from inside the speaker tags is also considered as a sentence, even though it usually just constitutes a name. Figure 2 shows a part of our final pre-processed dataset which we use for training and testing. While the first column SENTENCE contains all the tokenized sentences, the second column Decider contains numeric class labels for the tokenized sentences.
Overall, the dataset contains 1 410 783 sentences with 10 021 598 tokens and 240 794 types. Since the sentences in the dataset are of varying length, we use padding to make all sequences have the same length. Since the vast majority of sentences has ten or less tokens, we set the maximal sequence length to 10.
We implement a simple baseline to compare the results of the transformer models against. For the baseline, we choose a conditional random fields (CRF) model, which is able to consider sequential information. To make the baseline comparable to the BERT models, we also choose sentences as the input and let the model predict if a sentence belongs to one of five classes: act (0), scene (1), stage direction (3), speaker tag (4) or utterance (5). The CRF receives features extracted from each sentence, namely: • The lower-cased surface string of the sentence. • If the sentence contains the German word ‘Akt’. • If the sentence contains the German words ‘Szene’ or ‘Scene’. • If the sentence begins with an uppercase letter. • If the sentence only contains uppercase letters. • If the sentence contains a digit.
For training, we make use of the limited-memory BFGS algorithm and elastic net regularization.
Training For the BERT models, we use pre-trained models provided by HuggingFace3 to
ifne-tune on. We use the AdamW algorithm [
We consider accuracy, precision, recall and F1-score as metrics.
In this section, we investigate the performance of our proposed model on various tasks. We split the dataset randomly into three sets: train, validation, and test, where the train set is 70% and validation and test set are both 15% of the overall data. We fine-tune the model using the train and validation set, and evaluate on the test set.
Detecting Act Boundaries We extract the data from <stage>, <head> and <speech>. The sentence splitter recognizes 1 208 899 sentences. The goal of the prediction is to mark all sentences: If the sentence is the first sentence of an act it is classified as 1, otherwise as 0. Hence it is a binary classification task. For this task we use ’bert-base-uncased’ (the identifier at HuggingFace) as a base model. In Table 2 we can see the results as classification report and confusion matrix.
We can see that the model is able to predict a non-boundary for nearly 100 % of the cases. Yet, the model is not overfitting on class 0, as the prediction for an act boundary still gets high scores with an F1-score of 0.89. From Table 2b we can see that the model makes more mistakes in wrongly classifiying non-act-boundaries as act changes than the other way around (i.e., the number of false positives is higher than the number of false negatives).
Detecting Stage Directions For this task we use the <stage> and <speech> tags, which contain 1 203 911 sentences. Each sentence is classified as to whether it is (part of a) stage direction or not, using the ’bert-base-uncased’ model. As stage directions are much more frequent and much more similar to character speech, we consider this task to be more difficult than the one discussed above: Instead of relying on lexical cues, it needs to take discourse structure and semantic information into account.
Table 3 shows the results. In this, 0 represents character speech and 1 represents stage direction. The model has indeed more difficulties with correctly predicting stage directions. With a precision of 0.7 and recall of 0.95, the detection nevertheless performs reasonably well. The confusion matrix shows that false positives are much more common than false negatives, which can probably be explained by the imbalance in the training dataset.
238364 (b) Confusion Matrix. GS is the gold standard, SO the system output. 1
238364 (b) Confusion Matrix. GS is the gold standard, SO the system output.
All tasks combined For this task, we extract the data from the all above mentioned tags, which in total contain 1 589 090 sentences. The task now is a 5-way classification, as we classify sentences as being (part of) a stage direction (2), name of a speaker (3), character speech (4) or act (0) or scene boundary (1).
For this task we use diferent types of BERT models and compare them. Table 4 shows results for ’bert-base-uncased’. All results are still comparable to the results of classifying the tags individually. Some of the results are lower, but not by much. This is promising, as it shows that we can potentially predict the complete structure of a plain text drama at once without loosing much in predictive power over classifying the single types of structure individually.
As mentioned earlier, all models so far have been pre-trained on English data. The above evaluation shows that even on German data, they can make good predictions, which can be explained by the fact that most of the distinguishing features needed so far for prediction are structural rather than content-based. However, for the task of predicting all tags together, we now use a model trained on German data and see if the results can be further improved. Table 5 shows the results for applying the ’bert-base-german-uncased’ model. We can see that especially for predicting stage directions, the performance improves significantly by 7 percentage points F1 score. The other results are either identical or slightly higher in the case
238364 (b) Confusion Matrix. GS is the gold standard, SO the system output. 1
238364 (b) Confusion Matrix. GS is the gold standard, SO the system output. of speech with a plus of 2 percentage points. This is absolutely expected, as these two types are more content based. Still, the English model is able to pick up on enough structural cues to also predict well on German data.
Lastly, we check if it makes a diference to use a model that was trained on cased data, as all other models before were trained on uncased data. Here, the ’bert-base-german-cased’ model has been used. The results for this can be found in Table 6, and are sightly lower than in the uncased setting. This suggests that preserving case lets the model generalize less well. Baseline We compare these final results to the baseline system. The results are shown in Table 7. The baseline performs rather well for the tasks of predicting act and scene boundaries and recognizing speaker tags. However, the BERT-based models achieve slightly higher values for all these classes. For the task of character speech identification, it performs worse than the BERT-based models in term of precision, but achieves a higher recall than all other models. For the crucial task of recognizing stage directions, it returns a rather low recall value, but the highest precision value.
In all experiments, we observe that the models achieve precision and recall scores around 95 % to 99 % for most of the categories. For stage directions, the evaluation yields lower scores: The model misclassifies some of the sentences in character speech as stage directions. By experimenting with diferent BERT models we are able to achieve a precision of 77 % for stage directions which means that BERT German Uncased is the most suitable model for these predictions. While the CRF-based model sets a high baseline for the tasks of act, scene and speaker recognition, the BERT-based models outperform the baseline in all measures. Only for the content-based tasks of speech and stage direction recognition, the baseline achieves higher results in recall and precision, respectively. In future work, the transformer models might benefit from combining them with the CRF model.
In this paper, we have shown that the BERT model is a reasonable model for predicting and extracting structural segments from dramatic texts. Based on this finding, we have proposed a novel fine-tuned model based on BERT. From the above results we can conclude that ’BERT_German_Uncased’ is the most efective base model. We can also conclude from the above results that all the models perform quite well, whether we predict the segments with binary classification or in the full model with five classes. We were further able to show that models trained on English data are able to predict the more structural elements of German dramatic texts with high accuracy. However, for the structural elements that rely more on text content, a model trained on German data performs better.
Both recall and precision for all classes except class 2 (stage directions), are quite high which means that the model predicts these classes accurately. The recall for class 2 is 0.93 which means that the model was able to find 93 % of the stage direction sentences. However, precision is a bit lower for class 2, which means that the model misclassifies some of the class 4 sentences (character speech) as stage.
In the future, we plan to extend on the presented work to create a fully automatic mapping tool to convert plain text scans of dramatic texts into properly structured TEI/XML documents. Even if this automatic conversion is likely to contain some errors, correcting it manually is much less labor-intensive than coding the entire play by hand. We plan to add texts currently only available in plain text to the DraCor corpora once the above mentioned tool is developed and functioning. One challenge we will most likely face will be that OCRed texts usually contain mistakes which might throw of the transformer model. Hence we will also experiment with text normalization techniques. This opens a path towards large scale data analysis of plays that currently are not available as part of the DraCor repository. In addition, our analysis has shown that the trained model works reasonably well even if used across language boundaries. This suggests that it is also possible to apply a very similar model on plays from other languages, as training data for many languages is already available.
The first and third author have conducted the described research within the QuaDramA project, funded by the Volkswagen foundation and within the Q:TRACK project, funded by the German Research Foundation (DFG) in the context of SPP 2207 Computational Literary Studies. We thank both for making this possible.