We present recognizers for four very different types of speech, thought and writing representation (STWR) for German texts. The implementation is based on deep learning with two different customized contextual embeddings, namely FLAIR embeddings and BERT embeddings. This paper gives an evaluation of our recognizers with a particular focus on the differences in performance we observed between those two embeddings. FLAIR performed best for direct STWR (F1=0.85), BERT for indirect (F1=0.76) and free indirect (F1=0.59) STWR. For reported STWR, the comparison was inconclusive, but BERT gave the best average results and best individual model (F1=0.60). Our best recognizers, our customized language embeddings and most of our test and training data are freely available and can be found via www.redewiedergabe.de or at github.com/redewiedergabe.
Speech, thought and writing representation
(STWR) is an interesting phenomenon both from
a narratological and a linguistic point of view.
The manner in which a character’s voice is
incorporated into the narrative is strongly linked to
narrative techniques as well as to the construction
of the narrative world and is therefore a standard
topic in narratology (e.g. McHale (2014);
To conduct either narratological or linguistic studies on STWR based on big data, being able to automatically detect different types of STWR would be of great benefit. This was our motivation to develop recognizers for the following four forms of STWR, which have been distinguished in literary and linguistic theory.1
Direct STWR is a quotation of a character’s speech, thought or writing. It is frequently – though not always – enclosed by quotation marks and/or introduced by a framing clause.
(Then he said: “I’m hungry.”) Free indirect STWR, also known as “erlebte Rede” in German, is mainly used in literary texts to represent a character’s thoughts while still maintaining characteristics of the narrator’s voice (e.g. past tense and third person pronouns).
Indirect STWR is a paraphrase of the character’s speech, thought or writing, composed of a framing clause (not counted as part of the STWR) with a dependent subordinate clause (often using subjunctive mode) or an infinitive phrase.
Er fragte, wo das Essen sei. (He asked where the food was.) 1The stretch of STWR is printed in italics in the following examples.
Reported STWR is defined as a mention of a speech, thought or writing act that may or may not specify the topic and does not take the form of indirect STWR.
In the following, we will describe our approach in developing recognizers for these four STWR types and evaluate our results with a particular focus on the differences between the two contextual embeddings that proved most successful for our task, BERT and FLAIR. 2
Automatic STWR recognition focuses mainly on the forms direct STWR (e.g. Schricker et al. (2019); Jannidis et al. (2018); Tu et al. (2019); Brunner (2015); Brooke et al. (2015) for German texts; Scho¨ch et al. (2016) for French texts) and indirect STWR (e.g. Schricker et al. (2019); Brunner (2015) for German texts; Lazaridou et al. (2017); Scheible et al. (2016) for English texts; Freitas et al. (2016) for Portugese texts). For free indirect and reported STWR, recognizers were implemented by Brunner (2015) and Schricker et al. (2019), the latter builds upon work by the former. In addition to that, Papay and Pado´ (2019) propose a corpus-agnostic neural model for quotation detection.
Since we developed recognizers for German, we will only take a closer look at recognizers trained and tested on German texts. Jannidis et al. (2018) developed a deep-learning based recognizer for direct speech in German novels, which works without quotation marks and achieves an accuracy of 0.84 in sentence-wise evaluation. The algorithm by Brooke et al. (2015) is a simple rulebased algorithm, which matches quotation marks. They do not report any scores. Like Jannidis et al. (2018), Tu et al. (2019) focus on developing a recognizer for direct speech which works without quotation marks, but they used a rule-based approach, achieving a sentence level accuracy between 80.5 to 85.4% for fictional and 60.8% for non-fictional data. Brunner (2015) uses a corpus of 13 short German narratives. For each type of STWR, she implements a rule-based model and trains a RandomForest model, evaluated in tenfold cross validation. The best F1 scores, evaluated on sentence level, were achieved by the rule-based approach for indirect (F1=0.71) and reported (F1=0.57) STWR and by the RandomForest model for direct (F1=0.87) and free indirect (F1=0.40) STWR. Schricker et al. (2019) use the same corpus, but split the data into a stratified training and test set. They use different features than Brunner and train three different machine learning algorithms, RandomForest, Support Vector Machine and Multilayer Perceptron. RandomForest was most successful and gave sentencewise F1 scores of 0.95 for direct, 0.79 for indirect, 0.70 for free indirect and 0.49 for reported STWR. Papay and Pado´ (2019) test their corpus-agnostic quotation detection model on Brunner’s corpus and approximate her RandomForest results.
Our recognizers fill a need regarding the recognition of STWR in German texts, as they deal with all four forms of STWR and are, at the same time, trained and tested on a much larger data base than Brunner’s corpus, making our results much more reliable. In addition to that, our data not only comprises fictional, but also non-fictional texts. An earlier version of our recognizer for free indirect STWR was discussed in Brunner et al. (2019). We improved on this version by adding more training data and achieving higher scores with the BERT based model. 2.2
As the testing of different language embeddings
was a central component in the development of our
recognizers, we will briefly outline characteristics
and research concerning the two most successful
ones that will be in focus in the rest of this
paper: FLAIR embeddings
Both have in common that they produce
context-dependent embeddings as opposed to
static word embeddings, such as fastText
FLAIR produces character-level embeddings. A sentence is passed to the algorithm as a sequence of characters and the task is to predict the next character based on the previous characters. When using FLAIR embeddings, it is recommended to combine two independently trained models: i) a forward model and ii) a backward model. The forward model reads every character in a sentence from left-to-right, the backward model from the opposite direction. This character-based embedding architecture gives advantages in capturing morphological and semantic structures (cf. Akbik et al. (2018)).
While FLAIR embeddings only know their previous context, BERT embeddings know the previous as well as succeeding context at the same time. The training of BERT embeddings consists of two tasks: 1) given a sequence of tokens, where 15% of tokens are masked by the masked language model, predict the masked token based on its context, 2) given two pair of sentences, predict if the second sentence is the subsequent sentence of the first one. The advantage of this model is that it learns associations between tokens as well as sentences. This is important for token-level tasks, such as question answering (cf. Devlin et al. (2019)).
There are no systematic analyses comparing
the performance of embeddings for a sequence
labeling task in German, like we will do in
in this paper. However, there is work
focusing on the use of different embeddings in NLP
tasks in English texts: e.g. Wiedemann et al.
(2019) compare BERT, ELMo and FLAIR
embeddings in a word sense disambiguation task,
where BERT performed best. Sharma and Daniel
(2019) compare BioBERT
We defined the recognition of STWR as a sequence labeling task on token level. For each of the four types of STWR, a separate model was trained on binary labels (“token is part of this type of STWR: yes/no”). The input data consists of chunks of up to 100 tokens, which may span several sentences. The chunks may never cross borders between different texts or cut sentences (except when a sentence exceeds 100 tokens) and can therefore also be shorter than the maximum.
To train our tagging model, we used the
SequenceTagger class of the FLAIR framework
We tested many different configurations for the language embeddings in this setup. Initial tests were done with just fastText embeddings. The results were much worse than the two configurations that became our main focus: a) a fastText model stacked with a FLAIR forwards and a FLAIR backwards model (as recommended in Akbik et al. (2018)) and b) a BERT model.
Except for free indirect, all of our recognizers
were trained and tested on historical German.
Using out-of-the-box embeddings, which are trained
on modern texts like German Wikipedia dump,
Open legal data dump, Open subtitles or the EU
bookshop corpus
Skip-Gram fastText models: We used the default setting as recommend by the fastText tutorial, i.e. we trained for five epochs, set the learning rate to 0.05, adjusted the minimum of the character ngram-size to 3 and the maximum to 6. We varied the model dimensions as well as the training material: fastTextTrain clean is a smaller, cleaner corpus, fastTextTrain contains additional material with OCR errors (cf. section 4.1). On each training set, one model with 300 and one model with 500 dimensions was trained.
FLAIR: A forward and a backward FLAIR embedding with a hidden size of 1024 were trained. All settings were chosen according to the recommendation of the FLAIR tutorial, i.e. the sequence length was set to 250, the mini-batch size to 100, the learning rate to 0.20, the annealing factor to 0.4 and the patience value to 25. The model stopped training after 10 epochs due to low loss.
BERT: We used the PyTorch script finetune on pregenerated.py to fine-tune the pretrained bert-base-german-cased-model with the recommended default configuration: epochs: 3, gradient accumulation steps: 1, train batch size: 32, learning rate: 0.00003 and max seq len: 128. 4 4.1
For the training/fine-tuning of the embeddings 9,577 fictional and non-fictional German texts from the 19th and early 20th century were selected.
For the fine-tuning of the BERT embeddings, we fed all data – split into sentences – into the script pregenerate training data.py from PyTorch, which transforms it to BERT embedding compatible input data. The BERT fine-tuning tutorial recommends to create an epoch of input data for each training epoch, so BERT will not be trained on the same random splits in each epoch. We fine-tuned BERT for 3 epochs, so we generated 3 epochs of data.
For the FLAIR embeddings, 70%, i.e. 4,508,960 sentences, of the 6,441,372 sentences from the data were randomly drawn to form the training corpus. For the validation corpus 15%, i.e. 966,206 sentences, were randomly drawn. The rest was used for testing purposes.
For training the fastText embedding, we used two different inputs. FastTextTrain contained all of the 137,093,995 tokens of our data. From fastTextTrain clean we removed all texts that were recognized with OCR and thus contained typical OCR errors. This resulted in a smaller input set of 131,360,863 tokens. 4.2
The recognizers for direct, indirect and reported STWR were trained on historical German texts – excerpts as well as full texts – that were published from the middle of the 19th to the early 20th century. It comprises fiction as well as nonfiction (newspaper and journal articles) in near equal proportion; fiction is somewhat more dominant. Roughly half of the data was manually labeled by two human annotators independently of one another. Then a third person compared the annotations, adjudicated discrepancies and created a consensus annotation. The rest was labeled by a single annotator.
For indirect STWR, the training data was supplemented with 16 additional historical full texts (9 fictional and 7 non-fictional) to increase the number of instances. To speed up the annotation process, these texts were automatically annotated by one of our earlier recognizer models and then manually checked. The annotators looked at the whole texts, so false negatives were corrected as well.
All the historical data is published as corpus
REDEWIEDERGABE
As the historical data contained much too few
instances of free indirect STWR, we had to
create a separate training corpus for this STWR
type. The basis were 150 instances of free
indirect STWR with little to no context, manually
extracted from 20th century novels. In addition to
that, full texts and excerpts from modern popular
crime novels as well as dime novels were
automatically annotated with a basic rule-based
recognizer that used typical surface indicators. Those
annotations were then verified by human
annotators. On this data, we trained an early recognizer
For model training, our data was split into a training corpus (648,338 tokens for direct and re
Tokens 212,467 49,222 66,817 236,011
Training corpus
Percent Instances 32.77 6,293 7.03 3,505 10.31 7,522 6.30 6,887 ported; 700,202 tokens for indirect; 3,804,226 tokens for free indirect) and a validation corpus (97,316 tokens for direct, reported and indirect; 181.942 tokens for free indirect). Table 1 shows the occurrences of each form of STWR in its training and validation corpus, given in tokens, percentage of tokens in its corpus and instances2. Our test data for the direct, indirect and reported STWR recognizers has 97,863 tokens and comprises excerpts from historical fictional and nonfictional texts in equal proportions. They were labeled with a consensus annotation as described in section 4.2. The test data for the free indirect STWR recognizer has 22,935 tokens and comprises 22 excerpts from dime novels, which were manually labeled by one human annotator. Table 1 shows the occurrences of each form of STWR in its test corpus, given in tokens, percentage of test corpus tokens and instances. 5
We report the scores of our most successful language embedding configurations, i.e. fine-tuned BERT and fastText stacked with FLAIR forwards and backwards.
Notably, our fine-tuned BERT model performed better than the regular BERT model for all STWR types, even free indirect, where the STWR recognizers were tested on modern German fiction (as opposed to historical German fiction and nonfiction for the other models). The same is true for the custom-trained fastText + FLAIR models 2An instance is defined here as an uninterrupted sequence of tokens annotated as the same type of STWR, which can be longer than one sentence. This is of course a simplification, as two conceptually separate stretches, such as lines of dialogue by two different people, will be counted as one instance if they follow directly after each other, but can serve as a rough guideline. On average, a direct instance is 46 tokens long, an indirect instance 15 tokens, a reported instance 10 tokens and a free indirect instance 34 tokens. which outperformed models pretrainend on modern German for all STWR types as well.
We speculate that this is because the customization made the models better suited for literary texts in general, even though it was done on historical German.
The most successful configuration of fastText + FLAIR varies slightly between the different forms of STWR with respect to the fastText model that gave the best results. The fastText specifications for the four types of STWR are detailed in table 2.
Direct Indirect Reported Free ind
We trained each model with the same configuration for three times to correct for random variation in the deep learning results. Table 3 reports the average value of each score and the standard deviation, calculated on token level.
On average, the recognizers using BERT embeddings scored better for all types of STWR except direct, for which the recognizers of the stacked fastText and FLAIR embeddings proved consistently more successful. Most striking was BERT’s advantage for free indirect, where especially the recall improved. It should be noted though, that the FLAIR-based freeIndirect model consistently gave better precision.
However, when looking at the standard deviation over the three runs and the range of results, we see that the F1 score ranges of the FLAIR and BERT recognizers overlap for reported, so the results of the comparison are not conclusive for this STWR type. For the other three STWR types, the F1 score ranges are clearly distinct, even though the free indirect models show a high variance.
Table 4 lists the scores for the individual recognizers from the three training runs that produced (0.017) (0.0236) (0.0163) (0.017) (0.0082) (0.017) (0.034) (0.0309) the best results.
To give an impression how difficult it is for humans to annotate these forms, table 5 presents the agreement scores between human annotators. The scores for direct, indirect and reported STWR are based on corpus REDEWIEDERGABE, the corpus of fictional and non-fictional historical texts our test data was drawn from. The score for free indirect was calculated directly on the free indirect test corpus.
We performed two types of error analysis: First, we looked at the first 10,000 tokens of our test data and categorized the types of errors made by our top recognizers (cf. table 4). This gives an impression of the types of challenges the four forms of STWR pose and how well our recognizers can deal with them which is important practical information for anyone using them.
Second, we also looked at the first 20 differences between the results of the best models trained with BERT vs. the best models trained with FLAIR. The goal was to find indicators which specific properties of the two different contextual embeddings made them better or worse suited to a particular task.
As the four types of STWR have very different characteristics, we will discuss each of them separately. Direct STWR has two main characteristics: First, being a quotation of the character’s voice, it tends to use first and second person pronouns and present tense. Second, it is often marked with quotation marks, but the reliability of this particular indicator varies dramatically between different texts. Its instances can also be very long, spanning multiple sentences. We observed that about half of the false positives as well as the false negatives are partial matches, i.e. the recognizer did correctly identify a stretch of direct STWR, but either broke of too early or extended it too far.
A main cause for false negatives were missing quotation marks, i.e. unmarked stretches of direct STWR, especially if those occured in first person narration. In these cases, the recognizer is missing its two most reliable indicators to distinguish direct STWR from narrator text at the same time. Another source of false negatives are very long stretches of direct STWR, such as embedded narratives. The recognizer looses the wider context and tends to treat this STWR as narrator text, especially if it contains nested direct STWR and exhibits characteristics such as third person pronouns and past tense.
The main source of false positives is also related to narrative perspective: In a first person narration or a letter, the recognizer tends to annotate narrator text as direct STWR – the reverse problem to the one described above. Note that these cases are very hard for human annotators as well and can only be solved by knowing a wide context. The recognizer knows a context of 100 tokens maximum and we observed that wrong decisions often occur at the beginning of a context chunk and are then propagated to its end. Another source of false positives are – predictably – stretches of texts in quotation marks that are not direct STWR, though these are a relatively rare occurance. We also observed mix-ups with the forms indirect and free indirect STWR, especially if unsual punctuation was used, though this was rare as well.
In summary, we can say that for direct STWR narrative perspective is a major factor. The test material was deliberately designed to contain texts written both in first and third person perspective. If evaluated separately, we could observe a significantly better performance for third person perspective (see table 6).3
First person Third person
Direct STWR is the only type of STWR where FLAIR embeddings performed better than BERT with a clear advantage. Looking at the first 20 differences between the recognizers, we found that BERT is more prone to annotate letters and first person perspective narratives as direct STWR. It also breaks off prematurely more often, indicating that FLAIR seems to be better in maintaining the context of the annotation. On the other hand, FLAIR tends to make more minor mistakes, such as not annotating a dash when it is used instead of a quotation mark to introduce direct STWR. This points to the more character-based behaviour of FLAIR which – in general – seems to serve well for direct STWR, maybe because of the prevalence of typographical indicators. The wider context of the BERT embeddings does not seem to help with the perspective problem, but instead introduces additional errors. 5.2
Indirect representation in our definition takes the form of a subordinate clause or an infinitive phrase, dependent on a framing clause which is not part of the STWR itself. Thus, instances of indirect STWR are always shorter than one sentence. Of the four STWR forms, it is the one that is most strongly defined by its syntactical form.
One difficulty are cases where the indirect STWR contains subclauses or, conversely, is followed by a subclause that is not part of the instance. In these structures, the recognizer tends to 3We experimented with training two specialized direct models, one only using texts with first person perspective and one only texts with third person perspective as training material, and evaluated them on the matching types of texts. Unfortunately the performance was worse than that of the model trained on the complete training corpus, probably because of the significant reduction of training material. have trouble identifying the correct borders of the STWR. When looking at the error analysis, about one third of the errors for both false positives and false negatives are partial matches, mostly caused by this problem.
The biggest cause for errors are cases where the typical indirect structure – a subclause starting with dass, ob (that, whether) or an interrogative prounoun – is paired with an unusual frame. This leads to false positives, if the frame contains words that usually indicate STWR, such as es scheint außer Frage, dass ... (it seems out of the question that ...). Though this phrase does not introduce STWR, the word Frage (question) still triggers an annotation. On the flipside, cases of indirect STWR tend to be missed if they are introduced by phrases that have an unsual structure and don’t contain words that are strongly associated with speech, thought or writing acts. We also observed that unusual punctuation, such as dashes, multiple dots and colon (used instead of comma at the border of an indirect STWR), have negative effects on recognition accuracy.
In a comparison between the indirect models using BERT and FLAIR embeddings, we observed that both models make errors of the types described above, though at different places. However, overall FLAIR seems more susceptible to interference in the form of unusual punctuation or framing phrases that are interjected in the middle of a stretch of indirect STWR. It is also less successful than BERT in recognizing STWR instances that are introduced with nouns instead of verbs. 5.3
Reported STWR is a fairly difficult form even for human annotators, mainly because it is so similar to pure narration that it can be hard to distinguish. It should be noted that the gold standard annotation in this case contains a number of uncertain instances that could be debatable for humans as well. Reported instances tend to be rather short, varying from one token to one sentence at most, and syntactically diverse. The most reliable indicators are words referring to speech, thought and writing acts.
Only about a fifth of the false negatives and false positives observed for reported STWR were partial matches, a significantly lower percentage than for direct and indirect STWR. This indicates that for this form, finding the correct borders of the annotation is less of a problem than deciding whether STWR is present at all.
Most errors can be attributed to problems related to speech, thought or writing words, the main indicator for reported STWR. Such words can trigger a false annotation and are the main cause of false positives. The reverse problem is even more prominent: Instances that do not use lexical material commonly associated with speech, thought and writing tend to be overlooked. Missing such unusual instances is the main problem of the recognizer and though the direct and indirect recognizers also have better precision than recall scores, the difference for reported is clearly more pronounced. Another recurring error type is that the borders of the STWR were not detected correctly, missing modifiers or annotating part of the surrounding narration. We also observed some rare mixups with indirect STWR.
As noted above, the F1 score ranges of the FLAIR and BERT based recognizers are not distinct for reported, though BERT does perform somewhat better on average. Looking at the differences, we found that the recognizers make the same types of mistakes, but BERT is generally more open to unusal instances of reported STWR, leading to a better recall, which is the main reason for its better overall performance. 5.4
Free indirect STWR is structurally similar to direct STWR in that it usually spans one or more consecutive sentences. It is very hard to identify using surface markers, as it is basically a shift to a characters internal thoughts, but still uses the same tense and pronouns as the surrounding narration. The best indicators are emphatic punctation such as ?, !, -, words indicating a reference point in the present (such as now, here) and characteristics of informal speech such as dialect or modal particles.
The free indirect recognizers show the largest gap between precision and recall: nearly 0.4 points. Clearly, the problem here lies in undetected cases. Notably however, over 40% of the false negatives are partial matches, meaning that the recognizer at least correctly detected an instance of free indirect, though it failed to capture it completely.4
4The recall problem might be exacerbated by the false negatives in the training data. We ran tests where we cut out the marked instances in the training data with some context The main cause for false positives are cases in which some of the main indicators of free indirect (as described above) occur in narration. In addition to that, unmarked direct STWR is prone to be labeled as free indirect. As for the false negatives, about half of the missed instances contained at least one surface marker, but many are only recognizable via wider context clues or an understanding of the content.
Comparing BERT and FLAIR again, we find that BERT gives a much better recall – the same effect as with reported STWR, but more pronounced. BERT is clearly better in picking up subtle signals for free indirect STWR than FLAIR. The flip-side of this is that the BERT model also produces more false positives than the FLAIR model. An interesting observation is that it sometimes annotates sentences that are not part of the free indirect STWR itself, but introduce it. Though the borders of the STWR are not detected correctly in these cases, this might indicate that the model learned that these context clues are highly relevant to identify free indirect STWR which is indeed the case. An example for this scenario is the following passage:
Jetzt war er mit dem La¨cheln an der Reihe. Ihre
ungeku¨nstelt und ehrlich. Hoffentlich wu¨rde sie das nie verlieren. (Now it was his turn to smile.
BERT also marks the introductory sentence (underlined) that shifts the focus to the character to introduce the free indirect instance (in italics) that tells us his thoughts. The FLAIR model on the other hand has its strength in precision: The few false positives that it produced are often borderline cases that are attached to free indirect passages and could be read as plausible extensions. 6
We presented recognizers for four types of STWR which differ strongly in structure and difficulty. Our models for direct, indirect and reported were trained and tested on historical German fictional and non-fictional texts, the model for free indirect on modern German fiction. The success rates (25 or 50 tokens) and used this as training input. A detailed evaluation is beyond the scope of this paper, but the resulting recognizers had better recall but worse precision, leading to similar F1 scores. correspond closely to the reliability of humans: For indirect and reported, we even achieved similar scores to the human annotator agreement on a comparable corpus. For the types direct and free indirect, humans still clearly outperform our best models. In both cases, we believe that the need for wide contextual knowledge plays an important role to explain the gap: For direct, the models fail most often in distinguishing between a first person narrator and a character quote. Free indirect in general is a highly context dependent form that requires an understanding of the narrative structure.
We tested a variety of different language embeddings for our task and provided a comparison of the most promising: FLAIR and BERT embeddings. For both, we also trained/fine-tuned models on historical texts. FLAIR gave the best scores for direct, BERT for indirect and free indirect. For reported, the results were not conclusive: Though BERT performed better on average, we observed an overlap in F1 score range of the BERT and FLAIR models over multiple runs.
Most striking was the improvement achieved with BERT for free indirect STWR. In particular, BERT improved recall for the most difficult forms, free indirect and – to a lesser degree – reported, showing a greater ability to detect unusual instances.
Direct STWR was the only form where FLAIR clearly outperformed BERT. It seems like the higher sensitivity of BERT is more of a disadvantage here, as it tended to misclassify even more instances of first person narration than FLAIR.
To further improve performance, one idea is modifying our input strategy: instead of consecutive chunks of up to 100 tokens, overlapping chunks could be used as input. This might prevent the recognizers from loosing context at the beginning of a chunk, which would be especially relevant for the direct and free indirect recognizer.
The top models and customized embeddings described in this paper are freely available via our homepage www.redewiedergabe.de and via GitHub. In detail, our customized BERT embeddings can be found at huggingface.co/ redewiedergabe/bert-base-historical-german-rwcased, the custom-trained FLAIR embeddings are integrated into the FLAIR framework as de-historic-rw-forward and de-historic-rwbackward. The top recognizer models are available at github.com/redewiedergabe/tagger along with the code used for training and execution.
In addition to that, all the material used
for the direct, indirect and reported recognizers
and part of the material used for the free
indirect recognizer5 is available as corpus
REDEWIEDERGABE