In this work, we describe a system for scene segmentation that, relying on character constellations as one of the defining characteristics of scenes, employs a state-of-the-art coreference system. Conceptually building on one of the presented baseline systems, we use a transformer model, enhanced with additional coreference-based features, to identify scene boundaries on the basis of sentence pairs. Finding one of our system's core weaknesses to lie in its local decision making, we adapt an equidistance constraint, avoiding the common error of predicting very short scenes that in many cases only cover a single sentence. We show that coreference is a suitable feature for scene segmentation and experiment with dynamic programming approaches for non-local decisions. This work is a submission for the shared task scene segmentation (STSS) held at KONVENS 2021, where task participants were asked to, given annotated training data, build systems that split novels into scenes: segments narrating a coherent action in one location with the same characters. Our system ranks 4/4 and 4/5 in Track 1 and Track 2, respectively.
One of the most defining characteristics of scenes
are character constellations, in this work we
describe a scene segmentation system exploiting this
characteristic. Other defining aspects of scenes
such as the story and discourse time being equal
and the fact that they contain a coherent sequence
of actions will not be explicitly modeled in this
work. The shared task scene segmentation hosted
by Zehe et al. (2021b) provides training data in
the form of 22 dime novels, with an additional (for
the task duration) unpublished test set and a single
trial document. We chose a transformer-based
approach as a starting point; we use BERT
Pethe et al. (2020) approach the task of chapter segmentation, the task of splitting a document into its chapters. This task is related to scene segmentation in that it operates on a similar domain. As we conjecture, chapter boundaries may also correspond with changes in location or characters, making this work more relevant still. Pethe et al. (2020) take an equidistant approach to chapter segmentation, thereby enhancing local decisions with the knowledge that chapter boundaries tend to be somewhat evenly placed throughout a novel. The equidistant approach is applied by minimizing the following equation: cost(n,k)=mini∈[0,n−1] cost(i,k−1)+(1−α) |nL−i| −α·sn Where k is the number of breaks to be inserted, n the position at which to insert a break and L the target length of each segment. α is a hyperparameter controlling the impact of the local boundary score sn with values approaching one placing more importance on local decisions.
In our previous work
In order to maximize the contextual information
input to BERT, we do not pass an explicit
context in conjunction with the two sentences in
question
While we did experiment with a BERT model
trained on German literary data1, we did not find
success with it which, we attributed to the fact
that it is fine-tuned on named entity recognition
and may have, in a case of catastrophic
forgetting, lost the ability to perform the NSP task.
While the coreference-based features rely on
previous work of ours
Leveraging coreference features we seek to model one of the central components of scenes: the character constellations. To this end, we pass the number of unique characters appearing in each of the input sequences, together with the number of unique characters appearing in both sequences to the model.
Taking a more global approach to coreference
would also be possible, in this case, the number
of characters involved in the current context may
be compared to the global number of characters.
While this approach may yield further
improvements, we did not test it, partly due to the fact that
global coreference resolution for long documents
still is much more susceptible to errors than local
approaches
One feature that we, following manual inspection of the training data, expect to be predictive of scene boundaries are named entities. The explicit mention of characters as well as that of locations should indicate a scene change. We extract the named entity tags for persons, locations, and miscellaneous entities and use document-length-normalized counts of each of them as a model input. While the coreference features capture some similar information, they capture neither location mentions nor are they able to differentiate between explicit and anaphoric character mentions.
Using a NER system trained specifically on
literary data could help this step, such data is available
in the DROC dataset
In an effort to improve our model, we added a set of surface features that we believed may be indicative of scene changes. We passed the number of tokens (including special characters such as quotes and punctuation) fulfilling different properties to our model • being punctuation • being uppercased • being quotation marks • being a stop word • being the start of a sentence
While all these features could, in principle, be picked up by means of representation learning in our neural model, we still add them due to the relatively small number of training samples.
While, in principle, our model is capable of
predicting both scene boundaries and scene types, our
final system uses two distinct models with the same
architecture and inputs for the two tasks. Joint
training presents non-trivial challenges in
balancing the two target objectives but may yield
improvements in final results. Both models were
trained with early stopping on the trial data (i.e.
one document provided with the task description
but not as part of the training data); a
hyperparameter search for individual learning rates for the final
layers (between 1 × 10−3 and 1 × 10−5) and the
BERT model (between 1 × 10−4 and 2 × 10−5)
was performed using the Tree-structured Parzen
Estimator
Using the features described so far we reach an F1-score of 33.7 on the task’s trial document2, presumably already outperforming the baseline system. Figure 1 illustrates the predicted boundaries together with the networks output values for each of the potential scene splits, i.e. each pair of sentences. Notably, there are multiple cases of two or more directly adjacent instances of false positives. Sometimes, like at the very end of the document, in conjunction with a true positive boundary. This illustrates what we see as a key weakness of our initial model; since decisions are purely local, when in doubt about the placement, the model creates multiple boundaries where one would be sufficient.
2Unless otherwise specified F1-score refers to the boundary class’s F1-score throughout this document
As discussed in Section 4 we see an issue in the local nature of scene segmentation boundaries. One approach to remedy this may be, training on sequences of adjacent sentence pairs; this would have the advantage of allowing for non-local decisions, informed by any part of neighboring inputs. At the same time, however, this increases the memory requirements, and with scene boundaries occurring about every 43 sentences on average, a large enough context may (depending on available GPU memory) be infeasible to jointly train. Our early approaches instead focused on using neural sequence models on local decision outputs but using this approach we did not manage to improve upon local-decision-based results.
Instead, we chose a purely algorithmic approach without training: the dynamic programming (DP) approach by Pethe et al. (2020), a technique that requires prior knowledge of the number of chapters, or in our case scene, boundaries. Applying their approach to the task’s trial document which was held-out, given the correct number of scene boundaries, (with α = 0.9) results in an F1-score of 39.1. This represents is an improvement of around 5.4 on the local F1-score of 33.7. For comparison, when only using the k highest confidence values, where k is the number of gold boundaries, we only get an F1-Score of 34.8, illustrating that the mere knowledge of the number of scenes is not as impactful. Figure 2 shows the effect the cost function can have on decisions, while α = 0.7 actually entails a worse F1-Score, the effect is very subtle when using larger α values (i.e. when incorporating local decisions to a larger extent).
Figure 3 illustrates that the coefficient of variation (CV) for the shared task’s scene boundary is much higher than it is for the chapter data in the work by Pethe et al. (2020), where the distribution is centered around a value below 0.5. This can be interpreted as the length of chapters inside most documents being less variable than the length of scenes in many documents in our dataset. Although it is to be noted that the two statistics are made on the basis of very different datasets. The standard deviation of the distribution of average per document scene lengths (in sentences) is 10.84 with a mean of 45.3 and, accordingly, a CV of 0.24.
Another very simple approach to using non-local information is to, in a fixed window, only consider the top value to actually constitute a boundary. For this, we walk across the boundary candidates and, in a fixed-sized window, set the boundary class to zero for all but the largest value in the window. With a window size of five, for example, this means that no candidate with larger confidence values in its four neighbors (two to either side) will be predicted. Using this simple strategy, however, we adversely impact the quality of our predictions, going from an F1-Score of 33.7 to one of 27.8.
The improvements attained by application of the DP technique by Pethe et al. (2020) in combina
600 Sentence Number 800 1000 tion with the variance of 0.74 in the task’s trial document illustrate just how important non-local information is to improving performance in this task. Further work on neural sequence models may yield significant improvements.
Our final model uses the DP approach by Pethe et al. (2020) with α = 0.8, a strong focus on local values. As explicitly stated in their paper, this method assumes knowledge of the actual number of boundaries, which is not the case for our data. We apply the heuristic of assuming the number of actual boundaries to be equal to the number of locally predicted boundaries. This way our the nonlocal approach effectively only moves the positions at which splits happen but does not change their total number. Unsurprisingly, given the variance in scene lengths, we found this to outperform the heuristic of dividing the text length by the average scene length. Further, we adapt the cost function to be more lenient with regard to scenes shorter than the average, as long as they are not too short.
Figure 4 shows how we adapt the equidistant constraint by Pethe et al. (2020) to punish very short distances. Where their cost function is linear in both directions, we adapt it to only punish very Equidistant Cost (Pethe et al., 2020)
max(−log(x + 1) · β1 , x2 ) For this, we apply the cost function in Equation 1 to negative distances relative to the target distance L, β is a hyperparameter controlling how close to a distance of zero very large costs set in; we use β = 2. For positive distances, we use x2 effectively increasing the inherent α but also changing the relation of long distances to short ones.
Evaluating the same technique on our training data yielded a marginal improvement of around 0.01 F1, this is to be expected as some memorization of training samples should lead to improved local decisions. This result does give us confidence the approach will not adversely impact test set performance.
While, after optimizing alpha on the held-out data, the equidistant cost function performed on par with our cost function on the same data, when adapting to the training data (on which our α value was not optimized) the equidistant function only increased performance by 0.003 F1.
Further analysis is needed to provide a clear picture of cost function’s impact on unseen data. It however already seems plausible that our adaptation of the cost function presents an improvement over the equidistant cost function. 6
We present an approach to scene segmentation that relies on character information. While we do not produce irrefutable evidence of its advantages, we propose a cost function more suitable to the needs of scene segmentation, adapting the work by Pethe et al. (2020) to a new task.
On the official evaluation metric we only reach an F1-score of 0.02 for Track 1 and an F1-score of 0.11 for Track 2. These are below the boundary class performance discussed earlier as they include the correct classification of scene types. With out system focusing mostly on the placement of scene boundaries it could potentially be extended with features more suitable for scene classification.
The system performs relatively poorly in Track 1, reaching the last place with quite a margin to the next system, but much better in Track 2 where it is close behind the third-placed system, what exactly causes this difference in performance remains unclear. We stay far behind the performance of the top-scoring systems but coreference seems to be a salient feature that may be useful to include in future systems.