In this paper, we describe our submission to the Shared Task on Scene Segmentation (STSS). The shared task requires participants to segment novels into coherent segments, called scenes. We approach this as a sequential sentence classification task and offer a BERTbased solution with a weighted cross-entropy loss. According to the results, the proposed approach performs relatively well on the task as our model ranks first and second, in official in-domain and out-domain evaluations, respectively. However, the overall low performances (0.37 F1-score) suggest that there is still much room for improvement.
Scene segmentation is a novel task introduced in
1Interested readers are referred to annotation guidelines available at https://zenodo.org/record/4457177 for further details.
Our main interest in the current paper is to
explore whether scene segmentation can be handled
as a sequential sentence classification task. To this
end, we follow the methodology proposed in
Cohan et al. (2019), which encodes all sentences in
a sequence jointly through BERT
According to the official results, our model achieves the best performance on the in-domain texts, significantly outperforming the secondranking system. However, the performance drops when evaluated on out-of-domain novels, suggesting that the proposed methodology only poorly generalizes over different domains. We release our system to facilitate reproducibility and future work.2 2 2.1
The scene segmentation task can be framed in
several ways. Within the shared task, it is defined as
the identification of the boundaries that delimit the
consecutive segments
The participating teams are evaluated only according to their success at finding and labeling 2https://github.com/MurathanKurfali/scene segmentation 3Unlike Scenes, Nonscenes, naturally, are not distinguished from one another; hence, Nonscene-Nonscene is not a valid transition. these boundaries. That is to say, classification of an individual sentence as belonging to a Scene or a Nonscene means very little in the evaluations. Of the possible three transitions, Scene-Scene is the most common one as Nonscenes are significantly less frequent in data (see Table 1). 2.2
We model scene segmentation as a Sequence Sentence Classification (SSC) task where the goal is to understand whether a given sentence is segmentinitial or not along with the type of segment it belongs to. Similarly to the more common token classification tasks, e.g. POS-tagging or NER, we employ the IOB2 format and assign a tag to each sentence. Specifically, we label segment-initial sentences (boundaries) as #X-B and other sentences as merely #X where X indicates the type of segment (Scene or Nonscene).
Our classifier closely follows the methodology proposed in Cohan et al. (2019). Here, the authors employed BERT to perform several document-level classification tasks, e.g.abstract sentence classification, where the aim is to classify sentences in a scientific abstract into their rhetorical roles such as introduction, method, etc. The rest of this section describes the model along with our modifications.
The proposed methodology follows the standard way of using BERT through fine-tuning on the target task but uses a novel input representation. The classifier used in the experiments is illustrated in Figure 1. As input, a sequence of N sentences is concatenated by BERT’s special delimiter token [SEP], yielding one long sequence. This sequence, after the insertion of the standard [CLS] token at the beginning, is fed into BERT. However, unlike the standard way of using the [CLS] token as the representation of the input sequence, the representations of the individual [SEP] tokens are used as the representations of the sentences that precede them. Hence, instead of the [CLS] token, [SEP] representations are classified by a multi-layer feedforward network to reach labels.
The rationale for using [SEP] as sentence
representation has to do with the next-sentence objective
of BERT: “Intuitively, through BERT’s pretraining,
the [SEP] tokens learn sentence structure and
relations between continuous sentences”
Loss function The model is trained to minimize
the cross-entropy loss between the probabilities
over the possible labels computed using a
softmax activation and the target distribution.
However, during the initial experiments, we observed
that the model severely suffered from the highly
skewed label distribution, namely the low
number of boundary sentences in comparison to
nonboundary ones.4 In order to mitigate this issue,
following the previous studies
Pi freq(i) freq(c) where freq indicates the count of a certain class. Overall, the weighted cross-entropy becomes LossW CE = − PcC wctclog(sc) where wc is the weight, tc is the gold truth value (taking either 0 or 1), and sc is the corresponding Softmax probability of the class c. 3 3.1
The dataset used in this shared task is based on
an expanded version of the annotation effort
introduced in
The official evaluation is performed on two different test sets: i. Test suite 1 focuses on in-domain evaluation and consists of 5 annotated dime novels, ii. Test suite 2 focuses on out-of-domain evaluation and consists of 2 annotated contemporary high-literature texts.
Table 2 presents the breakdown of our results into each possible transition whereas the official ranking of the participating systems, according to 5Files 9783740941093 and 9783732522033 are used the dev and test set, respectively.
6https://github.com/allenai/sequential sentence classification 7https://huggingface.co/deepset/gbert-large the mean micro-averaged F1 scores, is provided in Table 3. According to the official rankings, the proposed approach is good at segmenting in-domain novels and outperforms the second best system by some margin. However, the performance significantly drops when evaluated on out-of-domain novels, suggesting that the system generalizes poorly across domains.
According to Table 2, our model is best at recognizing Scene to Scene transitions; however, it is almost completely incapable of finding the borders between non-scenes and scenes. Suggested by the high-recall, low-precision scores, our model tends to over-segment the novels. On average, the system divides the in-domain novels into 1.76 and outof-domain novels into 1.61 times more segments. This tendency towards over-segmenting hints at over-sensitivity to certain markers which is further discussed in the next section.
Overall, the results clearly demonstrate that the
task is extremely challenging even in the in-domain
setting. The poor performance of solutions based
on contextual embeddings8 highlight the need for
novel architectures. One obvious drawback with
BERT-based models is their inability to encode
long sequences. Hence, a straightforward extension
of the current model would be to employ a model
which supports longer contexts, e.g. Longformer
In addition to the official evaluation, we performed
a manual error analysis of our model’s predictions
on the in-house test set (see Section 3.1). One
observation was that in certain cases, although the
model correctly recognized the type of transition
(e.g. Scene-Scene), it misplaced the boundary only
8A BERT-based baseline in the original resource paper
similarly fails on this task
Out-of-domain by a single sentence. An instance of this can be seen in Example (1), where the predicted boundary appears just before sentence (1a), whereas the gold boundary appears just before the subsequent sentence (1b):9 (1) a. Und bald darauf fuhr der Wagen aus dem Wald und einen allma¨hlich ansteigenden Berg hinan. (And soon afterwards the car drove out of the forest and up a gradually rising mountain.) b. Dort oben lag das Schloss Treuenfels.
(Treuenfels Castle was up there.)
Furthermore, as mentioned in the previous section, the system tends to over-segment the novels. A manual inspection of false positives (sentences that are erroneously identified as segment boundaries) reveals that despite being incorrect, these predictions are not completely random. Most of the false positives involve an adverbial or other kind of phrase which signals a shift in time and/or place. Some cherry-picked examples are given in Examples 2–5:10 (2) (3) (4) (5)
Als das Gefa¨hrt das Bergplateau erreicht hatte, ließ der Fahrer einige Male laut die Hupe erto¨nen. (When the vehicle had reached the mountain plateau, the driver sounded the horn a few times.) Eines Abends, als Graf Harro von einer Herrengesellschaft zeitiger nach Hause kam, als man erwartete, fand er seine Gattin in einer sehr za¨rtlichen Stellung mit dem jungen Prinzen. (One evening, when Count Harro came home earlier than expected from a gentlemen’s company, he found his wife in a very affectionate position with the young prince.) Und am na¨chsten Morgen fand man die Gra¨fin Alice tot auf ihrem Lager. (And the next morning the Countess Alice was found dead in her bed.) Er wandte sich um und ging wieder zuru¨ck, bis in das Zimmer, wo der Schreibtisch der Gra¨fin Alice stand. (He turned and went back to the room where Countess Alice’s desk was.).
Similar to the behavior of the baseline system proposed in Zehe et al. (2021a), these examples highlight the model’s sensitivity to the local cues rather than the larger context. That is, to a certain extent, the system makes its predictions according to the individual phrases that signal shifts in time or place, paying too little attention to the global context.
9The English translations have been produced by Google Translate.
10In these examples, the system has predicted the shift immediately before the sentences displayed. The current paper summarizes our submission to the Shared Task on Scene Segmentation (STSS). We handle scene segmentation as a sequential sentence classification task and offer a BERT-based solution. The proposed model achieves the best performance in the in-domain evaluations but falls short of transferring its performance across domains. Error analysis further reveals that the predictions are more sensitive to local cues rather than the global structure of the text, highlighting the need for better document-level modeling.
This work has been partly funded by an infrastructure grant from the Swedish Research Council (SWE-CLARIN, 2019–24; contract no. 201700626). We thank the Swedish National Infrastructure for Computing (SNIC) for providing computational resources under Project 2020/33-26.