The task of Style Change Detection (SCD) aims at detecting author switches within one document based on their individual writing style. In this notebook, this task is divided into three sub-tasks: detecting multi-authored documents, finding style change positions, and attributing each paragraph to a unique author [1, 2]. We chose different machine learning approaches for the first task of multi-author detection, and the second task of style change detection. Our approach to the third task of SCD-based authorship attribution is a hybrid method building upon the prediction of the style change detection and extended by an attribution algorithm. The data was given by the PAN'21 challenge and is a data set collected from an English written Q&A forum [1, 2]. While the approach to task three showed to be very computationally expensive, we found good results for task one and two with F1-scores of 86% for task one and 78% for task two on the validation set.
Due to the ever increasing interconnection and collaborations resulting in multi-authored texts, identifying authors within texts becomes an interesting effort. The steadily increasing amount of readily available data gathered form online text forums or short message boards allows for construction of methods and models for text analysis based on machine learning approaches. Possible applications for such models can relate to e.g. plagiarism detection or style analysis. Style analysis opens the possibility for authors to adjust their writing styles to that of other collaborators and consequently to write more coherent texts.
The task of Style Change Detection (SCD) aims at analysing texts by detecting whether or
not a document is multi-authored, and if so, where a Style Change occurs in order to find author
changes and predict authorship in the context of one document [
Especially the last task, which is a type of author attribution task, adds additional complexity in
comparison to the previous years. Traditionally in author attribution, the setting contains a fixed
and known set of authors. The data set for this year’s PAN Shared Task represents real-world
data [
The following notebook focuses on our efforts to construct machine learning based methods to solve the three given tasks. After this introduction we will give a brief overview of related works in Section 2, describe our methods in Section 3, state our results and their evaluation in Section 4, and lastly give a short discussion and conclusion in Section 5.
As mentioned before, the goal of the PAN’21 Shared Task is to detect the exact positions of authorship changes. In order to lead up to this task, it is split into three sub-tasks: deciding whether a document is multi-author, finding style changes between paragraphs, and lastly attributing the paragraphs uniquely to one of the authors.
Regarding the first task of detecting multi-authored documents, several different solutions have
been proposed over the last decades [
Similar to the first task, the second task received attention by researchers over the last years.
Especially in the PAN Shared Task series, the detection of style changes between paragraphs
has been addressed repeatedly and various approaches have been proposed. Iyer and Vosoughi
(2020) [
Finally, the third task, namely assigning paragraphs to specific and unknown authors, is a more
recent problem and has consequently been addressed less frequently. It is similar to the problem
of authorship identification of multi-author documents (AIMD) [
In all three tasks, individual writing style plays a role and can be addressed more or less
specifically. Castro-Castro et al. (2020) [
In the next section, we will focus on our contribution to the research in Style Change Detection and Author Attribution, while taking into account past research and results.
For the three tasks tackled in this notebook we chose different machine learning approaches
for the first task of multi-author detection, and the second task of style change detection. We
attempted to solve the first task using per-document embeddings and an MLP, while we utilized
per-paragraph embeggins and textual features as input for an LSTM for the second task. Our
approach for the third task of SCD-based authorship attribution is a hybrid method building upon
the prediction of the style change detection and extended by an attribution algorithm. The training
of the models was performed on an Intel®Core™i7-7500U CPU at 2.70GHz without hardware
acceleration. Additionally, a server environment on TIRA [
In the following we will describe our approaches in more detail and state other important considerations for our application.
The data set provided by the PAN’21 SCD challenge consisted of one training and one validation
set with 11200 and 2400 problems, respectively, as well as solutions for these problems. The
problems consist of uniquely English posts scraped from the StackExchange network that where
put together in one to four author documents of total length between 1000 and 3000 characters.
Additionally to the training and validation sets a test set was generated by the PAN’21 team but
not provided during development. The test set is similar to the validation in terms of size and
content [
We manually extracted a small set of traditional features that are often used in Authorship
Attribution tasks [
if < 20 , otherwise 2− 1 We chose to manually implement those rather basic features as they have been shown to be good predictors of style. However, we did not expect the model trained on this set of vfie features to outperform the model trained on word embeddings, simply because of the seemingly small amount of information that such a set would achieve.
The word embeddings used for our model are the pretrained fastText word vectors provided by
Facebook’s AI Research lab [
For the task of multi-author prediction we chose to calculate the embeddings on a per-document basis. We anticipated that multi-author documents would be separate from single-author documents in embedding space.
For the task of style-change detection we use a per-paragraph embedding. Since each paragraph
is guaranteed to have been written by a single author, we computed the average of the word
embeddings for each paragraph, thus generating a paragraph embedding. As pointed out by
Kenter et al. [
Due to the promissing results reported by Hosseinia et al. (2018) [
The number of neurons and the learning rate were determined using a grid search approach in the range of 32 to 512 with increments of 32 for the number of neurons and the options of 10− 2, 10− 3 and 10− 4 for the learning rate. For the hidden layers this resulted in 480, 480, 288 neurons, respectively. The optimal learning rate was found to be 10− 3.
The activation of hidden layers was set as the ReLU function as this has proven to be successful
in MLP applications [
As input data we used only the per-document embeddings as described in Section 3.3 as we wanted to analyze the discriminative abilities of MLPs on per-document level. We believe that calculating per-document complexity measures would not yield a satisfying separation since the documents are constructed to be similar in their global structure.
We trained a two-layered Bidirectional LSTM model with 128 hidden units per layer, adding a Masking layer, and a Time-Distributed layer as the output layer with a sigmoid activation function. We used binary cross-entropy as the loss function. In order to prevent overfitting, early stopping was used.A threshold of 0.5 was established in order to decide whether there is a style change. We applied normalization. We found that a batch size of one achieved the best results, compared to batch sizes vfie and ten, which yielded results worse by 60% points.
Our anticipation for using an LSTM was that the model could learn similarities in writing style on a per-paragraph basis by using the paragraphs as time steps in its input.
We implemented different approaches for this task, which turned out to be more difcfiult than the previous two tasks. Our first approach was to train a simple LSTM model similar to that of task two, but because there are no consistent classes across the data, the model achieved bad results. The second approach was a two-fold pipeline containing a k-means clustering algorithm and a classification model. As the data for the Shared Task should represent real-world data the number of authors of every text is unknown and may differ from text to text. Therefore, it was not straightforward to build a clustering system that would predict the number of authors.
The third approach showed to yield the best result. For this approach, we used an LSTMpowered Attribution Algorithm, visualized in Figure 1 and represented as pseudo-code in Algorithm 1. The algorithm functions as follows. For every paragraph the authorship attribution decision is made. We utilize the style change detection prediction that is generated by our LSTM-model for the whole document. The first paragraph is always attributed to author A 1, we assume that the paragraphs are written by the same author, A1, as long as no style change occurs. After a style change was detected in the predictions we construct a new prediction problem for our LSTM to solve. This problem consists of the current paragraph up for authorship attribution and preceded by a previously attributed paragraph p. p is iterated until either, no style change is detected or p is equal to the current paragraph. The first case implies that the author of p is the same in the current paragraph while the second case implies that the author was not detected before and a new author should be selected.
In this section, we present the results we achieved by testing the models on the validation set that
was made available by the PAN’21 committee [
In this section we present our results for the task of detecting multi-authored documents, shown in Table 1. It can be seen that the model yields good results across all measures. The small drop in recall and accuracy suggest that the model classifies some multi-authored documents as single-authored documents. When evaluated on the test set, the F1-score drops noticeably, this can indicate that the model might not generalize well.
Nevertheless, the F1-score achieved by our model outperforms the winning’s team model from
last year’s PAN Task [
In this section we present our results for the task of detecting style change positions. As can be seen in Table 2 the three categories yielded similar results. It is interesting to see that the model trained with the textual features only achieved high accuracy. Nevertheless, the comparatively low recall score of 66.27% suggests that this model tend to predict no style change, the reason for the high accuracy being the imbalance in the data. The other two models trained with the embeddings and the combination of the embeddings and features achieved recall scores of 74% and 72.5%, respectively. This suggests that these models are more robust to imbalance in the data and therefore more suitable for real-world data.
Compared to the self-reported results from the winning team of last year’s PAN Shared Task,
our model does not outperform Iyer and Vosoughi’s model [
Nevertheless, all three models yield good results and despite the drop in recall, the feature model achieves satisfying results.
The implementation of the algorithm showed to be very computationally expensive and time consuming. Due to our limited access to computational power we were not able to run this code on the validation set in order to get results. Nevertheless, when running it on a smaller corpus we yielded promising outcomes. However, considering the small amount of data we used we decided to refrain from reviewing the results here.
The PAN team reported a F1-score of 26.25 %. Considering this result, we assume that a sequence of two paragraphs might not be enough information for the model to make a legitimate prediction.
In this notebook, we presented our approaches to solve the PAN’21 Shared Task of Style Change
Detection [
With regards to the stylometric aspect of the problems, we implemented only vfie different and rather traditional features. Surprisingly, it was possible to achieve good results with only those features, which shows that for this data set, the individual style of the authors was accurately representable by mean word length, mean sentence length, function word frequency, CTTR, and the LWF.
Our approaches to the first and second task yielded good results on the validation set, which
shows that simple machine learning models can be good solutions for the tasks. Future work could
try to implement attention based models [
The third task is a newer problem and more complicated to solve, especially on real-world data with an unknown number of unknown authors. For future projects, we plan on attempting to implement a clustering/classification model for this problem. Furthermore, it is possible to build up on our model by increasing the number of paragraphs used to make predictions and parallelizing the loop of the attribution algorithm to increase computation speed. We therefore consider our model to be a promising approach despite achieving low results. It is also important to note that we challenged ourselves to a more real-world-like problem by not taking into account the maximum number of authors, as it would certainly lower the complexity of the task if this information is used.