PAN 2024 conducted Multi-Author Writing Style Analysis task which aims to detect style changes between consecutive paragraphs in a text. The task provides datasets with three levels of complexity to test the submissions. This paper describes our attempt towards solving this problem. It involves multiple stylometric features extracted from the input text and detecting any style changes using a trained Neural Network based on CNN and Bi-LSTM along with global max pooling layers. The proposed system obtained a F1 score of 0.78, 0.724, 0.601 for the 3 subtasks on validation data set provided.
Multiple features are extracted from the source documents and embeddings generated before feeding them to the neural network. The proposed system has been able to achieve good results on the data set provided by this task.
PAN 2024 Multi-Author Writing Style Analysis task is further subdivided into 3 sub tasks with increasing level of difficulty. 1. Easy - The paragraphs of a document cover a variety of topics, allowing approaches to make use of topic information to detect authorship changes. 2. Medium - The topical variety in a document is small (though still present) forcing the approaches to focus more on style to effectively solve the detection task. 3. Hard - All paragraphs in a document are on the same topic.
PAN provided three different data sets for each of these sub tasks. These
datasets are further sub divided into 3 sets one each for Training, Validation
and Test. The proposed system discussed in this paper is trained using the
above training dataset and validated with the Validation dataset. The trained
system is submitted to TIRA [
PAN at CLEF, have been in the past, successively conducted style change
detection tasks since 2017. Some of the work in this area include Supervised
Contrastive Learning for Multi-Author Writing Style Analysis [
In the proposed system the input text is divided into pairs of consecutive paragraphs. The system then generates embeddings based on multiple stylometric features extracted from the input text. These features include: TFIDF for character n-grams TFIDF for n-grams of POS tags TFIDF for n-grams of POS tag chunks TFIDF for punctuation marks used in the text Frequency of stop words Count of characters in the text Count of words in the text Such multiple features are extracted from the text to better represent the style of the author. These embeddings are fed into a neural network which is trained on the training data to predict if a pair of paragraphs has similar stylometric properties or not.
The neural network consists of a combination of one dimensional convolution neural network and Bi-directional LSTM layers which are concatenated along with Global Max pooling followed by a dense layer. The final (output) layer does the classification. Fig 1. shows the structure of this neural network. The neural network was trained 3 times one with each dataset corresponding to the subtasks (Easy, Medium and Hard) and three different models were generated for each sub task.
LSTM (Long Short-Term Memory) network is a special type of recurrent neural network which is better suited for maintaining long range connections within a sequence. Bi-LSTM (Bidirectional LSTM) is a combination of two LSTM layers with inputs flowing from both directions unlike LSTM where the input flows only in one direction. In other words, Bi-LSTM can analyse both past and future information and thus give a more meaningful output especially in natural language processing. In the proposed system Bi-LSTM layer is set with dropout of 0.2. Adding dropouts improves the generalization and avoids over fitting the training data.
Convolutional Neural Network (CNN) extracts important features from the input which helps in reducing the number of features and thereby improving the accuracy and performance of the model. In the proposed system the CNN layer uses ‘Relu’ as the activation function. This is followed by the Global Max Pooling which reduces the input dimensions thereby reducing the input parameters. This helps in further improving the accuracy and speed. Further the output of the Bi-LSTM and the global max pooling is concatenated together followed by a dense layer with ‘Relu’ activation function. Finally, the output layer produces the classification using ‘Softmax’ function.
Fig 1. Structure of the proposed neural network The system was trained on two different data sets one from PAN 2023 and the other from PAN 2024 style change detection tasks. Both these datasets are similar in structure. These datasets are divided into three parts based on the increasing levels of difficulty (Easy, Medium and Hard) as described earlier.
With this training two models were generated.
The proposed system was submitted to the PAN Multi-Author Writing Style Analysis task. The two submissions based on the two sets for models (one based on the PAN 2023 style change detection Task and the other based on the PAN 2024 style change detection Task) were named ‘rancid-factor’ and ‘knurled-starter’ respectively. Going forward these systems would be name System1 and System2 respectively. The three different models trained earlier did the predictions for the three different subtasks. The results of the run on the validation data shows F1 Scores for the 3 tasks in Table 1. Table 2 shows the results of the run on the Test data set. The score of the two baseline predictors are also mentioned in Table 2. The first baseline predictor (Baseline Predict 1) always predicts 1 i.e. change in the author between the consecutive paragraphs and the second baseline predictor (Baseline Predict 2) always predicts 0 i.e. no change in the author between the consecutive paragraphs of a document.
The two systems performed much better than the two baseline systems provided. System2 performed much better in Task1 but both the systems got similar scores for Task 2 and Task 3. Both the systems did not do well in the Task 3 which is corresponding to the ‘Hard’ subtask which means more work is required in the area where the variety of the topics were very less and the system needs to be more style oriented rather than topic oriented. This calls for a better feature extraction techniques.