With the advent of internet and generative AI tools it is quite easy to copy someone else's work and embed in one's work or to copy from multiple sources and claim to be your work. This puts a great emphasis in the intellectual property rights. Detecting such plagiarism manually is quite difficult. However using a style change detection system greatly improves the accuracy and time of this task. These techniques could also be used to classify authors. This type of analysis along with other textual analyses comes under forensic text analysis. PAN workshops series has been quite active in this area since 2009. One of the tasks conducted as part of PAN at CLEF 2024 [1] is 'Multi-Author Writing Style Analysis' [2] which is continuation of a series of such tasks conducted in the past since 2018. The aim of this task in 2024 is to detect stylometric changes across consecutive paragraphs of a document. The task is divided into three sub tasks based on the difficulty levels. Task 1 involves documents that cover a variety of topics. Task 2 also, includes documents with a small variety of topics but not as much as with Task 1. Task 3 on the other hand consists of documents of the same topic. This paper proposes a solution for PAN 2024 Multi-Author Writing Style Analysis task after analyzing the past work around this area. The proposed solution here employs Neural Network model with a combination of CNN and Bi-LSTM along with Global Max Pooling to help detect the style changes.
Multiple features are extracted from the source documents and embeddings generated before feeding them to the neural network. The proposed system has been able to achieve good results on the data set provided by this task.
PAN 2024 Multi-Author Writing Style Analysis task is further subdivided into 3 sub tasks with increasing level of difficulty.
1. Easy -The paragraphs of a document cover a variety of topics, allowing approaches to make use of topic information to detect authorship changes.
2. Medium -The topical variety in a document is small (though still present) forcing the approaches to focus more on style to effectively solve the detection task.
3. Hard -All paragraphs in a document are on the same topic.
PAN provided three different data sets for each of these sub tasks. These datasets are further sub divided into 3 sets one each for Training, Validation and Test. The proposed system discussed in this paper is trained using the above training dataset and validated with the Validation dataset. The trained system is submitted to TIRA [3] platform where the system was evaluated based on the Test dataset.
PAN at CLEF, have been in the past, successively conducted style change detection tasks since 2017. Some of the work in this area include Supervised Contrastive Learning for Multi-Author Writing Style Analysis [4] in 2023, Ensemble-Based Clustering for Writing Style Change Detection in Multi-Authored Textual Documents [5] and Style Change Detection Based On Bi-LSTM And Bert in 2022 [6]. The last one proposed a system using a neural network involving Bi-LSTM and CNN with BERT embeddings as the input. The system proposed in this paper is to some extent based on this work but differs in the structure of the neural network and the input to it. Other similar works include -Style Change Detection using Siamese Neural Networks [7]. This proposed system included a Siamese network with GloVe embedding layer, a Bi-LSTM layer along with other layers.
In the proposed system the input text is divided into pairs of consecutive paragraphs. The system then generates embeddings based on multiple stylometric features extracted from the input text. These features include:
TFIDF for character n-grams TFIDF for n-grams of POS tags TFIDF for n-grams of POS tag chunks TFIDF for punctuation marks used in the text Frequency of stop words Count of characters in the text Count of words in the text Such multiple features are extracted from the text to better represent the style of the author. These embeddings are fed into a neural network which is trained on the training data to predict if a pair of paragraphs has similar stylometric properties or not.
The neural network consists of a combination of one dimensional convolution neural network and Bi-directional LSTM layers which are concatenated along with Global Max pooling followed by a dense layer. The final (output) layer does the classification. Fig 1 . shows the structure of this neural network. The neural network was trained 3 times one with each dataset corresponding to the subtasks (Easy, Medium and Hard) and three different models were generated for each sub task. LSTM (Long Short-Term Memory) network is a special type of recurrent neural network which is better suited for maintaining long range connections within a sequence. Bi-LSTM (Bidirectional LSTM) is a combination of two LSTM layers with inputs flowing from both directions unlike LSTM where the input flows only in one direction. In other words, Bi-LSTM can analyse both past and future information and thus give a more meaningful output especially in natural language processing. In the proposed system Bi-LSTM layer is set with dropout of 0.2. Adding dropouts improves the generalization and avoids over fitting the training data.
Convolutional Neural Network (CNN) extracts important features from the input which helps in reducing the number of features and thereby improving the accuracy and performance of the model. In the proposed system the CNN layer uses 'Relu' as the activation function. This is followed by the Global Max Pooling which reduces the input dimensions thereby reducing the input parameters. This helps in further improving the accuracy and speed. Further the output of the Bi-LSTM and the global max pooling is concatenated together followed by a dense layer with 'Relu' activation function. Finally, the output layer produces the classification using 'Softmax' function. The system was trained on two different data sets one from PAN 2023 and the other from PAN 2024 style change detection tasks. Both these datasets are similar in structure. These datasets are divided into three parts based on the increasing levels of difficulty (Easy, Medium and Hard) as described earlier.
With this training two models were generated.
The proposed system was submitted to the PAN Multi-Author Writing Style Analysis task. The two submissions based on the two sets for models (one based on the PAN 2023 style change detection Task and the other based on the PAN 2024 style change detection Task) were named 'rancid-factor' and 'knurled-starter' respectively. Going forward these systems would be name System1 and System2 respectively. The three different models trained earlier did the predictions for the three different subtasks. The results of the run on the validation data shows F1 Scores for the 3 tasks in Table 1. Table 2 shows the results of the run on the Test data set. The score of the two baseline predictors are also mentioned in Table 2. The first baseline predictor (Baseline Predict 1) always predicts 1 i.e. change in the author between the consecutive paragraphs and the second baseline predictor (Baseline Predict 2) always predicts 0 i.e. no change in the author between the consecutive paragraphs of a document.
The two systems performed much better than the two baseline systems provided. System2 performed much better in Task1 but both the systems got similar scores for Task 2 and Task 3. Both the systems did not do well in the Task 3 which is corresponding to the 'Hard' subtask which means more work is required in the area where the variety of the topics were very less and the system needs to be more style oriented rather than topic oriented. This calls for a better feature extraction techniques.