This paper presents our participation in the Multi-Author Writing Style Analysis Task for PAN at CLEF 2024. The primary goal of this task involves detecting style changes in multi-author documents at the paragraph level. The task consists of three sub-tasks: Easy, Medium, and Hard, each varying in dificulty, mainly depending on the range of topics covered in the paragraphs. We discuss the significance of style change detection in various applications such as plagiarism detection, authorship verification, and writing support. Our Approach leverages LLMs such as Electra, Deberta, Squeezebert, and Roberta.Hence we test diferent models to come up with the one that is most suitable for our use case, based on the F1-scores.
The PAN challenge [
PAN stands for the acronym Uncovering Plagiarism, authorship, and Social Software Misuse. The
MultiAuthor Analysis task evolved from Author Clustering/Diarization and Author Masking/Evaluation,
detailed in [
Subsequently, sentences were split, employing a nuanced approach to punctuation for accuracy.
Utilizing BERT tokenization, embeddings were generated at the sentence level, with model selection
guided by task specifics and performance metrics. Embedding was then amalgamated using tailored
methods to suit the requirements of each task. At the document level, sentence vectors were averaged
to encapsulate the document’s essence. Conversely, embedding was averaged between consecutive
paragraphs at the paragraph level to discern stylistic changes. Hence, this resulted in the approach of
Iyer and Vosoughi [
Moreover, two diferent top-performing approaches for PAN 2021, Zhang et al. [
Furthermore, in PAN 2022, the top-performing [
Finally, in PAN 2023 the approach by Hashmi et al. [
In addition to that [
The dataset is neatly divided into training, validation, and test sets of data, The training set contains 70 percent of the dataset including the text files, ground-truths, and JSON files, which are used to develop our models. The validation dataset includes 15 percent of the whole dataset and the testing dataset comprises the remaining 15 percent. As the dataset is already split into relevant sections, we can carry on and consider the pairs of successive paragraphs as input samples to our model, hence concatenating them in the process, each one is then assigned a label that shows whether the data has changed or not.
Furthermore, this means that for n set of paragraphs in the text file, we have n-1 paragraph pairs and Labels, hence we have converted the task of identifying the authors into binary classification, where 0 means there is no change and 1 means there is a change of authors between the paragraphs. Two such sets of datasets would be created in our case, one for training the model and the other one for validation of the model.
In Table 1, it is evident that the easy dataset exhibits class imbalance, with a significantly higher occurrence of class 1 (left column) compared to class 0 (right column). To address this imbalance, we implemented a weighted random sampler. To implement this we first ensure that the ’changes’ column in the training data frame is appropriately formatted. It then computes the class distribution by counting the occurrences of each class label. By inversely weighting the class frequencies, the function calculates a set of weights such that minority class samples are given a higher probability during sampling. These weights are then used to create a weighted random sampler, which ensures that each class is represented proportionally in the training process, thereby mitigating the efects of class imbalance and potentially improving model performance on minority classes.
Pre-trained transformer models have been trained on enormous volumes of text data. They gain an understanding of language’s structure and patterns, which helps them produce text of the highest caliber and carry out various natural language processing (NLP) functions. By fine-tuning the pre-trained models, we can take advantage of their language comprehension abilities and apply the knowledge they have learned from their thorough pre-training to our particular task. Hence for this project, we employed popular pre-transformer models RoBERTa, ELECTRA, DeBERTa, and SqueezeBERT.
RoBERTa uses dynamic masking and omits next-sentence prediction to improve BERT’s pre-training, resulting in stronger word representations. ELECTRA focuses on eficiency, using synthetic data with replaced words to teach the model to distinguish them from originals, achieving good results with less training data. DeBERTa combines ideas from ELECTRA and BERT, separating word and positional information for better context understanding and using an ELECTRA-like task for efective training and high performance. SqueezeBert reduces model size by compressing a large pre-trained model like BERT into a faster, smaller version while preserving most of its functionality through knowledge distillation
We used the base versions of several pre-trained models available on HuggingFace. These models were implemented on Kaggle notebooks, and they were then refined further. We used the following hyperparameter values to improve the model’s performance: a maximum sequence length of 256, a learning rate of 0.00001, a batch size of 16, and 12 epochs.
We calculated the F1 score on the given evaluation set to assess the models’ efectiveness for every subtask. This F1 score was determined by comparing the models’ predictions for each sub-task evaluation set’s ability to identify style changes between consecutive paragraph pairs.
From the results shown in Table 2, we observe a distinct trend in model performance. On the easy dataset, the RoBERTa model achieves the highest F1-score of 0.940, followed closely by ELECTRA with an F1-score of 0.939. DeBERTa secures the third position with an F1-score of 0.756, while SqueezeBERT ranks last. A similar performance pattern is evident in the medium dataset, with RoBERTa leading and SqueezeBERT trailing.
However, in the hard dataset, the performance dynamics shift. Although RoBERTa maintains its leading position, DeBERTa surpasses ELECTRA, achieving an F1-score of 0.756 compared to ELECTRA’s 0.713. SqueezeBERT continues to underperform in this dataset as well.
The observed trend in model performance across the Easy, Medium, and Hard datasets can be attributed to several factors related to the models’ inherent architecture and training strategies.
Thanks to a dynamic masking method, intensive training on large-scale datasets, and a resilient architecture, RoBERTa regularly outperforms competing models. These elements enable RoBERTa to more successfully identify complex patterns in the data, as seen by its high F1 scores in every dataset. ELECTRA, on the other hand, is comparable to RoBERTa. Still, its distinct pre-training method—which uses substitute token detection—might not be able to handle the subtle complexity seen in more dificult datasets. This explains the modest decline in ELECTRA’s performance compared to RoBERTa, especially in the Hard dataset.
Diferent datasets show diferent performances for DeBERTa, which uses relative position embeddings and disentangled attention mechanisms. While it does well overall, its architecture may not fully take advantage of the simpler structures in the Easy and Medium datasets, which could explain why it ranks third in these situations while SqueezeBERT’s lower performance across all datasets can be attributed to its design, which prioritizes model eficiency and reduced computational resources over capturing complex patterns.
Based upon the results it was best to work with RoBERTa and fine-tune the model further to achieve better scores.
To further improve the performance metrics of the RoBERTa model, we initially considered utilizing the larger variant available on HuggingFace. However, given the widespread use of large language models (LLMs) in the current research community and their significant environmental impact, we opted for alternative optimization methods. These approaches are environmentally sustainable and capable of achieving comparable performance enhancements.
One way we thought of improving the performance was by increasing the size of the dataset, this method is also called data augmentation. To realize this we had a unique approach since we were giving models two paragraphs and a label. Let’s consider paragraphs and with a label of 0, indicating no change. We pass this data to our models, but if we reverse the order of the paragraphs—placing paragraph before paragraph —the label remains unchanged. This method efectively doubles the size of our dataset.
As demonstrated in Table 4, employing the new strategy has improved scores across all datasets. This suggests that augmenting the dataset size positively impacts performance metrics. Specifically, the Easy dataset achieved an F1-score of 0.958, the Medium dataset reached an F1-score of 0.816, and the Hard dataset attained an F1-score of 0.787. Based on these results, we conclude that the RoBERTa base models, trained on the Easy, Medium, and Hard datasets, should be submitted for evaluation on the TIRA platform.
The approaches titled "presto-door" and "null-directory" are the same models, i.e. both are the score obtained through the same Roberta-base-discriminator model. The only diference between the approaches is that one was submitted manually through Docker, whereas the other was submitted through the GitHub Actions tool. Moreover, test set results on approaches titled "presto-door" and "null-directory" displayed in Table 5 reiterate our revised strategy of expanding the dataset and then utilizing the Roberta model on it. The scores for the test sets are much better and closer to the validation set provided. In conclusion, this validates that, first of all, Roberta is the best-performing model amongst the models tested, and secondly the strategy of Data Augmentation aids in providing a better model F1-scores than, the models trained on just the initial datasets provided.
As students, academians, and part of a community that is aimed towards protecting the environment, building upon the United Nations sustainability goals, and gearing towards lower greenhouse emissions, we choose to work with models that have minimal impact on the environment, as we eficiently tried to utilize the resources available to us, to complete the task of multi-author classification in PAN 2024
Our solution builds upon binary classification, hence it works by treating the Paragraph as two instances of input text and giving output a label that tells whether the author is changing or not. While our approach achieved high accuracy on the task of multi-author classification using the best models out of the many fine-tuned transformers, it’s important to acknowledge some limitations. Running multiple models can be computationally expensive, and fine-tuning often requires significant labeled data. Additionally, the complex nature of transformer models can make it dificult to understand how they arrive at their predictions. Despite these limitations, our approach achieved strong performance by leveraging the best-performing models based on F1 scores on both the training and test sets. However, it’s important to note that this binary classification approach treats the task as a simple presence or absence of a style change, neglecting the potential for nuanced stylistic variations within a single author’s work. Future work could explore multi-class classification to capture a wider range of stylistic distinctions or delve deeper into interpretability techniques to understand the reasoning behind the models’ predictions.
The authors would like to acknowledge the support provided by the Ofice Of Research (OoR) at Habib University, Karachi, Pakistan for funding this project through the internal research grant IRG-2235.