Our study addresses the intricate task of detecting style changes within documents authored by multiple individuals. The primary aim is to pinpoint instances where authors transition within the text. This task holds significant importance in author identification, particularly in situations where no comparative texts are available. By discerning variations in writing style, we can unveil instances of plagiarism, identify instances of gift authorship, authenticate claimed authorships, and potentially develop novel technologies to support writing endeavors. Our methodology employs sophisticated ensemble learning techniques, incorporating fine-tuned Bidirectional Encoder Representations from Transformers (BERT), Robustly Optimized BERT Pretraining Approach (RoBERTa) and Eficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA) models for medium and hard subtask and fine-tuned RoBERTa for easy subtask, to efectively address this complex challenge across diverse datasets, such as those found in Reddit comments. Our team's approach achieved F1-scores of 97.2%, 66.4%, and 64.2% for the Easy, Medium, and Hard subtasks, respectively.
A RoBERTa and employed data augmentation and ensemble learning techniques. Their models ranked
ifrst in two sub-tasks and second in others, showcasing strong performance. Chen et al. [
While feature extraction methods like TF-IDF have yielded satisfactory results in author profiling
and Authorship Attribution tasks [
In this study, we investigated the eficacy of transformer-based models for detecting changes in authorial style at the paragraph level. Initially, we fine-tuned three transformer-based models on the training dataset. Furthermore, we employed Ensemble learning techniques to augment the performance of our approach.
The datasets provided by PAN@CLEF for writing SCD are categorized into three dificulty levels: Easy, Medium, and Hard. Each level presents unique challenges for detecting shifts in authorship within documents. On the Easy level, documents contain paragraphs covering diverse topics, allowing approaches to utilize topic information efectively in detecting authorship changes. Conversely, the Medium level entails documents with limited topical variety, requiring a stronger emphasis on stylistic analysis to address the detection task. Finally, the Hard level poses the most challenging scenario, where all paragraphs within a document focus on the same topic, demanding approaches to rely solely on stylistic cues for accurate detection.
Moreover, for each subtask, PAN@CLEF provides distinct datasets comprising multiple documents, each containing paragraphs. Accompanying these datasets are ground truth files, which furnish essential information: the number of authors associated with each document and the identification of consecutive paragraphs where style changes occur, signifying transitions in authorship. These datasets are partitioned into training and validation sets to facilitate experimental setup. In our approach, we treat every pair of consecutive paragraphs as a sample, concatenating them and assigning a label indicating whether a style change occurred between the two paragraphs (labeled as 1) or if they were written by the same author (labeled as 0).
For each subtask, our methodology involves concatenating two consecutive paragraphs and assigning them a binary label, thereby framing each subtask as a binary classification problem. In our study, we harness the power of three state-of-the-art transformer-based models: BERT, RoBERTa, and ELECTRA complemented by their corresponding tokenizers. Given the constraint of 512 tokens imposed by the medium-sized models and the rarity of samples exceeding this limit, we adopt a uniform approach by adhering to the 512-token threshold and truncating longer samples as necessary. Additionally, to ensure consistent attention distribution across paragraphs within each sample, we implement a truncation strategy that involves removing tokens from the end of the sequence. This meticulous approach enables us to efectively leverage the capabilities of these advanced models while addressing the practical constraints of our dataset.
Transformer models have revolutionized natural language processing, capturing complex relationships
in text data through self-attention mechanisms. Unlike traditional models, they process all words in a
sentence simultaneously, enabling eficient handling of longer sequences. Pre-trained on large corpora,
models like BERT and Generative Pre-trained Transformer (GPT) learn rich language representations,
adaptable to various tasks with minimal labeled data. Widely used in machine translation,
summarization [
BERT, RoBERTa, and ELECTRA are prominent transformer-based models in NLP. BERT, introduced by Google, utilizes bidirectional training and transformer architecture to capture contextual information efectively. RoBERTa, an improvement upon BERT, optimizes training strategies and scales model size, achieving superior performance on various NLP tasks. ELECTRA, employing a novel pre-training approach, replaces tokens with plausible alternatives and trains a discriminator to distinguish between real and replaced tokens, enhancing eficiency and learning efectiveness.
Ultimately, we augmented each language model with a binary classification layer to detect changes in writing style. We fine-tuned the models for each subtask using their respective datasets. Additionally, we aggregated the datasets from all three subtasks and fine-tuned the RoBERTa and ELECTRA models on this combined dataset.
Hard Ensemble Learning refers to a sophisticated approach in machine learning where multiple models, known as base learners, are combined to form a stronger, more robust predictive model. Unlike traditional ensemble methods, such as bagging and boosting, which primarily focus on combining diverse but weak models, Hard Ensemble Learning integrates several complex and high-performing models to tackle challenging tasks. This technique leverages the collective intelligence of diverse models, each trained on diferent aspects of the data, to improve overall predictive accuracy and generalization.
In this study, we developed five single models for each subtask, all based on three fine-tuned models: BERT, RoBERTa, and ELECTRA. Initially, we fine-tuned all three models with the dataset corresponding to each subtask. Then, we combined the datasets of all three subtasks and fine-tuned the RoBERTa and ELECTRA models on them. Our goal with ensemble learning is to leverage the strengths of each approach to enhance system performance. Detailed specifics will be provided in the next section.
In this study, we utilized Google Colaboratory’s GPU to fine-tune the BERT, RoBERTa, and ELECTRA models. Due to token constraints, we limited our consideration to 512 tokens. The hyperparameter learning rate for all three transformer models was set to 2e-5. For the dataset associated with the "easy" subtask, we employed an epoch value of 10, while for the other two subtasks, we used 7 epochs. Moreover, for the fine-tuning models chosen for the entire subtasks dataset (RoBERTa, ELECTRA), the epoch value was set to 3.
The F1-score is a balanced measure of a model’s accuracy, combining precision and recall. Precision measures the accuracy of positive predictions, while recall measures the model’s ability to find all positive instances. The macro F1-score averages F1-scores across all classes, providing a balanced evaluation, especially in datasets with class imbalances. Following our experimentation and analysis of results from evaluation sets, we identify the most efective model for each subtask. Subsequently, we execute the selected model on an unseen test set through the TIRA platform [12].
In this section, we present the results of our experiments on the SCD task. Table 1 summarizes the performance of the tested models. We conducted 12 diferent experiments for each subtask, both with standalone models and ensemble learning. Our findings show that for the "easy" subtask, the pre-trained RoBERTa model achieved the highest macro F1-score on the provided validation dataset. However, for the remaining subtasks, ensemble learning models outperformed others. Specifically, fine-tuning RoBERTa with the entire subtasks dataset (Combined1) and ELECTRA (Combined2), as well as training ELECTRA, RoBERTa, and BERT models on the medium subtask dataset, yielded the highest macro F1-score accuracy. Moreover, for the "Hard" subtask, models trained on RoBERTa and ELECTRA, along with Combined1, delivered the best performance across all experiments. Therefore, for each subtask’s test dataset, we selected the approach with the best performance. The results of our work on the unseen dataset are presented in Table 2.
This paper provides an overview of our team’s performance in the PAN Shared Task for the SCD task. We embarked on a series of diverse experiments to pinpoint the most efective strategy for identifying shifts in writing style within consecutive paragraphs. Initially, we trained five distinct language models, meticulously fine-tuning each to grasp the intricacies of our task. Following this, we delved into an extensive exploration of various combinations of these models, tailoring our approaches to suit the unique demands of each subtask. For the "easy" subtask, we opted to leverage the capabilities of the RoBERTa model exclusively, given its robust performance in preliminary assessments. However, for the more complex subtasks, we adopted an ensemble learning approach, harnessing the collective power of multiple models to tackle the nuanced challenges presented. [11] M. K. Sheykhlan, J. Shafi, S. Kosari, Pars-hao: Hate speech and ofensive language detection on persian social media using ensemble learning, Authorea Preprints (2023). [12] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023, pp. 236–241. URL: https://link. springer.com/chapter/10.1007/978-3-031-28241-6_20. doi:10.1007/978-3-031-28241-6_20.