We propose a training model based on BERT series. This method uses sliding window technique to preprocess data sets to train and solve multi-author writing style analysis tasks. Our method is to combine adjacent paragraphs into a training sample, and efectively extract the characteristics of style changes between paragraphs. We conducted systematic training and evaluation on three multi-author writing style analysis datasets (easy, medium, and dificult) at diferent dificulty levels provided by the PAN organization. We obtained f1 scores of 0.993, 0.831 and 0.825 on the test set, respectively, which proved the efectiveness and robustness of the proposed method.
The Multi-Author Writing Style Analysis task is designed to identify where authorship has changed
in multiple author documents. In practical applications, we can analyze the author’s writing style to
determine the author’s identity, verify whether the document has been tampered with, and whether the
article is suspected of plagiarism [
Multi-author recognition research is mainly divided into two directions: traditional methods and deep
learning-based methods [
We created data samples using a sliding window. We encoded and classified the data using Bert
[
In this study, we utilized four diferent pre-trained language models [
DeBERTa-base and DeBERTa-v3-large introduce eficient attention mechanisms and improved sentence encoding strategies, making them particularly suitable for tasks that require complex language understanding [8].
ALBERT-large-v2, through parameter sharing techniques, maintains a large model capacity while reducing the number of parameters, making it suitable for processing large datasets and balancing performance and eficiency [9].
RoBERTa-base has shown excellent performance in a variety of natural language processing tasks through dynamic masked language model training, and its robustness makes it a reliable choice for experiments [10].
We designed targeted training strategies for each model to adapt to tasks of diferent dificulty levels. During the training process, we used precision, recall, and F1 score as the main evaluation metrics to ensure that the models can achieve optimal performance in detecting changes in author style.
The writing style change detection dataset provided by PAN@CLEF [
2.Medium: There is little diversity in the topics within the documents (although still present), forcing methods to focus more on style to efectively address the detection task.
3.Hard: All paragraphs in the documents are related to the same topic.
In the dataset provided by PAN, the label information available to participants includes the number of authors in the document and labels indicating whether there are changes in writing style between paragraphs. We segmented the documents in the dataset according to natural paragraphs and labels, and recalculated the quantities of the dataset. The statistical results are shown in the following table.
In this study, we first carried out preprocessing on the dataset. Initially, we conducted paragraph segmentation. We segmented the documents into natural paragraphs, treating each paragraph as an independent data item. Subsequently, we performed text pair extraction. Utilizing the sliding window method, we construct a text pair adjacent paragraphs. Finally, we read the corresponding label information for each text pair from the JSON file, which will be used for subsequent training and evaluation.
In this experiment,We chose the CrossEntropyLoss as the loss function, which measures the error based on the similarity between probability distributions and is used in conjunction with the softmax activation function, making it suitable for classification tasks. The optimizer selected was AdamW, an improved version of the Adam optimizer, particularly suitable for weight decay, which helps reduce overfitting. The learning rate was set to 1e-5. A smaller learning rate helps the model to train stably and converge to the global optimum. The batch size was set to 2 and the number of iterations was set to 10. Smaller batch sizes and numbers of iterations help enhance the model’s generalization ability and reduce overfitting.We set the maximum length of the encoder to 256, which means the total length of the text pairs is 512, since the number of tokens in most text pairs is less than 512. We established the dropout layer ratio at 0.1 to prevent overfitting during fine-tuning [11].
We then continuously replaced the pre-trained models and their corresponding neural network frameworks to compare the performance of the trained models. By comparing the accuracy on the validation set, we found that the model trained with DeBERTa [8]had the highest accuracy on the easy and hard datasets, while the model trained with RoBERTa [10]had the highest accuracy on the medium dataset.
In machine learning, the batch size is a crucial hyperparameter that can significantly afect the model’s convergence rate and final performance. A larger batch size usually provides a more stable gradient estimate but also increases memory consumption and may lead the model to become trapped in local optima. Based on these considerations, we increased the batch size by tenfold, hoping to improve model performance through more stable gradient estimation.
However, after a series of experiments, we found that increasing the batch size did not lead to the expected performance improvement. Specifically, after increasing the batch size, the model’s accuracy on the validation set decreased by 10%, and the training time also increased. This indicates that for the current task and the adopted model architecture, a larger batch size may not be the optimal choice. Therefore, we ultimately set the batch size to 2.
We carried out four experiments: the BERT series pre-training models (DeBERTa-base,
Deberta-v3large, ALBERT-large-v2 and RoBERTa-base) were used to train on the training set, and the model with
the highest accuracy was selected in the verification set to fine-tune the hyperparameters. Then we
implement the model with the highest F1 score on the verification sets corresponding to diferent
dificulty levels submitted by the TIRA platform [
We also compare the performance of diferent Bert-based models, denoted as DeBERTa-base, Debertav3-large, ALBERT-large-v2 and RoBERTa-base. The experiment results concerning accuracy are shown in Table 3, where Easy, Medium and Hard denote the dataset of diferent dificulty levels.
In this study, we successfully developed a system based on DeBERTa-base and RoBERTa-base models to efectively process the multi-author writing style analysis task by using sliding window technique. By applying sliding Windows to text sequences, our method can efectively capture local context information and extract stylistic changes between paragraphs. On three datasets of diferent dificulty levels provided by the PAN organization,our approach achieves excellent scores on the f1 index, demonstrating its efectiveness and robustness.
Despite the positive results of our study, there are still some limitations and issues that need to be further explored. In experiments, it was found that increasing batch size did not improve performance, but led to a decrease in accuracy. This suggests that batch size has a significant impact on model performance, but its optimal value may depend on the specific task and model architecture and requires more in-depth research to determine. We are interested in experimenting with emerging pre-trained model architectures to explore whether they can bring further performance improvements for multiauthor style detection tasks.
This research was supported by the National Social Science Foundation of China (22BTQ101) . [8] P. He, X. Liu, J. Gao, W. Chen, Deberta: Decoding-enhanced bert with disentangled attention, 2021.
URL: https://arxiv.org/abs/2006.03654. arXiv:2006.03654. [9] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, Albert: A lite bert for selfsupervised learning of language representations, 2020. URL: https://arxiv.org/abs/1909.11942. arXiv:1909.11942. [10] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, 2019. URL: https://arxiv.org/abs/1907. 11692. arXiv:1907.11692. [11] X. Liang, L. Wu, J. Li, Y. Wang, Q. Meng, T. Qin, W. Chen, M. Zhang, T.-Y. Liu, R-drop: Regularized dropout for neural networks, 2021. URL: https://arxiv.org/abs/2106.14448. arXiv:2106.14448.