This paper introduces the application of Virtual Softmax for the PAN 2024 multi author writing style analysis task. We found that tasks with the same topic are particularly challenging due to dificulties at classification boundaries. To address this problem, we integrated Virtual Softmax into the Transformer architecture to provide additional feature supervision, enhancing the recognition ability of model. Finally, we achieved F1 scores higher than the baseline method on the three tasks in the oficial test set.
With the rise and improvement of large language model technology, the methods for handling complex
tasks have undergone significant changes. Traditional work, such as the research by Gómez-Adorno
et al. [
In our work, we compared the performance of three pre-trained transformer-based models on Task 1 to select the most suitable base model. Additionally, we applied data augmentation techniques and used Virtual Softmax to enhance the model’s performance in handling samples near the classification boundaries.
The Transformer [
In our work, we use the RoBERTa-based model as the encoder to process the input text. The input text paragraphs are first tokenized and then fed into the RoBERTa model for encoding. The pooled output of the [cls] token, represents the contextual features of the entire paragraph. This output is then passed through a Virtual Softmax layer. The model is trained using a cross-entropy loss function to perform our classification task. During training, the extracted paragraph features are fed into a Virtual Softmax layer, enabling the model to perform a three-class classification task. An additional class is introduced to provide feature supervision, which compresses the inter-class space of the other two classes, thereby enforcing stricter boundary constraints.
To enhance the model’s discriminative power, we integrate a Virtual Softmax layer. During the training phase, no additional processing is performed on the input data and Virtual Classes are directly added. During the evaluation phase, we choose the category with the highest probability among the non-virtual classes. These classes do not correspond to actual categories but are used to increase the complexity of the training process, thereby strengthening the model’s discriminative abilities. The core idea is to add noise by incorporating these virtual classes, forcing the model to generalize better when faced with real data. Specifically, for a given classification task, the additional injected classes introduce a new and tighter decision boundary for the original classes, compressing their inter-class distribution.
In this work, we select the RoBERTa-based model [
In the data provided by PAN, three tasks, categorized by dificulty (task1-easy, task2-medium, task3hard), were divided into a training set (70%), a validation set (15%), and a test set (15%). The training set for each dificulty consists of 4200 documents, consisting of multiple paragraphs. We connect two adjacent paragraphs in the same document using a separator token [cls] to form a sample, and label it whether the author of the sample text has changed.
Samples constructed according to the above method are made up of adjacent paragraphs. We extended the data set by linking discontinuous paragraphs together based on some logical judgment based on the number of authors and whether the document paragraphs changed. For example, if there is no author change for three consecutive paragraphs, the first paragraph and the third paragraph can form a new sample.
We submit the model to TIRA [10] for execution to get the final metrics for the model. Table 3 shows the F1 scores obtained by our model in the oficial test and validation set. In addition, the paper compares the performance of some methods in 2023, Chen et al. [11], Jacobo et al. [12].
In task1 and task2, authors can be distinguished by capturing the characteristics of the topic. Due to the architectural limitations of Transformer, style changes and context dependencies cannot be fully captured. This results in a low F1 score in task3, where each pair of paragraphs has the same topic and keywords.
In this paper, we presented a RoBERTa-based model enhanced with Virtual Softmax for detecting style changes in multi-author documents. Our approach showed significant promise, particularly in the more challenging scenarios where documents share the same topic throughout. By injecting additional classes, we were able to improve the model’s ability to distinguish between diferent authors, thereby enhancing the robustness and accuracy of style change detection.
This work is supported by the National Natural Science Foundation of China (No.62276064). [10] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023, pp. 236–241. doi:10.1007/ 978-3-031-28241-6_20. [11] H. Chen, Z. Han, Z. Li, Y. Han, A writing style embedding based on contrastive learning for multi-author writing style analysis, in: Conference and Labs of the Evaluation Forum (CLEF), 2023. [12] G. X. Jacobo, V. Dehesa-Corona, A. D. Rojas-Reyes, H. Gómez-Adorno, Authorship verification machine learning methods for style change detection in texts, in: Conference and Labs of the Evaluation Forum (CLEF), 2023.