The objective of writing style change detection is to identify the positions in multi-author documents where the author's writing style shifts. This paper elaborates on the strategy of using a comparative learning method adopted in the PAN 24 shared task and demonstrates the research progress in handling author style changes by utilizing RoBERTa as a pre-trained model encoder. By integrating comparative learning into the training of the encoder, its ability to capture subtle features between text pairs is enhanced. Additionally, data augmentation techniques were employed to expand the training set, thereby enhancing the model's generalization capability. On the oficial dataset across three dificulty levels, our method achieved F1 scores of 0.985, 0.818, and 0.807, respectively, where two exceeded the baseline method using comparative learning from the previous year, and one was on par with it.
The task of multi-author writing style analysis requires participants to identify all changes in writing
style at the paragraph level within a given text. For each pair of consecutive paragraphs, the goal is to
assess whether there is a change in style between them [
To efectively tackle this challenge, traditional methods often rely on text features such as vocabulary,
syntax, and structure to capture the author’s writing style [
In PAN 24, the task focuses more on addressing intrinsic style change detection, rather than relying
too much on thematic information as a signal for style change [
Pre-trained Language Models (PLMs) are an important technology in the field of Natural Language
Processing (NLP). They learn rich linguistic representations by pre-training on large-scale text datasets
and can then be fine-tuned for specific downstream tasks to improve performance. In past style change
detection tasks, Zhang et al. [
However, the task of writing style change detection is not a semantic matching problem, but a
problem of capturing writing style features of diferent texts. Therefore, contrastive learning can be
used, which can efectively capture and compare stylistic diferences between diferent texts, thereby
identifying changes in the author’s writing style. In PAN 23, Chen et al. [
In this study, we employed the method of contrastive learning to optimize the training process of the RoBERTa [11] encoder. The goal is to utilize the cosine distance between two sentences to implement these text classification tasks. The core feature is to bring the embeddings of paragraphs written by the same author closer together in vector space, while pushing the embeddings of paragraphs by diferent authors further apart.
First, given a dataset , the method of contrastive learning can categorize the data into a set of positive examples and a set of negative examples, where the set of positive examples is defined as Ω and the set of negative examples is defined as Ω . We aim to have the embeddings of the samples in the Ω set be as close as possible in vector space, and conversely for the Ω set.
In the testing phase, we obtain the embeddings of the two paragraphs in the vector space of the paragraph pair to be tested, and map them through a linear layer to the classifier’s output space to complete the classification task.
Therefore, our model consists of three parts. The first part is the encoder block, where we use the RoBERTa-base model to encode the inputs Ω and Ω respectively. Next is the contrastive learning part, where during the training phase we apply to optimize the outputs of Ω and Ω from the encoder block. Finally, there is the classification part, where in the testing phase we use a connection layer to complete the classification. The details of the contrastive learning training are in section 2.1.
In the training stage, we convert the dataset into two sets containing positive and negative examples. These are defined as Ω = {1, 2, . . . , } and Ω = {1, 2, . . . , }, where Ω is the set of positive examples, with each representing a paragraph pair (, ) with an unchanging author’s style, and Ω is the set of negative examples, with each representing a paragraph pair (, ) with a changing author’s style.
Subsequently, each example is input into the encoder for encoding. For the positive examples in the paragraph pair (, ), we obtain the embedding representations u and u , and the same applies to the negative examples , in order to obtain the embedding representations u and u for the paragraph pair (, ). Then, the similarity between u and u is calculated using the cosine distance.
To quantify the diference in similarity between positive and negative examples, we define a loss function based on the comparison of cosine similarities. The loss function is represented as follows: = −
∑︁ (,)∈Ω,(,)∈Ω log (1 + · cos(u, u) − cos(u, u )) (1)
Here, Ω denotes the set of positive instance pairs, Ω denotes the set of negative instance pairs, is a hyperparameter greater than zero, used to balance the similarity diferences between positive and negative instances. u represents the embedding representation of paragraph x. This loss function encourages the model to reduce the cosine distance of positive pairs while increasing that of negative pairs.
Section 2.2’s Algorithm 1 provides a detailed description of the training process.
In the test stage, for a test paragraph pair x, we pass it through the encoder using the method from the training phase to obtain the embedding representation u. Then, we feed u into a fully connected linear layer for linear transformation, resulting in a real-valued vector B. Then, we apply the sigmoid activation function to convert it into a probability value that lies between 0 and 1. Paragraphs with a probability value greater than 0.5 are classified as positive examples, and those with a probability value less than 0.5 are classified as negative examples. These classifications are compared with the true labels, and the F1 score is used for evaluation.
The algorithm’s input includes: a set of positive instances Ω and a set of negative instances Ω as training data, is a hyperparameter greater than zero, used to balance the similarity diferences between positive and negative instances, the number of training cycles , and the encoder model that needs to be trained. The output is the trained model .
The intermediate variables include: paragraph pairs (, ) and (, ) from the sets of positive and negative instances, as well as their corresponding embedding representations (u, u ) and (u, u). The specific process is as follows:
PAN 24 provided three datasets of varying dificulty levels for this task, each dataset consists of a training set (70%), a validation set (15%), and a test set (15%).
Task 1 (easy): The paragraphs of the document cover various topics, allowing methods that utilize thematic information to detect changes in authorship.
Task 2 (medium): There is minimal thematic diversity within the document (though still present), forcing methods to focus more on style to efectively address the detection task.
Task 3 (hard): All paragraphs in the document are related to the same topic.
For each dificulty level of the dataset, the documents are first read, then divided into paragraph
pairs, and the labels indicating whether the author’s style has changed are recorded. Subsequently, we
obtain a large number of high-quality paragraph pairs for training the encoder based on a special data
augmentation method [
In this experiment, the hyperparameters for RoBERTa are set as follows: the batch size is set to 32, the maximum sequence length is set to 512, and any excess will be truncated. The initial learning rate is set to 1e-5, and the number of training epochs is set to 10.
The baseline method selects the test set scores of Chen et al. [
The model is submitted to TIRA [13] for execution to obtain the final metrics of the model. Table 1 provides the scores achieved by the model presented in this paper on the oficial test set. Table 2 demonstrates the scores of the method presented in this paper on the validation set.
In terms of experimental results, the RoBERTa model shows a significant improvement over the baseline method on Task 1 and Task 3, and it performs roughly on par with the baseline method on Task 2. This indicates that the RoBERTa model can exhibit superior performance when dealing with documents that have a high or low degree of topic diversity. The RoBERTa model is capable of adapting well to style transfer detection tasks of varying dificulties.
This paper briefly introduces the work results of finetuning the RoBERTa encoder and linear classifier using a contrastive learning method based on CoSENT, applied to the writing style change analysis in the PAN 2024 shared task. Comparative experiments were conducted on datasets of varying dificulty levels against baseline methods. The results indicate that the writing style change analysis model, which leverages RoBERTa encoding and contrastive learning, performs exceptionally well in the multi-author writing style analysis task. It can accurately identify shifts in author style, especially with the aid of contrastive learning and data augmentation techniques.
This research is supported by the Social Science Foundation of Guangdong Province, China(No.GD24CZY02) [11] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov,
Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019). [12] J. Bevendorf, X. B. Casals, B. Chulvi, D. Dementieva, A. Elnagar, D. Freitag, M. Fröbe, D. Korenčić, M. Mayerl, A. Mukherjee, et al., Overview of pan 2024: multi-author writing style analysis, multilingual text detoxification, oppositional thinking analysis, and generative ai authorship verification, in: European Conference on Information Retrieval, Springer, 2024, pp. 3–10. [13] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous integration for reproducible shared tasks with tira. io, in: European Conference on Information Retrieval, Springer, 2023, pp. 236–241.