In this paper, we address the Multi-Author Writing Style Analysis task for PAN 2024, which involves detecting style changes at paragraph levels within multi-author documents. To tackle this problem, we adopt the Entropybased Stability-Plasticity (ESP) method, which dynamically adjusts the learning rates of diferent neural network layers based on their entropy values. This approach efectively balances stability and plasticity, allowing the model to retain essential knowledge from previous tasks while eficiently learning new information, thereby mitigating catastrophic forgetting. Our experiments, conducted on datasets of varying dificulty levels (Easy, Medium, Hard), demonstrate that ESP significantly outperforms traditional methods in detecting writing style changes. The results highlight the efectiveness of ESP in leveraging prior knowledge and reducing interference between tasks, making it a robust framework for continuous learning in text analysis applications.
The style change detection task aims to identify text positions within a given multi-author document at
which the author switches[
The ability to continuously learn remains elusive for deep learning models, which cannot accumulate knowledge in their weights when learning new tasks. For the task Multi-Author Writing Style Analysis 2024 of PAN that has three diferent dificulty datasets: Easy, Medium, and Hard, we adopted a method called Entropy-based Stability-Plasticity (ESP) [10] for lifelong learning to address this problem, which can decide dynamically how much each model layer should be modified via a plasticity factor. The results show the robust framework of the approach in leveraging prior knowledge by reducing interference, ofering slight improvements over the baseline in maintaining performance across sequential tasks.
For the task of Multi-Author Writing Style Analysis, some studies employ the use of feature combinations such as lexical, syntactic and character features to analyze the variance in the styles of writing by diferent authors [ 11, 12, 13, 14]. Some of these methods rely on the analysis of the diferent stylometric features to detect stylistic changes in a document [15], while others adapted the outlier detection methods used in plagiarism detection problems. In addition some studies investigated the use of artificial neural networks to solve this problem [16, 15].
The task Multi-Author Writing Style Analysis 2024 of PAN is defined as, for a given text, finding all
positions of writing style change on the paragraph level (i.e., for each pair of consecutive paragraphs,
assess whether there was a style change) [
• Hard: All paragraphs in a document are on the same topic.
Artificial neural networks learn in a bounded environment, where the input distribution is assumed to be fixed. When the input distribution changes, the model must adapt its weights to perform correctly on the new task. Due to those modifications, the model overwrites previously learned patterns, creating interference between old and new tasks, causing a problem known as catastrophic forgetting[17, 18]. The lifelong learning methods to alleviate catastrophic forgetting can be categorized into three classes, based on how storing and using the task-specific information throughout the sequential learning process, the replay methods, the regularization-based methods, and the parameter isolation methods [19]. Vladimir Araujo proposes the ESP method, which relies on an entropy-based criterion to decide how much a model has to modify the weights in each of its layers, which performs well compared to the Stability version[10], and outperforms all baselines when trained on all experiments such as Replay.
The Entropy-Based Stability-Plasticity (ESP) [10] method for lifelong learning utilizes an entropy-based criterion to manage the trade-of between stability and plasticity. The plasticity factor is a crucial component in ESP, determining how much each layer of the neural network should be updated. This factor is computed using entropy to assess the importance of the parameters in each layer. The plasticity factor for the -th layer can be expressed as: where is the entropy of the -th layer and is the total number of layers.
The entropy for the -th layer is calculated based on the activations of the neurons in that layer. The entropy helps in identifying how much information is being processed by the layer. The entropy for the -th layer can be defined as: =
∑︀=1 = − ∑︁ log =1 where is the number of neurons in the -th layer and is the probability associated with the -th neuron’s activation.
ESP uses the plasticity factor to scale the gradients during backpropagation. This ensures that layers with higher entropy (more important) receive smaller updates, preserving their stability, while layers (1) (2) with lower entropy (less important) are updated more, enhancing plasticity. The gradient update for the -th layer is scaled by its plasticity factor : ∆ = · ∇ (3) where ∇ is the gradient of the loss with respect to the parameters of the -th layer. The ESP training process involves the following steps: 1. Forward Pass: Compute the output and activations of each layer. 2. Entropy Calculation: Calculate the entropy for each layer. 3. Plasticity Factor Calculation: Determine the plasticity factors based on the entropies. 4. Backward Pass: Scale the gradients using the plasticity factors and update the model parameters.
ESP efectively balances stability and plasticity by dynamically adjusting the learning rate for diferent layers based on their entropy. This method ensures that important layers (with high entropy) are preserved while less important layers (with low entropy) are more flexible to learn new tasks. These formulas and the underlying methodology provide a robust framework for lifelong learning, addressing the challenge of catastrophic forgetting while allowing the model to adapt to new information.
We use BERT [20] as the encoder. For the decoder, following the original BERT model, we use the ifrst token (special token [CLS]) of the sequence and a classifier to predict the class. Additionally, we use the default BERT vocabulary in our experiments. We utilize the Adam optimizer with a learning rate of 3 × 10− 5 and a training batch size of 32. To enhance the training for Task 2 and Task 3, we adjusted the training sequence of the data to hard → medium → easy.
Following the above experiment design, the results are as table 1 follows:
In this work, we addressed the Multi-Author Writing Style Analysis task for PAN 2024, which involves detecting style changes at paragraph levels in multi-author documents. The approach, Entropy-based Stability-Plasticity (ESP), efectively manages the trade-of between stability and plasticity in lifelong learning scenarios by dynamically adjusting the learning rates of diferent network layers based on their entropy values. This ensures that layers with high entropy, which are deemed more important, receive smaller updates to preserve stability, while layers with low entropy, considered less critical, are updated more to enhance plasticity. Our experiments utilized BERT as the encoder and demonstrated the efectiveness of ESP across diferent dificulty levels of datasets (Easy, Medium, Hard). The results showed that ESP outperforms traditional methods by efectively leveraging prior knowledge and reducing interference between tasks. Specifically, ESP’s adaptive gradient scaling mechanism allows the model to retain essential information from previous tasks while eficiently learning new tasks, thus mitigating the issue of catastrophic forgetting.
In conclusion, the ESP method provides a robust framework for continuous learning in multi-author writing style analysis, ofering slight improvements over the baseline in maintaining performance across sequential tasks. Future work may explore the integration of ESP with other neural architectures and its application to additional domains beyond text analysis. [6] C. Zuo, Y. Zhao, R. Banerjee, Style change detection with feed-forward neural networks., CLEF (Working Notes) 93 (2019). [7] S. Nath, Style change detection using siamese neural networks., in: CLEF (Working Notes), 2021, pp. 2073–2082. [8] P. Rosso, F. Rangel, M. Potthast, E. Stamatatos, M. Tschuggnall, B. Stein, Overview of pan’16: new challenges for authorship analysis: cross-genre profiling, clustering, diarization, and obfuscation, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 7th International Conference of the CLEF Association, CLEF 2016, Évora, Portugal, September 5-8, 2016, Proceedings 7, Springer, 2016, pp. 332–350. [9] E. Zangerle, M. Mayerl, M. Potthast, B. Stein, Overview of the style change detection task at pan 2020., CLEF (Working Notes) 93 (2020). [10] V. Araujo, J. Hurtado, A. Soto, M.-F. Moens, Entropy-based stability-plasticity for lifelong learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3721–3728. [11] E. Zangerle, M. Mayerl, M. Potthast, B. Stein, Overview of the Multi-Author Writing Style Analysis Task at PAN 2024, in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR-WS.org, 2021. [12] M. L. Brocardo, I. Traore, S. Saad, I. Woungang, Authorship verification for short messages using stylometry, in: 2013 International Conference on Computer, Information and Telecommunication Systems (CITS), IEEE, 2013, pp. 1–6. [13] M. P. Kuznetsov, A. Motrenko, R. Kuznetsova, V. V. Strijov, Methods for intrinsic plagiarism detection and author diarization., in: CLEF (Working notes), 2016, pp. 912–919. [14] K. Safin, R. Kuznetsova, Style breach detection with neural sentence embeddings., in: CLEF (Working Notes), 2017. [15] A. Rexha, M. Kröll, H. Ziak, R. Kern, Authorship identification of documents with high content similarity, Scientometrics 115 (2018) 223–237. [16] M. Kestemont, M. Tschuggnall, E. Stamatatos, W. Daelemans, G. Specht, B. Stein, M. Potthast, Overview of the author identification task at pan-2018: cross-domain authorship attribution and style change detection, in: Working Notes Papers of the CLEF 2018 Evaluation Labs. Avignon, France, September 10-14, 2018/Cappellato, Linda [edit.]; et al., 2018, pp. 1–25. [17] M. McCloskey, N. J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem, in: Psychology of learning and motivation, volume 24, Elsevier, 1989, pp. 109–165. [18] R. Ratclif, Connectionist models of recognition memory: constraints imposed by learning and forgetting functions., Psychological review 97 (1990) 285. [19] M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, T. Tuytelaars, A continual learning survey: Defying forgetting in classification tasks, IEEE transactions on pattern analysis and machine intelligence 44 (2021) 3366–3385. [20] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).