Authorship verification can be applied to plagiarism detection, paper duplication checking, etc. Authorship verification is a category of text classification tasks, which can be regarded as a fundamental problem of NLP, that is, to determine whether the same author writes two texts. The work presented in this paper was developed as a solution for the Authorship verification task of the competition PAN@CLEF 2022. Unlike the previous year, this year's task dataset contains four types of text: essay, text_message, email, and memo.
Different from the previous year, the overall distribution of this year's texts tends to be short. We try to blend the Pre-trained model Bert, which has performed well in natural language processing, with TextCNN, which is suitable for dealing with a short text. Our work proposes an authorship verification model based on Pre-training and TextCNN. Firstly, we used a pre-trained model to initially extract the author's writing style features of the text. Secondly, we used the TextCNN to extract the texts' style features and complete the authorship verification task. Meanwhile, we propose a data reorganization method to obtain a large amount of training data. The model is trained through a more enormous amount of training data to improve the judgment ability of the model further.
The PAN@CLEF 2022 Authorship verification task dataset contains 12,264 text pairs and labels. The meaning of the label is whether the two texts are written by the same author. If it is, it will be 1. If not, it will be 0. This year's texts consist of four types, essay, text_message, email, and memo. There are six combinations of text pairs, and their distribution types and quantities are shown in Table 1.
1 CLEF 2022 -Conference and Labs of the Evaluation Forum, September 5-8, 2022, Bologna, Italy EMAIL:oldsport996@gmail.com(A. 1); hanyong2005@fosu.edu.cn(A. 2)(*corresponding author); pengzeyang008@163.com(A. 3); mingjiehuang007@163.com(A. 4); kongleilei@fosu.edu.cn(A. 5); hanzhongyuan@fosu.edu.cn(A. 6); ORCID: 0000-0002-7369-7537 (A. 1); 0000-0002-9416-2398 (A. 2); 0000-0002-8605-4426 (A. 3); 0000-0002-0889-5027 (A. 4); 0000-0002-4636-3507(A. 5); 0000-0001-8960-9872(A. 6) This year, the authorship verification task focused on the scene of the open domain set and applied cross-text type recognition.
All the texts in the training data contain a total of 56 authors. Each author has written hundreds of texts, by collecting all the texts written by each author, recombining these texts and assigning labels to generate new training samples.
The texts with the same author label are placed in the same list, and a total of 56 lists are established. The texts in each list are connected in pairs to obtain training samples with a label of 1. The texts in different lists are combined in pairs to get training samples with a label of 0.
Suppose The texts of different authors are much more than the text pairs of the same author. When expanding the training data, we adopt different text pairing strategies. Firstly, we pair all the texts under each author to obtain samples of the same author. The first text of the same author is paired with the second text, and the second text is paired with the third text until the penultimate text is paired with the last text. Secondly, we pair the first text of the first author with the first text of the second author, the first text of the first author with the first text of the third author, until the first author and the first text is matched with the last The first text of one author is paired, then the first text of the second author is paired.The first text of the third author and so on, until the first text is matched in a loop. In the same circle, the second text of the first author is matched with the second text of the third author. For the text pairs generated by different author combinations, we apply the first 17 texts of each author to combine. For the specific combination strategy, please refer to the figure below. The figure below illustrates the process of matching text pairs with six authors. Before entering the data into the model, we first remove all emojis in the text. Secondly, we eliminate the angle brackets in the text and the messages in the angle brackets.
The structure of TextCNN is the same as that of CNN. The TextCNN model is a variant of the CNN model [4]. TextCNN uses one-dimensional convolution to obtain the n-gram feature representation.
We use BERT-Base, Cased:12-Layer, 768-hidden, 12-heads, 110M parameters. There are 12 Transformers in total.
Firstly we take out each layer of the CLS features of the 12 Transformer layers one by one and add them together to calculate the mean value. Secondly, we take out each layer of the Embedding vector of the 12-layer Transformer layer and send it to TextCNN one by one. After TextCNN outputs the features one after another, they will be added together to calculate their average value. Thirdly, we concatenate the averaged CLS features with the averaged TextCNN features. Finally, the concatenated result is fed into two fully connected layers to output the classified result. The Network architecture is shown in Figure 2.
Experiments and Results
In this work, we use Keras to construct the Network Architecture. We make use of a three-layer convolution network. Firstly, we set the features map to 256 and the convolution kernels to 3, 4, and 5 , respectively. Secondly, The convolution output of each layer is fed into a GolbalAveragePolling layer. Thirdly, the first and second convolution network layer is activated by ReLU. Finally, we concatenated the pooled outputs of each layer and sent the concatenated result to a layer with a dropout rate of 0.2.
The output vector corresponding to the CLS symbol is used as the semantic representation of the entire text. The head of the shallow transformer means the shallow semantic representation, and the head of the deep transformer means the deep semantic representation. The CNN network has advantages in processing short text classification. We intend to average the heads of each layer of transformers and then input them into the CNN network to extract text features further. This model is constructed according to this idea.
We obtained 41,192 new text pairs by data reorganization as the training set. At the same time, we used 11,000 text pairs from the official data as part of the training set and 1,264 text pairs as the validation set.
The maximum length of the token we send the combination of text1 and text2 to bert is set to 128. In the last part of the network, the first fully connected layer output hidden size is set to 320, and its activation is ReLU. The second fully connected layer output hidden size is set to 2, and its activation is Softmax. We train ten epochs with the model, and its optimization is Adam with a 2e-5 learning rate.
For the final classification, we set the classification score between 0.35 and 0.65 as 0.5 as a nonjudgment sample.
To verify the impact of data reorganization on model performance. We split all texts of 6 authors into a validation set on the official dataset. At this time, the text on the training set contains 50 authors, and the validation set includes six authors. Table 3 shows the final evaluation of the TIRA platform. Our model name is denoted as ye22.
In this work, we propose a method that data reorganization to augment training data and a model using Bert blend TextCNN features to solve the Authorship verification task in the PAN@CLEF 2022. This model has been experimented on the test dataset, and the result is AUC=0.542, c@1=0.526, f_05_u=0.461, F1=0.398, Brier=0.565, overall=0.499.
It can be seen from the results that our model does not perform well on the open domain authorship verification task, and in the follow-up work, we should try more efficient methods to extract the features of each text. In our work, the model with more training data leads to better performance, a breakthrough point our approach tries to improve this year.
This work is supported by the Social Science Foundation of Guangdong Province (No. GD20CTS02).