As large language models (LLMs) continue to advance at astonishing speeds and are increasingly widely adopted, it becomes increasingly difficult for people to discern whether a given text is written by a human or a machine. Authorship verification has become a crucial and challenging task. This paper employs a text classification model that combines BERT and Convolutional Neural Networks (CNNs) in order to leverage BERT's powerful contextual understanding capabilities and CNN's efficient local feature extraction abilities to enhance text classification performance. The introduction of CNN effectively compensates for BERT's shortcomings in extracting features at the phrase level, particularly in capturing local features in the text, such as capturing n-gram features. Experimental results demonstrate that our approach outperforms baseline models significantly, with improvements of up to 6% in the ROC-AUC metric and nearly 3% in the Mean metric. We thus validate the effectiveness of this approach.
average score of 0.734. These results significantly outperformed other baseline models such as Unsking and Fast DetectGPT, highlighting the enhanced ability of Bert_Transformer in distinguishing between human and machine-generated text. This success underscores the effectiveness of our approach in addressing the complexity of generative AI authorship verification.
In recent years, neural networks have seen extensive practice in text generation tasks,
encompassing rule-based systems, statistical methods, and more recently, deep learning
techniques. One of the most significant advancements has been the development of
Transformer-based models such as GPT and BERT . An approach [
BERT, proposed by Devlin et al.[
On the other hand, Convolutional Neural Networks (CNNs) are well-known for their ability to efficiently capture local patterns through convolution operations. Initially popularized in computer vision, CNNs have successfully adapted to text processing tasks, where they can effectively identify n-gram features. Some scholars[6] have reexamined CNNs' application in sentence classification tasks, particularly how fine-tuning pre-trained models and data augmentation techniques enhance performance. Others[7] have studied CNN performance across multiple text classification datasets, comparing the effects of different convolutional layer configurations and hyperparameter settings.
Combining the strengths of BERT and CNNs, recent studies[8][9][
This study utilized the "Generative AI Authorship Verification Task" dataset provided by PAN@CLEF, which is a publicly available dataset specifically designed for verifying the authorship of generated AI text. The PAN@CLEF dataset contains a large number of text samples generated by different algorithms as well as text samples written by humans. These texts cover various types and topics, including news articles, blog posts, comments, and more. The PAN@CLEF dataset is typically organized into a series of text pairs, with each pair accompanied by a label indicating whether they were written by the same author. The structure specifically includes text pairs and labels, and the file formats are as follows: {"id": "alpaca-7b/news-2021-01-01-2021-12-31-bideninauguration/art-022", "text": "..."} {"id": "alpaca-7b/news-2021-01-01-2021-12-31-bideninauguration/art-025", "text": "..."} The validation set provided by the "Generative AI Authorship Verification Task" in PAN@CLEF is a crucial component for testing and optimizing authorship verification models. Each text pair in the dataset is labeled as "0" or "1" to identify whether the texts are from the same author. The validation set is saved in JSONL format, with each line containing a text pair： {"id": "VHSN9BmKWQqLABGg", "text1": "...", "text2": "..."} {"id": "ywVXxXG12zUebGyiL8Q", "text1": "...", "text2": "..."}
In this study, the PAN@CLEF dataset was utilized to train and evaluate a hybrid model that combines BERT and CNN, with the aim of improving the accuracy of text authorship verification. Specifically, this dataset helped us systematically understand and identify the characteristics of text generated by different types of generators. Through a series of experiments on the PAN@CLEF dataset, we assessed the model's performance in distinguishing between machine-generated text and human text. We employed various evaluation metrics such as accuracy, recall, and F1 score to comprehensively analyze the model's effectiveness. The results indicate that our model can achieve satisfactory performance in handling this specific task.
In our research, we designed and implemented a hybrid neural network model that combines BERT and CNN to perform complex text classification tasks, specifically aimed at distinguishing between human-written and machine-generated text. The structure of this model is illustrated in Figure 1. This model architecture is intended to fully leverage BERT's deep semantic processing capabilities and CNN's local feature extraction abilities to enhance the model's performance in handling fine-grained text analysis tasks. We use the BERT-based uncased model from Hugging Face's Transformers library as our base pre-trained BERT layer. The model first processes the input text using the pre-trained BERT layer, extracting word embeddings rich in contextual information. BERT, serving as the foundational feature extractor, captures long-range dependencies in the text through its Transformer architecture. Subsequently, an attention mechanism layer is applied to the BERT output to enhance the focus and processing of critical parts of the text, thereby optimizing information flow to the subsequent layers. Multiple CNN layers then process these attention-weighted embeddings, utilizing convolutional kernels to extract local features of the text, such as n-gram patterns, which are particularly important for capturing the local semantics and style of the text. To improve the model's generalization ability on unseen data and prevent overfitting, a Dropout layer is introduced. For optimizing the model, we use a binary cross-entropy loss function to fine-tune the model's accuracy. Finally, a fully connected layer maps the output features from the CNN to classification results, determining whether a text is human-authored.
In this study, we utilized the pre-trained bert-base-uncased version of the BERT model as the foundation for text feature extraction. Due to its balanced performance and computational efficiency, it is suitable for complex text processing tasks under limited resources. We configured three sizes of convolutional kernels (3, 4, 5), each with 100 filters, to capture different lengths of n-gram features through multi-scale convolutional layers, thereby improving the model's sensitivity to local text patterns.During the training process, we chose a batch size of 8, a learning rate of 2e-5, and a total of 50 training epochs to ensure that the model could learn sufficiently and avoid overfitting. Additionally, we used the Adam optimizer, selected for its optimization effectiveness in training deep learning models, particularly in handling gradient sparsity and weight decay.To ensure the reproducibility of experiments, we set a fixed random seed, and all experiments were conducted in a computing environment equipped with NVIDIA GeForce GTX 1660 Ti and Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz.
The main experimental process consists of three stages: data preparation, model training, and performance evaluation.Firstly, in the data preparation stage, the dataset undergoes preprocessing, including text cleaning, tokenization, and encoding using the BERT tokenizer. Additionally, the data is randomly split into training and validation sets.In the model training stage, the model iteratively learns on the training set. At the end of each epoch, the model's performance is evaluated on the validation set to monitor for overfitting during the training process.Finally, in the performance evaluation stage, we use standard classification metrics such as accuracy, ROC-AUC, etc., to evaluate the model. Special attention is given to the model's performance on an independent test set to validate its generalization ability in real-world applications.Through this series of detailed and rigorous experimental procedures, we ensure the accuracy and practicality of the research results.
To comprehensively evaluate the performance of our proposed BCAV, we selected a series of metrics, including ROC-AUC, Brier score, C@1, F1, and F0.5u. These metrics not only reflect the overall performance of the model but also provide different perspectives on performance evaluation, helping us to understand the model's performance in specific aspects. Additionally, we calculate the arithmetic mean of these metrics to provide a single measure for comprehensive comparison of different models.
The specific performance metrics are as follows: in a classification task. It is calculated based on integration:
ROC-AUC measures the model's ability to distinguish between positive and negative samples Brier score is used to measure the accuracy of probability predictions. It is calculated as: ROC-AUC = ∫ TPR( ) (FPR( ))
Brier Score =
∑(fi − oi)2 1 0 1 =1 actual label of sample i (0 or 1).
C@1 is a metric used for evaluating classification tasks. It is calculated as:
N
N1 ∑i=1 I(ŷi = −1)∑jN=1 I(ŷj = yj))
N where N is the number of samples, is the predicted probability for sample i, and is the
where I is the indicator function that takes a value of 1 if the condition inside the parentheses is true, otherwise 0. ̂ is the predicted label for sample i, and is the actual label of sample F1 score is the harmonic mean of precision and recall. It is calculated as:
F1 = 2×Precision×Recall
Precision+Recall Where Precision=
TP TP+FP =
TP .
TP+FN F0.5u score is a weighted F score that emphasizes precision. It is calculated as: comprehensive performance score, which facilitates direct comparison between different （1） （2） （3） （4） （5） models to determine which model structure or parameter settings are more suitable for our application needs. This multi-metric evaluation approach ensures the comprehensiveness and reliability of the evaluation results, providing a solid foundation for the optimization and application of future models. Our evaluation results are shown in Table 1:
Approach Baseline Unmasking Baseline Fast-DetectGPT
BCAV
ROC-AUC
In this study, we proposed and implemented a hybrid neural network model combining BERT and CNN aimed at improving the performance of text classification tasks, particularly for authorship verification. The results indicate that the BCAV hybrid model proposed in this study achieves satisfactory results in text classification tasks. Its excellent performance across multiple evaluation metrics demonstrates the effectiveness of this model structure, providing valuable insights for future natural language processing tasks.
In future work, we will continue to refine our approach and strive for better results in authorship verification.
This work was supported by grants from the Guangdong-Foshan Joint Fund Project (No. 2022A1515140096) and Open Fund for Key Laboratory of Food Intelligent Manufacturing in Guangdong Province (No. GPKLIFM-KF-202305). [6]Carpuat M, de Marneffe M C, Meza-Ruiz I. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies[C]//Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022. [7]Lv C, Xu J, Zheng X. Spiking convolutional neural networks for text classification[C]//The
Eleventh International Conference on Learning Representations. 2022.
[8]Quoc Tran K, Trong Nguyen A, Hoang P G, et al. Vietnamese hate and offensive detection
using PhoBERT-CNN and social media streaming data[J]. Neural Computing and
Applications, 2023, 35(1): 573-594.
[9]Xiong G, Yan K, Zhou X. A distributed learning based sentiment analysis methods with Web
applications[J]. World Wide Web, 2022, 25(5): 1905-1922.
[