Generative AI detection has been of interest for at least the past decade, but especially since the emergency of transformer powered LLMs. This paper treats the task as a binary classification problem, where ∈ [0, 1]. I chose to use a traditional Support Vector Classifier (SVC) with sets of features chosen from examination of the training data set to determine what features human authors, as opposed to AI, are more likely to employ. I found the top 40 unigram and bigrams, along with the top 15 punctuation features, to be the most informative. When combined and input into my SVC, I achieved a mean (the mean of all scores used for the task) score of 94.89.
In terms of impressionistic feature diferences between human and A.I. authors, it has been stated
that overall LLMs are more focused (that is to say, they never leave the subject matter at hand), more
objective and highly formal. In contrast, their human counterpoints employ more subjective language,
less formal and demonstrate an increased propensity to stray from the topic. Linguistically, humans
employ less nouns and conjunctions and LLMs employ less punctuation and adverbs. Dependency
relations for humans are also much shorter. Lastly, humans show more ‘creativity in terms of word
choices’, therefore human texts on average show a higher type to token ratio. [
In terms of models, [
The present task involves a binary classification task, whereby documents in the dataset are classified as either being human authored or machine authored. I experiment with systems that return both binary labels and probabilities, where scores approaching 0 indicate probable human authorship and scores approaching 1 indicate probable machine authorship. Scores of 0.5 indicate that the system is unsure.
The training set used had a total of 23,707 documents while the validation set had a total of 3,589. The human class was lower for both splits, with it representing roughly two fifths of training set and roughly a third of the validation set. Various models were used to create the machine documents. Table 1 contains a more detailed summary.
As stated in section 2, my goal for this project was specifically to not use a transformer model. I wanted to use a model that could be easily employed on more traditional, less computationally expensive model. SVC’s and SVM’s have both proved to be successful in the domain of authorship attribution, so I chose to use an SVC as my model. In terms of hyperparameters, I experimented with diferent kernels and class weights. I found RBF to be the most dynamic kernel and ‘balancing’ (see equation 1: is the class weight, is the number of documents, is the number of class and () is the bin count of the classes).
= () (1)
My initial approach to feature selection was examining the dataset for anomalies. I wanted to first analyze the claim that human texts haver higher type to token ratios. Figure 1 shows a density plot for the training dataset of type to token ratios. This graph shows that although the human texts show a normally distributed curve and the machine generated text is much more irregular distributed, there is a lot of overlap, meaning this feature probably would not be helpful for classification. Next, I examined the top unigrams, bigrams and trigrams for both classes, to see if certain n-grams were more common in one class. I came to the conclusion that the top forty unigrams and bigrams were the most informative in distinguishing between the two classes. After this, since the literature had stated AI uses less punctuation, I considered of using this as a feature for the SVC. After careful examination of the dataset, the first 15 punctuation patterns proved to be the most informative. A summary of the features used for my model can be found in Table 2.
To evaluate the performance of the model, the metrics used by the task were: • AUC (a.k.a. AUC Roc score): The area under the curve score. • C@1 (a.k.a. Classification at 1): The percentage of instances where the top score was the correct one. • F0.5: The harmonic mean of precision and recall, with = 0.5. • F1: The harmonic mean of precision and recall, with = 1. • Brier: The complement of the Brier Score Loss, which is the mean squared diference between the predicted class and actual class. • Mean: The arithmetic mean of all previous scores.
All metrics naturally produce a score between 0 and 1 and are multiplied by 100.
To obtain my results, I experimented with diferent numbers of features. While experimenting with just unigrams and bigrams, I ran models for up to 1,000 count features. My conclusion was that only the top 40 impacted the document class. For the punctuation features, I found there to be 153 unique punctuation patterns within the dataset. I ran experiments with varying numbers of these patterns and found the top 15 to be the most distinctive. My hypothesis was then that combining the features would yield an improvement in the model; I was proven correct. The best model I was able to train was with the top 15 punctuation features and the top 40 unigram and bigram features. Figure 2 shows a PCA of the SVM’s feature space. Table 3 shows my final results on the validation set and table 4 shows my results on the final test set compared to the other baselines. 86.47 93.53 95.49
F1 89.36 94.02 96.09
Brier 85.48 92.23 94.90
Mean 84.97 92.42 94.89
With this short paper, I have shown that feature selection is still an efective method in AI detection. LLM models clearly still produce text that over sample certain words, as found to the case by [21, 20]. In addition, punctuation patterns prove to be a distinguishing factor: humans use wider ranges of punctuation patterns and use them with higher frequency. I have been able to demonstrate all of this without finetune a State-of-the-Art transformer model, with experiments I have run on my local machine
F0.5u
Mean (when doing feature selection, I did at certain points have to use my university’s super computer1, however one feature selection was finished training for my models required less than a minute). While transformer models may still yield higher performance than the model I present in this paper, my work serves as a reminder that feature selection is still a powerful method within the domain of AI text detection.
My work, is of course, not without its limitations. I could have done more hyperparameter fine-tuning to possibly improve my model even more. I also could have carried a more profound analysis of what features were actually relevant in distinguishing the two classes. The top baseline for this task reports TF-IDF features as having achieved a mean of 0.98; while I experimented with TF-IDF features, my ifndings were that they did not outperform raw count features. Future work should focus upon creating more robust datasets that not only contain a diversity of diferent models, but also a diversity of sampling models, as demonstrated by [20] to be an important factor in AI detection.
During the preparation of this work, the author used not a single AI tool for any purpose. 1I acknowledge the Indiana University Pervasive Technology Institute for providing supercomputing and storage resources that have contributed to the research results reported within this paper [22] USA, 2018, pp. 2814–2822. URL: https://aclanthology.org/C18-1238/. [13] E. Ferracane, S. Wang, R. Mooney, Leveraging discourse information efectively for authorship attribution, in: G. Kondrak, T. Watanabe (Eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Asian Federation of Natural Language Processing, Taipei, Taiwan, 2017, pp. 584–593. URL: https://aclanthology.org/I17-1059/. [14] Y. Seroussi, I. Zukerman, F. Bohnert, Authorship attribution with Latent Dirichlet Allocation, in: S. Goldwater, C. Manning (Eds.), Proceedings of the Fifteenth Conference on Computational Natural Language Learning, Association for Computational Linguistics, Portland, Oregon, USA, 2011, pp. 181–189. URL: https://aclanthology.org/W11-0321/. [15] Y. Seroussi, F. Bohnert, I. Zukerman, Authorship attribution with author-aware topic models, in: H. Li, C.-Y. Lin, M. Osborne, G. G. Lee, J. C. Park (Eds.), Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Jeju Island, Korea, 2012, pp. 264–269. URL: https://aclanthology. org/P12-2052/. [16] Y. Seroussi, I. Zukerman, F. Bohnert, Authorship attribution with topic models, Computational Linguistics 40 (2014) 269–310. URL: https://aclanthology.org/J14-2003/. doi:10.1162/COLI_a_ 00173. [17] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023, pp. 236–241. [18] J. Gu, K. Cho, V. O. Li, Trainable greedy decoding for neural machine translation, in: M. Palmer, R. Hwa, S. Riedel (Eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 1968–1978. URL: https://aclanthology.org/D17-1210/. doi:10.18653/v1/D17-1210. [19] A. Fan, M. Lewis, Y. Dauphin, Hierarchical neural story generation, in: I. Gurevych, Y. Miyao (Eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Melbourne, Australia, 2018, pp. 889–898. URL: https://aclanthology.org/P18-1082. doi:10.18653/v1/P18-1082. [20] I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-Voss, J. Wu, A. Radford, G. Krueger, J. W.
Kim, S. Kreps, M. McCain, A. Newhouse, J. Blazakis, K. McGufie, J. Wang, Release strategies and the social impacts of language models, 2019. arXiv:1908.09203. [21] S. Gehrmann, H. Strobelt, A. Rush, GLTR: Statistical detection and visualization of generated text, in: M. R. Costa-jussà, E. Alfonseca (Eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Association for Computational Linguistics, Florence, Italy, 2019, pp. 111–116. URL: https://aclanthology.org/P19-3019/. doi:10. 18653/v1/P19-3019. [22] C. A. Stewart, V. Welch, B. Plale, G. Fox, M. Pierce, T. Sterling, Indiana University Pervasive Technology Institute, Technical Report, Indiana University, 2017. URL: https://doi.org/10.5967/ K8G44NGB. doi:10.5967/K8G44NGB.