In recent years, linguistic computational models have advanced to the point where they can generate stories and hold conversations. Despite being a useful tool, computer generated texts can be used for unethical purposes, which underscores the need to identify these texts. This paper presents a model that classifies human- or computer-generated texts, using vocabulary richness metrics and POS label ratios to train a simple artificial neural network for spanish classification, and some others features to build a Naïve Bayes Model for english classification. The objective is to classify texts in both English and Spanish. The results show a Macro F1 of 0.67 for the texts in English and 0.6441 for the texts in Spanish. Therefore, the performance of this model is consistent with that achieved in previous research.
2023 Copyright for this paper by its authors. the proposed method and the associated theoretical concepts; Section 4 presents the results obtained and their analysis; and finally, Section 5 contains the conclusions.
TGM.
Gehrmann, Strobelt & Rush (2019) developed Giant language Model Test Room (GLTR), a computer-generated text detector based on text statistics. The central idea is that the generated texts are written based on a very limited set of language distributions. The tests use the probability of occurrence of a word given that other words are already written, the rank of a word, and the entropy of the distribution model predicted by the text generator. By calculating the rank and entropy of each text, they showed that these metrics are higher in human texts, thus obtaining a 72% hit rate in text detection [3].
for testing. They proposed a refined variant of the BERT linguistic model and obtained approximately 70% accuracy with prior knowledge of how GPT-2 generates texts. By using decoding strategies, which consist of selecting the next word to write based on probability distributions, they determined that knowing the generation process has a large impact on the performance of the classifier [4].
idea of watermarking TGMs without retraining them. The water-mark consists of the use of certain words that must appear when generating a text, appearance determined by existing probability models.
must have a much higher frequency of use to allow the detection of the watermark. Although this idea is subject to the weaknesses of the watermarking mechanisms of any other multimedia element, it is a potential idea that, applied correctly, may allow detection in the future [5].
by the same model. This is the case of Gritsay, Grabovoy & Chekhovic (2022), who developed a model based on ROBERTA for the similarity detection of texts generated by the same TGM GPT-2. The results show that a large number of tokens and very wide windows are required to improve the results, although this may result in overfitting if it is decided to analyze very wide neighborhoods for the text tokens. They report accuracy of up to 97% in detection [6].
3.1.
provided by Spacy to tag the texts in English. The tags that interest us are those of part of speech and the dependency. Once the tags have been created, we create n-grams of the tokenized words (not of the tags), in the end both the words, as well as the n-grams, and the tags are counted to see their occurrences by class and measure their frequency of occurrence. This information will be used to later calculate the probabilities of occurrence. In Fig. 1, this process is observed.
In Figure 2 there is an example of how the input to the naive bayes model would be, the final features selected were the frequencies of appearance of the following elements: • N-gram data: 3-gram words (combinations of 3 words in the text). • POS data: Individual tags for verbs (VB, VBN, VBD), adverbs (RB, RBR, RBS, WBR, RP), nouns (NNP), punctuation (LS) and spaces (_SP). • Dependency data: Index of token head dependency (the position with respect to a sentence of the token heads of the dependency relations).
identification based on these metrics is possible. The method proposed here is based on the idea that a
bigrams, it is expected that computational models will tend to use some type of bigram mostly. In the same case with the richness of the vocabulary, naturally the generated texts could have a lower than human richness, depending on the texts with which the model that writes them has been trained.
of a corpus of texts in English and one of texts in Spanish, both corpus with the instances labeled as human and generated by computer. The method is applied to the texts of both languages. In total, 20,129 texts in Spanish were used for tests and 21,832 in English, for training 33,845 texts in English and 32,062 in Spanish were used.
process each token is tagged within a morphosyntactic category such as noun, adjective, personal pronoun, etc. [11]. The purpose is to obtain information on the structure of the sentences that are present in the text. The SpaCy library for Python performs this procedure, recognizing 20 different tags. An example of POS tagging made in the parts-of-speech.info site can be seen in Figure 3
,
Where is the proportion of the i-th POS tag, the numerator is the number of occurrences of the ith POS tag, and the denominator is the total number of words in the text. For each of the i tags considered by the Spacy library, in this case 20, we get the first 20 features.
POS bigram ratio is also examined. Since we have 20 possible tags, when generating bigrams up to
400 different pairs can be established. The proportion of bigrams is obtained by the expression:
(
indicates the number of labeled bigrams i,j for each pair of consecutive words, and the denominator is the total bigrams of consecutive words. For a text with n words, there are n-1 bigrams, this procedure yields 400 additional features.
= Another useful measure is the indicator λ defined by [13]:
, , =
∑ =1 √( − +1)2 + 1 ,
and V is the number of types. With these metrics, two more features are added.
this type of bigrams. Collocations are multiple word expressions, usually bigrams, that often go together, but none of them can be changed to a synonym without losing their meaning. For example, "red wine" cannot be substituted for "reddish wine" [14]. The NLTK library for Python is capable of placing collocations in a text. Therefore, the ratio of collocations to the number of bigrams in the text is calculated:
In total, there are 423 features for each text, which are passed to a neural network for training as a
classifier. The Neural Network was an architecture Feed Forward with 2 hidden layers with 423 neurons
per layer, batch size of 200 elements, adaptive learning rate with value of 0.001 and momentum of 0.9,
and 500 epochs, the activation function was ReLu. The diagram of the method can be seen in Figure 4.
(
Remove punctuation
POS tagging
Remove stop words
POS proportion POS bigram proportion Calculate
STTR Calculate
Collocations proportion
Artificial Neural Network
split into training set with 80% of the data and test set with the remaining 20%, this was applied to both languajes. The final performance metric of the classifier is the Macro F1 Score, although precision and recall are also shown. To understand how it works, it is necessary to refer to the confusion matrix.
From where the F1 metric is calculated as a way of combining both results:
=
=
,
,
,
(
These metrics assume that we are forcing a class to be positive, however it is worth checking the F1 when the positive prediction is the human class and when it is the generated class, so the F1 Macro is used, it’s applied for multi class classification: ,
Now, the confusion matrices for the English texts are presented in Fig 6. It can be seen that there are more positive than negative instances, and the system has better identified negative instances than positive ones ty 9665 i l a e R 5385 (a)
ty 5257 i l a e R 1525 5385 9665 (b)
given in terms of accuracy, so a direct comparison with related works cannot be made. However, the overall results of Macro F1 indicate a modest performance that is close to the work done so far and without the need to modify deep networks. The classifier in English could result in a higher score because the SpaCy library has a more extensive corpus in that language, which could make POS tagging more appropriate than that done in Spanish.
Proposed Classifiers Languaje English Spanish English Spanish English Spanish English English Spanish English Spanish English
respect to Logistic Regression, which is the best baseline result, in addition to exceeding Symanto
of Logistic Regression and the rest of the baseline methods. In English, a notable improvement has been obtained with respect to the baseline classifiers, while in Spanish the result of the majority has been exceeded, being only surpassed by RoBERTa.
generated by TGM and those generated by humans. Preprocessing was carried out on the texts in
a series of statistical descriptors, including bigram usage and vocabulary richness, were determined and used to train a neural network. The results indicate that the model is moderately successful, reaching
with the human texts, but a better recall with the generated ones. Also, the F1 suggests better detection of generated texts.
with some reliability, although it is clear that there is a lot of room for improvement. Future works may include the use of additional features or the implementation of different artificial intelligence models.
further. In the same way, the use of other ways of text processing can be reviewed.
In general, it was shown that the use of statistical features instead of the use of pre-trained deep networks is feasible and can present better results with proper direction and methodology. The importance of artificial text detection will increase in the near future due to the rapid growth of generator models and the risks associated with their unethical use. This work represents an initial step in that direction.
[9] Sarvazyan, A. M., Gonzalez, J., Franco Salvador, M., Rangel, F., Chulvi, B., & Rosso, P. (2023).
[10] Müller, A. C., & Guido, S. (2016b). Introduction to Machine Learning with Python: A Guide for
[11] Sammut, C., & Webb, G. I. (2017). Encyclopedia of Machine Learning and Data Mining. Springer. [12] WordSmith Tools. (s. f.).
https://lexically.net/downloads/version5/HTML/index.html?type_to%20ken_ratio_pr%20oc.htm. [13] Fang, Y., & Liu, H. (2015). Comparison of vocabulary richness in two translated Honglou-meng.
[14] Bird, S., Klein, E., & Loper, E. (2009). Natural Language Processing with Python. O’Reilly Media.