Text generated by Large Language Models (LLMs) is becoming less distinguishable from their human-written counterparts. Reliable detection of the diferences between the two is increasingly important to limit the spread of fake content, plagiarism and the manipulation of public opinion. We study the binary classification problem of distinguishing human-written from AI-generated text. We propose a two-step learning algorithm. In the first step, it calculates the correlation between the rows of the binary term-document matrix (TDM) and the binary labels associated with the documents. This step runs in (max + ) time, where is the number of texts, max is the maximum text length, and is the vocabulary size. In the second step, it uses these values to map any text to a sequence of correlations, which can be interpreted as a signal. This can be done in linear time () where is the size of the text. Together with other statistical measurements, this signal serves as a feature for standard machine learning algorithms. Furthermore, we give a perspective on the interpretability of our proposed approach for global and local (instance-level) explanations. Our work demonstrates that while large language models like RoBERTa remain state-of-the-art in terms of raw accuracy for AI-text identification, our interpretable and computationally eficient approach ofers a competitive alternative, particularly in scenarios where interpretability is important. We evaluate our approach within the Voight-Kampf Generative AI Detection task, which is part of the PAN lab at CLEF 2025.
To demonstrate the efectiveness of our approach, we use data from the Voight-Kampf Generative AI
Detection challenge, which is part of the PAN lab at CLEF 2025 [
Our contributions include: • A novel two-step learning algorithm that transforms text into a sequence of correlation values, interpretable as a signal. • A collection of global and local interpretations based on the output of the learning algorithm. • A simple approach to use hand-crafted linguistic features together with correlation signals, fed into a standard machine learning algorithm, to achieve competetive performance for distinguishing human-written from AI-generated text.
We formally introduce the problem setting and propose the concept of correlation signals as well as a simple way to use them for classification. We use binary term-document matrices and the Phi-coeficient as fundamental building blocks to obtain a word-correlation value for each word. We map the words in a given text to their respective word-correlation and call this sequence a correlation signal. We study the problem of distinguishing human-written from AI-generated text in a supervised binary classification setting. Let be the instance space, containing all possible texts, and let = {0, 1} denote the binary label space, where label 0 represents human-written text and label 1 AI-generated text. For training, we get a set of labeled training instances {(, )}=1 ⊆ × and try to find a function : → that correctly classifies unseen instances.
Let = {1, 2, . . . , } be the set of texts from the training data. We consider each text as a sequence of word tokens (1, 2, . . . , ), resulting from tokenization (cf. Section 4.3), and expand the notation of set inclusion to allow ∈ to denote that the word is contained at any position in the text . We define the vocabulary of as Vocab( ) = { | ∈ for all ∈ } and say that = |Vocab( )| is the number of words in the text corpus.
We construct a binary term-document matrix B from , where a row represents for a specific word the inclusion relation to each text from the training dataset.
Definition 1. A binary term-document matrix B ∈ {0, 1}× indicates at position B, whether a word ∈ Vocab( ) is contained in document for 1 ≤ ≤ and 1 ≤ ≤ :
B, = {︃1 if ∈
0 otherwise Given the ’th row B,· ∈ {0, 1} and the label vector y = (1, 2, . . . , ), we are interested in quantifying the predictive power that the occurrence of the word has (i.e. which label is more likely, after knowing that occurs in the text). For this, we calculate the correlation between these two vectors. defined as where Definition 2.
The Phi-coeficient
[
¯¯ ¯ = =1 1 ∑︁ , ¯ = 1 ∑︁ . =1 This leads to the definition of word-correlations. For a word , represented in the ’th row of the term-document matrix, we denote its word-correlation with the function () = (B,· , y), where y is the label vector. We further extend this notation to texts and say that text = (1, 2, . . . , ) is mapped to its correlation signal with
︁( ( ) = (1), (2), . . . , () ︁) calculation of the Phi-coeficient for each word individually takes For the given corpus of size | | = with a vocabulary of size |Vocab| = , let max be the length of the longest text. We do preprocessing of the training corpus in (max + ) time. Constructing the binary term-document matrix takes (max) time by reading through each text in (max) time and updating entries in the matrix corresponding to occurring words in (1) time. The subsequent () time and is done in cumulative () time. The preprocessing results in a associative datastructure of size (), that maps each word to its word-correlation. Given constant lookup in this datastructure (e.g. hash table), we only need () time to construct a correlation signal for a query text of size | | = .
Given the mapping from a text to its correlation signal, we define a classifier ( ) = {︃1 if |1 | ∑︀ 0 otherwise ∈ ( ) > for a given parameter . Intuitively, the average correlation signal acts as a soft decision boundary: if a text contains more words that tend to appear in AI-generated texts, its average correlation will be positive, and vice versa. The threshold determines the decision boundary in this latent correlation space. In practice the optimal decision threshold is chosen to minimize classification error for the given distribution of the training data (see Figure 1).
ifnal predictions.
This section gives a perspective on the interpretability of the proposed approach. Correlation signals are based on the word-correlations assigned to each individual word. This word level contribution ofers ways to analyze the underlying model on a global and local (instance-level) scale to explain the Globally, we can look at the magnitude of the correlations and see that AI models appear to avoid specific words more (strong negative correlation, min∈Vocab( ) () = − 0.4849) than they seem to
Human-written
Threshold
AI-generated 91.7% <
93.1% > 0.08 0.05
Signal Value 0.20 0.17 0.14 0.11 0.02 0.01 0.04 0.07 0.10 favor specific words (positive correlation, max∈Vocab( ) () = 0.3338). A list of tokens with the largest/smallest correlation scores is given in Table 7 in Appendix B. Furthermore, interpreting text as a correlation signal opens the door to more advanced analyses, such as spectral methods to investigate global patterns and structural trends (see Appendix C).
On the local scale, these scores can be used to explain individual instances, as the final output sum can be traced back to the specific token-level contributions at each point in the sequence. Predictions are constructed sequentially from the individual word-correlations in a text. This allows to pinpoint exactly the word or sub-sentence structure that lead to either predicted class. Given an appropriate threshold , we can see in Figure 2 how the models prediction changes from one class to the other as a result of words with an opposing word-correlation occurring.
We generalize our approach to -grams by treating them the same as simple word tokens. We calculate an -gram-correlation score analogous to word-correlations and build the final correlation signal as a sequence of such -gram-correlations.
Intuitively, we can capture more nuanced language interactions from the text by using -grams as they capture local contextual dependencies. However, -grams for > 1 are sparse. There is a total of 56987 tokens contained in the text corpus of the training data. Only 0.3% of the tokens in the validation set are not present during training. However, about 34% of the 2-grams and 84% of the 3-grams in the validation set have not been seen during training. This leads to poor generalization to unseen data, while the ability to find -gram-correlations that fit the training dataset improves with larger . (This efect explains the reduced performance of the corsig-2gram and corsig-3gram runs in Table 3.)
We evaluate the performance of correlation signal classifiers. There are two main objectives in our
experiments: (1) Determine the ability of our approach to generalize to new instances and (2) identify
if correlation signals contain additional predictive information, not contained in simple linguistic
measures. We will evaluate our approach on the dataset provided in the PAN Lab’s Voight-Kampf
Generative AI Detection challenge [
The experiments are implemented in Python and the code is available on GitHub1.
For both tasks of the Voight-Kampf Generative AI Detection challenge, separate datasets are provided. As shown in Table 1, the class distributions in the training and validation sets are relatively balanced for task 1. In contrast, task 2 shows significant imbalances, both across individual classes and between the training and validation splits. Specifically, in the training set, classes 3–5 together account for less than 10% of the data. This is even more prominent in the validation set, where classes 4–5 collectively represent only 1.01% of samples. The most significant inconsistency appears in class 3: while it comprises just 3.72% of the training data, it dominates the validation set with 51.16%. Such inconsistencies between training and validation distributions can severely impair controlled evaluation of model performance, as they lead to incorrect representations of the target data distribution during training.
We introduce a simple baseline classifier that takes several hand-crafted features into account. For
task 1, simple classification based on the respective optimal threshold of said features already achieves
a high performance that translates well from training to validation data (see Table 2). The features are
calculated separately for the train and validation set and then fed into any standard machine learning
algorithm (Random Forest, RF, in our case) to serve as a baseline. Additionally, we employ Facebook’s
RoBERTa base model [
To prepare the input data for processing into correlation signals, we first use a word-tokenizer that
is sensitive to punctuation for the English language. Subsequently, we employ the Porter stemming
algorithm [
For the RoBERTa baseline, we use the model specific tokenizer and do not further preprocess the inputs.
For task 1, we analyze six systems and present the evaluation metrics in Table 3. We run the statistical baseline with the name stats and the RoBERTa baseline as roberta. The systems corsig-<n>gram for ∈ {1, 2, 3} uses our main approach as presented in Section 2 as well as the extension to -grams from Section 3.2. Finally, the system stats-corsig is an adaptation to the statistical baseline, that uses the correlation signal |1 | ∑︀∈ () for each text ∈ as an additional feature. We can clearly see the negative efect -grams with > 1 have on the discriminative power of correlation signals, as we witness a slight decline in the performance metrics from corsig-1gram to corsig-2gram and a significantly more pronounced drop in performance when looking at corsig-3gram. The reason for this behaviour is the sparsity of -grams as explained in Section 3.2. Furthermore, system stats-corsig displays a substantial increase over the stats baseline. This indicates that correlation signals contain statistical information, not available from simple linguistic features. stats-corsig also shows that combined with correlation signals, a simple statistical baseline is suficient for competetive performance levels to the roberta baseline. 0.957 0.894 0.984
F1
For task 2, it is important to note that we are no longer dealing with binary classification, but rather a multi-class setting with six distinct classes. Consequently, our approach for creating correlation signals via a binary label vector y and classifying the summed up signals via , as introduced in Sections 2.2 and 2.3, no longer works. We define y = (1, 2, . . . , ) ∈ {0, 1, 2, 3, 4, 5} and build the correlation signals according to () = (B,· , y). Instead of using a threshold for classification, we use the RF classifier as described in Section 4.2, both with and without normalization |1 | ∑︀∈ () for each ∈ .
The results of our experiments on the validation-set can be seen in Table 4. The RoBERTa baseline (roberta) clearly outperformed the RF classifiers, both with ( stats-corsig) and without (stats) the correlation signal, which just slightly outperform guessing levels.
We hypothesize that the lack of performance can be attributed to an inconsistent class distribution between the training and validation sets, as described in Section 4.1. To verify this, we combined the original training and validation data and performed a new stratified split. The results on the new validation set confirm our assumptions, as we receive an F1-score of 96% via the RoBERTa baseline (roberta-strat). Additionally, we can now we observe a clear performance gain when using the correlation signal as a feature in the RF classifier ( stats-corsig-strat) compared to using baseline features alone (stats-strat). Nonetheless, the RF classifier still underperforms relative to the LM baseline, suggesting that our feature-based approach may be less efective for multi-class classification tasks.
In this work, we presented a lightweight and interpretable approach for distinguishing human-written from AI-generated text. Our method leverages the statistical correlation between individual words and class labels, encoding texts as correlation signals that can be processed eficiently and explained both globally and locally. We demonstrated that this signal-based representation achieves strong performance in the binary classification setting and adds complementary value when combined with standard statistical features.
In the multi-class classification setting, we observed that correlation signals alone may not capture the full complexity of mixed-authorship scenarios. However, they still ofer predictive gains when incorporated into classical models, provided that the data distribution is properly balanced. While language models like RoBERTa remain state-of-the-art in terms of raw accuracy, our findings show that interpretable, transparent, and computationally eficient methods can provide competitive alternatives—particularly when interpretability is a key concern.
In future work, we plan to introduce a relevance weight (e.g. tf-idf) for each word to calculate a weighted correlation signal, ensuring that more significant words impact the overall signal more. When removing stopwords, we already saw a performance improvement, which indicates that less relevant terms primarily add noise, hindering the prediction. Future work also includes extending correlation-based features to more fine-grained signals over richer linguistic representations (e.g., syntactic or semantic structures), and exploring hybrid models that combine the interpretability of correlation signals with the expressiveness of neural networks.
During the preparation of this work, we used ChatGPT to paraphrase and reword. After using this service, we reviewed and edited the content as needed and take full responsibility for the publication’s content.
The PAN Lab’s challenge organizers evaluated the submitted models from Task 1 on additional datasets. The test-set is a previously unknown part of the original dataset for competition purposes and the Eloquent dataset comes from a related competition, where participants are asked to generate text, such that it is indistinguishable from human text. We present the results in Tables 5 and 6. 0.916 0.933 0.575
F1
Since we are looking at texts in the form of signals (see Section 2.2), we hypothesize that there are certain structural diferences between human-written and AI-generated texts that can be uncovered by analyzing their frequency components. Specifically, let ( ) denote the real-valued correlation signal of the word at position of text . We interpret ( ) as a discrete-time process, which encodes some sort of evidence towards AI- or human-authorship. Our goal is to examine the power spectral density (PSD) for a text via the periodogram , which serves as a basic estimator for the PSD. is defined as where ∈ {0, 1, . . . , − 1} is the discrete frequency index and is the length of document .
To conduct spectral analysis, will will use Welch’s method, which segments the signal into overlapping
windows, applies a tapering function and finally averages the resulting periodograms. This method,
however, assumes stationarity of the signal, which means that the mean and variance do not change
over time; this is non-trivial for natural language. Similarly to [
After calculating for all ∈ , we average the values of these periodograms within each individual class; Figure 4 shows that there are indeed distinct diferences in the mean power density spectra of the correlation scores.
For task 1, we can see that both classes have a peak in the low-frequency range, which means that occurring patterns change slowly across the texts. In our context, this would indicate that the correlation scores remain mostly positive or negative over many words. This aligns with our expectation that human- and machine-authored segments typically span full sentences or paragraphs rather than just single words.
We see a similar trend in task 2. There are two large low-frequency peaks for human-initiated and machine-continued text as well as deeply-mixed texts, suggesting that machine- and human-authored parts are interleaved on the sentence- or paragraph-level. As expected, such a peak does not exist for fully human-written texts. Interestingly, we can see a minor peak at higher frequencies for the machine-written, then human-edited category. This could indicate that human editors made small local changes, such as modifying individual words or short phrases, rather than rewriting entire segments. Such finer-grained edits introduce higher-frequency peaks in the correlation signal.