This paper introduces Telescope, a Machine-Generated Text (MGT) detection system developed for the IberAuTexTification challenge at IberLEF 2024. Our approach is an adaptation of Binoculars, a technique which involves the change ratio of perplexity and cross-perplexity using two closely related language models to quantify the level of surprise in word selection in the generation of a sentence for identifying MGT. This is supported by an iterative threshold selection process that balances false positives and false negatives. Enhancements include ifne-tuning of pretrained linguistic models to improve performance in the minority languages present in the Iberian Peninsula. This approach obtained the best performance measured in a random split of the training used for testing in all the Iberian languages. Results extrapolate to other contexts and generation profiles, with our models finishing third of all participants in the final contest. Telescope demonstrated its robustness and eficacy across diverse linguistic contexts, with significant improvements in MGT detection over Iberian languages. These results highlight the potential of Telescope for enhancing content moderation strategies.
Machine-Generated Text (MGT) refers to natural language text produced, extended, or modified by
machines, predominantly large language models (LLMs) [
This paper introduces Telescope, a system developed for the IberAuTexTification [
Telescope is an adaptation of Binoculars [
We employ open pretrained language models that achieve state-of-the-art results under optimal conditions; however, their performance degrades when applied to specific domains or languages. Given that this challenge involves languages from the Iberian Peninsula, some of which are considered minority languages, Telescope proposes to enhance detection accuracy through a fine-tuning phase to develop specialized models for defined language groups. This process involves adapting models to better comprehend the nuances and syntactic structures of minority languages, which are often underrepresented in mainstream datasets.
As the Binoculars approach provides an indicator of the likelihood of text being HWT, the selection of the decision threshold is critical to the performance of the detection system. To determine this threshold value, we employ an iterative process, experimenting with various threshold values to optimize the trade-of between false positives and false negatives, thereby achieving a higher accuracy rate in MGT identification.
By incorporating these advanced techniques and enhancements, we achieved third place in the Subtask_1 ranking, demonstrating the efectiveness of our approach across diverse linguistic contexts. The remainder of this paper is organized as follows:
Section 2 provides an overview of current approaches to detect generated text. Section 3 describes the dataset utilized in our approach. Section 4 details the methodology employed to detect MGT texts. Section 5 presents the evaluation and model selection process. Section 6 reports the obtained results. Section 7 discusses the implications of these results. Finally, Section 8 concludes the paper and suggests directions for future research.
Current approaches to detect machine-generated text can be categorized into several distinct groups:
• Machine Learning Classifiers : These methods employ traditional text classifiers trained on
labeled HWT and MGT. This approach can involve fine-tuning pretrained models such as BERT
or RoBERTa, or utilizing simpler techniques like logistic regression [
Catalan Basque Galician Portuguese English
The dataset employed in our approach comprises a diverse collection of texts in various languages from the Iberian Peninsula, provided by the IberAuTexTification shared task. The corpus encompasses texts in Spanish (ES), Galician (GL), Basque (EU), Catalan (CA), Portuguese (PT), and English (EN). In total, the dataset consists of 109, 663 texts released by the organization as training data.
We conducted a preliminary analysis of the dataset to ascertain the distribution of texts and the
average text length per language. Table 1 presents a comprehensive overview of the text distribution
and average text length for each language represented in the corpus. The languages have been detected
using langID [
The corpus was strategically partitioned into three segments. The largest segment, comprising 87, 730 texts, was utilized for model fine-tuning. Two smaller segments, each containing 10, 000 texts (hereafter referred to as 10), were employed for evaluation and testing purposes. Throughout this paper, these segments are designated as eval-10K and test-10K, respectively, to distinguish them from the authentic unlabeled test set released by the organization during the test phase of the IberAuTexTification task.
The evaluation of statistical metrics for the detection of MGT is predicated on measuring the presence of a statistical signature characteristic of both HWT and MGT. The underlying rationale is that language models generally adhere to a measurable probability distribution in their word selection during text generation. Specifically, perplexity is employed as a metric to quantify the degree of surprise generated when selecting words that compose a text, given its prior context.
A common assumption has been that when a language model analyzes the perplexity of a text, human-written content will yield higher scores than machine-generated text. Consequently, previous contributions have proposed identifying a threshold to diferentiate between human and machinegenerated text based on this measure. However, this assumption has been invalidated by the advent of larger, prompt-based models. Their outputs tend to exhibit perplexity metrics similar to humangenerated texts due to their enhanced expressiveness. As a result, false positives are not uncommon when analyzing perplexity in isolation.
Binoculars proposes an alternative approach [
1 ∑︁ () () = − =1 • Concurrently, cross-entropy is calculated (equation 2), measuring the degree of surprise in the text generation by the performer from the observer’s perspective.
1 ∑︁ () * () , () = − =1 • The ratio between these two metrics is calculated (equation 3) and compared to a decision threshold. We adhere to the same order of performer/observer models as in the oficial Binoculars implementation [18], that difers slightly from the paper notation of the same authors. (1) (2) (3) = , () =
() , ()
The rationale for employing a ratio rather than absolute baseline-perplexity and cross-perplexity values is to mitigate the impact of complex prompts on perplexity scores. It is anticipated that both metrics will be correlated in MGT, but not in human texts. This implies that human values are higher than MGT ones, justifying the use of the ratio with a decision threshold.
In the original work, a specialized model for English was utilized, the FALCON 7B model [
We propose two main improvements over the seminal Binoculars work. First, we introduce a finetuning step with only generative content to enhance system performance. Second, instead of using a single threshold to discriminate between human and generated content, we obtain multiple thresholds using non-random splits of the data. Consequently, diferent text sizes, languages, or contexts could potentially have diferent thresholds.
The adjustments implemented by Binoculars in MGT detection are related to the influence of the prompt on MGT perplexity values. Although Binoculars achieves state-of-the-art performance using pretrained models without the need for fine-tuning, this implies that the performance of these systems will be strictly related to the quality of pretraining of the selected models. This is why one of the most relevant drawbacks is the noticeable decrease in performance of this algorithm when dealing with diferent languages and domains, particularly with minority languages and specific domains.
Our proposed method utilizes the dataset presented in Section 3, separated by languages into groups
that allow language models to specialize in minority languages (Basque and Galician) and in another
group of more widely spoken languages such as Spanish, English, and Catalan, with a final group for
Portuguese. Specifically, we fine-tuned the following models for each language:
• Basque [Latxa-7B]: A family of LLMs designed for Basque with 7 to 70 billion parameters. The
model is based on Llama 2 and was trained on a corpus of 4.3M Basque documents. Latxa was
released by a team from the HiTZ Center [
We utilized the HuggingFace Transformers library [
Following the naming convention of Binoculars, we call our models Telescope, as they are more focused and powerful in specific scenarios but, in contrast, are less portable as they require extra models per language and computational steps for fine-tuning. To diferentiate from the baseline that is closer to the Binoculars strategy, we use this name only to refer to the combination that uses the fine-tuned models.
The Telescope strategy based on Binoculars allows us to obtain a numerical indicator to decide whether a text is HWT or MGT. However, as important as the calculation of the perplexity metrics is the determination of the threshold value to decide the boundary between values that will be classified as HWT or MGT. As mentioned in the methods section, the value of the B-ratio will increase if the text was generated by a human and decrease if it was written by machines.
The optimal threshold value for Binoculars is directly influenced by factors such as text length, communicative context, linguistic environment, and the combination of models selected as performer and observer, respectively. This is why correctly selecting the threshold value for Binoculars is crucial for accurately determining the classification of a text at its source of generation. As this is a learning process influenced by multiple factors, it is essential to search through a wide set of possible threshold values around their potential combinations.
Given a subset of data from the training corpus, the optimal threshold for this subset is obtained with the AUROC score, using the ground-truth labels. For example, Figure 1 and Figure 2 show the ROC curve for the Spanish texts in the training corpus. In these examples, the best threshold for this split is 0.76 and 0.90, respectively, using this combination of models.
Due to resource constraints, we limit the search for the optimum threshold to a small subset of text
size splits (with a maximum of 10 splits), fixing the split by language and the subset of models used.
To select the optimum threshold for the training corpus, we employed Optuna [
In this comparative analysis, we evaluate four model combinations. Two serve as zero-shot baselines: FLOR-1B3/FLOR-1B3-instruct and BLOOM-1B7/BLOOMZ-1B7. The third combination, which we designate as Telescope, utilizes the fine-tuned and base models, one model pair per language. At test time, language is detected to apply the corresponding model pair and threshold. Finally, we aggregate the scores of all models into a single output, termed Ensemble, assuming equal contributions from all combinations.
It is crucial to note that the Binoculars score is not symmetrical, and the designation of a model as performer or observer significantly impacts the result. Table 2 presents a preliminary study used to determine the optimal combination of models for Telescope. The results demonstrate that the ifne-tuned models consistently perform better as performers across all language and model combinations for this task.
For each case (excluding the baselines), thresholds are derived from the eval-10K dataset and applied to the test-10K dataset to compute the Macro-F1 score. The baselines, being zero-shot, utilize the training split and eval-10k to determine the optimal thresholds.
Table 3 illustrates the splits by text size and language, along with the optimal thresholds in the eval10k for the FLOR-1B3 and FLOR-1B3-instruct model combination. Similar analyses are conducted for the other model combinations, as shown in Tables 4 and 5. Notably, the optimal threshold varies considerably from smaller to larger text sizes, although no clear trend is discernible.
To evaluate the performance of the various model combinations and their respective thresholds, we employ the Macro-F1 score on the random split of the training corpus, designated as test-10K. Table 6 summarizes these results. Upon initial examination, a significant disparity in performance across diferent languages is evident. The majority languages (English and Spanish) demonstrate comparatively superior results relative to the minority languages, with Basque exhibiting the lowest score overall. Surprisingly, certain models that purportedly support a given language, such as BLOOM-1B7, yielded lower scores than other model combinations that do not explicitly support it, such as FLOR-1B3.
When a uniform threshold is applied across all languages, as in the original Binoculars paper, the system’s performance decreases substantially. Implementing language-specific thresholds and incorporating thresholds based on text length enhances the performance of every model pair configuration. Telescope demonstrates robust performance even with a shared threshold, at least when the training and test data share the same context and model generation techniques.
The utilization of fine-tuned models improves metrics across all languages. However, it is important to note that some biases may be introduced, as we are using generated data from a diferent split of the training corpus that follows the same distribution as the test data (e.g., context and methods of generation). Nevertheless, the fine-tuning step appears to be particularly beneficial for minority languages that lack substantial support in terms of the number of models and training data used to pretrain the multilingual models.
The selection of generative models for certain minority languages is constrained. Recent developments have seen the creation of models for Catalan and Galician based on the FLOR family of models (2023) and the release of Latxa for Basque (2024). However, multilingual models have not demonstrated optimal performance in these languages for text detection, likely due to the limited representation of these languages in the pretraining data. Indeed, models exclusively pretrained in a single language demonstrated superior performance in the majority of cases. Nevertheless, model selection was often driven by availability, as it was frequently the only option for a given language. Our implemented strategy requires two models whose tokenizers share the same vocabulary, further constraining the availability of models for the study.
A secondary limitation arose from the constrained computing resources available. All models (with the exception of Latxa) were fine-tuned on a single NVIDIA GeForce GTX 1080, with a hardware limit of 12GB of RAM. The performance of larger models using this strategy remains a prospective avenue for future research.
An intriguing question is the extent to which the implemented strategy generalizes to other contexts.
To address this, we analyze the system’s performance using the training data from Autextification 2023
[
The 2023 training data comprised three distinct contexts: tweets (orange), how-to guides (red), and legal texts (green). These contexts are represented in Figure 3. The clusters were obtained through a three-step pipeline: sentence embeddings vectorization with the all-MiniLM-L12-v2 model, UMAP for dimensionality reduction, and K-Means with = 3.
We compare the performance of the FLOR-1B3 baseline with that of the fine-tuned models across the three clusters using the area under the receiver operating characteristic curve (AUROC). Table 7 summarizes these results. It is evident that the baseline performs poorly in detecting legal and formal texts, despite these texts being longer in length. This is likely due to the inherent dificulty in detecting these texts using this strategy. The baseline zero-shot model performs no better than random, and the fine-tuned Telescope is marginally better but still far from optimal. As observed, fine-tuning the model with generated data appears to be beneficial in all contexts, despite the fact that the training data were obtained from diferent sources, topics, and generation strategies.
We conducted a similar analysis with the 2024 data but did not observe substantial gains or drops in performance across the clusters. Consequently, the threshold was calculated only per language and text length.
Generally, strategies for detecting MGT text tend to exhibit a positive correlation between accuracy and text length, a trend we also observe in our proposal. Figure 4 illustrates the AUROC calculated at text length intervals of 25 tokens for the Spanish and English records using the FLOR-1B3/FLOR-1B3-instruct model pairs.
A slight performance drop is observed between 100-300 tokens, which we hypothesize may be attributable to factors other than text size (e.g., the context of the texts or the generation strategy). For texts with fewer than 100 tokens, the proposed system operates far from its optimal working region. This is logical, as this strategy relies on metrics that are averaged over the text length (such as entropy and cross-entropy).
This paper presents and evaluates our Telescope approach for MGT detection across multiple languages and domains, with a particular focus on languages of the Iberian Peninsula. This work was conducted within the context of the IberAuTexTification task for the IberLEF Campaign, whose primary objective is to develop models that exploit linguistic form and meaning cues to identify automatically generated texts from a wide variety of models, domains, and languages.
Our primary contributions are threefold: i) Development of the Telescope approach for MGT detection based on the statistical signatures that current language models impart on the texts they generate; ii) Evaluation of the system’s efectiveness across various contexts; iii) Analysis of the influence of factors such as text length and linguistic context on system performance.
These objectives were largely achieved, albeit with certain limitations and constraints detailed below. The Telescope system demonstrated eficacy in detecting generated texts across several languages, achieving competitive performance and securing third place in the IberAutextification 2024 competition.
To implement our approach, we fine-tuned a set of pretrained linguistic models for each language in the dataset. We then selected optimal thresholds for the scores based on a split of the language variety and text sizes using the eval-10K dataset. Finally, we evaluated the performance of diferent model combinations and threshold selection strategies on the test-10K dataset.
The results revealed notable variations in system performance according to language and text size. While Spanish and English texts exhibited satisfactory performance, Basque texts presented the greatest challenges. This was attributed to the limited representation of this language in training corpora, coupled with its linguistic dissimilarity to the other studied languages. These findings underscore the limitations of multilingual models in minority languages that difer significantly in structure from more widely spoken languages. Furthermore, system performance generally improved with longer texts. However, performance drops were identified for text sizes between 100 and 300 tokens, which may be attributed to contextual factors rather than text size itself, afecting measures of surprise such as perplexity.
The most significant limitation was the limited availability of high-quality generative models for minority languages such as Catalan, Galician, and Basque. For these languages, often only one robust option was available, negatively impacting our ability to train and tune models for optimal performance or to conduct comprehensive benchmarking of minority language models. It should be noted that model selection is one of the factors afecting Binoculars performance. Additionally, computational resource constraints prevented us from exploring larger models, more complex architectures, or extending the range of tests for threshold values. This limited our training to a single NVIDIA GeForce GTX 1080 GPU.
We identify three key areas for future research: i) Augmentation of robust models for minority languages and expansion of training datasets to improve representation and performance in these languages. ii) Exploration of more advanced hardware and optimization techniques to enable the training of larger models. iii) Further investigation into how diferent contexts and textual domains afect system performance to improve generalization results to new areas.
The generalization of the implemented strategy to other contexts is limited and influenced by the aforementioned factors. Therefore, these three areas represent critical avenues for future research. Nevertheless, Telescope represents a significant contribution to the field of MGT detection, having yielded successful results within a shared task in the IberLEF campaign.