Automatic subjectivity detection is a trending issue in Natural Language Processing. Responding to the challenge of finding models that are able to distinguish between objective and subjective segments, the CheckThat! Lab, which is part of CLEF 2025, invites in Task 1 to distinguish whether a sentence from a news article expresses the subjectivity of the author or not. The task has three settings: monolingual, multilingual and zero-shot. Our contribution focuses on monolingual classification in three of the proposed languages: English, Italian and German. The approach used is based on Transformers-based models. We have opted for BERT-base-uncased for English, BERT-base-italian-cased-sentiment for Italian and German BERT large for German. In our work, we have taken into account the lexical features, specifically the distribution of the diferent grammatical categories in each of the corpora. Despite the simplicity of our models, our results have been competitive, obtaining the second place in German, with a macro-F1 of 0.8280.
The semantic distinction between subjectivity and objectivity has traditionally been understood as a
dichotomy that distinguishes whether the author of a sentence appears immersed in the enunciation or
not [
For the objective of the present study, we understand subjectivity as statements that rely on personal opinions or emotions noticeable by the grammatical presence of an enunciator. The objective shall be to explore the application of some methods of advanced natural language processing that are still being developed in an attempt of improving the results of its functionality. This paper is an approximation to the tools that attempt to serve in the mentioned improvement of the techniques.
This work originates from the 2025 edition of the CheckThat! Lab [
Automatic subjectivity detection has been a fundamental topic in Natural Language Processing for years. Subjectivity detection, in particular, is an essential subtask of sentiment analysis because most polarity detection tools are optimized to distinguish between positive and negative text. Subjectivity detection, hence, ensures that factual information is filtered out and only opinionated information is passed on to the polarity classifier.
Early attempts to address the problem were based on the use of lexicons, dictionaries, and other
lexical resources that could detect words specifically related to subjectivity [
The need to create annotated corpora arose, especially with the emergence of machine learning
methods that would allow a supervised approach to the subject. Classical corpora include MPQA
(Multi-Perspective Question Answering)) [
With language models and large language models, all NLP tasks have undergone a major development, and subjectivity detection could not be diferent. Transformers are a technology where the main focus is in providing pre-trained models to reduce computational cost, reduce carbon emissions, and save time from training conventional models. BERT (Bidirectional Encoder Representations from Transformers) is one of these pre-trained models that will provide substantial output with lax parameters to detect subjectivity [15] [16]
In particular, the BERT-based models are being successfully applied to the task. Satapathy et al. [17] present a multi-task model for detecting and mutually supporting polarity and subjectivity detection. In 2024 CheckThat task, the teams that won in English [18], German [19], and Italian [20], applied BERT models to approach the problem.
In 2024 the JK PCIC UNAM [20] utilized Bert-based models focusing in two languages; English and Italian in news articles to explore whether sentences were written with tints of subjectivity or objectivity.
Our methodology does not include the use of LLMs, although the model has been shown to achieve competitive results [21].
The languages that we selected for this shared task were English, Italian, and German. The following analysis was made using the datasets for these languages.
First, we analyzed the label distribution (OBJ, SUBJ) to detect imbalanced classes, which could compromise model training.
As shown in Table 1, the German dataset exhibits a relatively balanced label distribution, whereas the Italian and English datasets display a significant imbalance, with a pronounced bias toward the OBJ class over SUBJ. This disparity may adversely afect model training, potentially leading to biased predictions.
After analyzing label distribution, we conducted a Part-of-Speech (POS) analysis to explore linguistic patterns across the datasets. In all cases, the "Others" category was predominant, accounting for over 50% of POS tags. This indicates a higher frequency of grammatical function words compared to lexical content (e.g., nouns, verbs).
Additionally, we observed a disparity in adverb usage: posts labeled as subjective contained a higher proportion of adverbs. This aligns with the linguistic function of adverbs, which often serve to express opinions, emotions, or personal perspectives.
For this task, we have decided to establish a comparison between the performance of traditional ML and Transformer-based methods. For this purpose, we have performed some tests with the classical classification algorithms and we have taken Logistic Regression, since it is the one that has given the best results. The LR has been used as a baseline to have a benchmark to compare the performance of the fine-tuned BERT-based models we have applied. We vectorized the text with bag-of-words, and used 3-grams and 4-grams. After some evaluations, we did not filter the stopwords
We maintain the protocol established last year [20], ensuring comparability between diferent editions. Our approach employed BERT (Bidirectional Encoder Representations from Transformers) as the primary architecture for subjectivity classification tasks. BERT models, being pre-trained on extensive text corpora, demonstrate particular efectiveness in sequence classification applications. For language-specific implementations, we utilized: BERT-base-uncased for English, BERT-base-italiancased-sentiment for Italian and German BERT large for German [22].
German BERT large is a language model for German, released in 2020 by the creators of the original German BERT and the dbmdz BERT. Pretrained on approximately 170 GB of text data, it leverages diverse linguistic sources, with the OSCAR corpus being one of its most significant training datasets [22]. Given OSCAR’s broad coverage of web-sourced texts, the model benefits from varied linguistic patterns, making it particularly suitable for this task.
The italian BERT sentiment model was pretrained by Neuraly AI from an instance of bert-base-italiancased, fine-tuned with a corpora of 45k tweets to perform sentiment analysis in italian. Regardless of the domain of the training dataset which was reported to be football, Neuraly AI claims eficiency in other topics which proved to be an accurate promise when the model was presented with the competition’s news datasets. [23]
BERT-base-uncased is also a pretrained model trained in a self-supervised manner with two objectives:; Masked language modeling (MLM) and Next Sentences Prediction (NSP). This way, the model learns an inner representation of the English language that can then be used to extract features useful for downstream tasks [15]. The main objective of our application is to classify pairs of sentences as either positive or negative based on selected parameters. In this case, we focus on subjectivity detection, using words with factual language as a starting point to categorize text as subjective or objective. This approach is also commonly applied in sentiment analysis.
All models were fine-tuned on the provided dataset, with hyperparameter optimization performed exclusively on the training set to maintain evaluation integrity. The fine-tuning process focused on adjusting the supervised classifier’s parameters, using: 4 training epochs, Batch size of 16, Maximum input length of 256 tokens.
Model performance was assessed using precision, recall, F1 score, and macro-averaged F1 score for each experimental condition. We prioritize macro-F1 as our primary optimization metric due to its robustness in addressing class imbalance issues inherent in subjectivity classification tasks.
All experiments were conducted in Google Colab using GPU acceleration to handle the computational demands of fine-tuning. 3.3.1. Model Comparison and Hyperparameter Optimization We conducted a systematic performance comparison between our baseline Logistic Regression (LR) model and BERT-based classifiers across all target languages (English, Italian, German). This dual-model approach served to establish transformer performance gains over classical methods as we can see in Table 3.
The Transformer results, which we obtained through systematic hyperparameter tuning on the organizers’ development datasets, are presented in Tables 4 (English), 5 (Italian), and 6 (German).
For German and Italian, we observed that reducing the batch size to 16 while increasing the maximum token length to 256 yielded superior performance. This improvement likely stems from the need to preserve complete linguistic structures in longer sentences and the risk of losing critical contextual information (and introducing bias) with shorter sequences.
In contrast, English achieved optimal results with shorter sequences (128 tokens) and smaller batches (16), suggesting diferent processing requirements for this language.
The strong performance of the German model may be attributed to the model’s pretrained datasets like OSCAR. While OSCAR’s web-sourced texts provide broad linguistic coverage, they may also introduce challenges due to potential misinformation and inaccuracies inherent in internet content.
We evaluated our models’ performance against the oficial shared task baseline using the organizers’ evaluation framework. The standardized scorer provided computes the following classification metrics: • Accuracy: • Macro-Precision (macro-P): • Macro-Recall (macro-R): • Macro-F1 (macro-F1):
Accuracy =
+ + + + macro-P = 1 ∑︁ , =1 macro-R = 1 ∑︁ , =1 where = where =
+
+ macro-F1 = 1 ∑︁ 1, =1 where 1 = 2
× × + • Class-specific metrics (e.g., SUBJ) : – Precision (SUBJ-P): – Recall (SUBJ-R): – F1-score (SUBJ-F1): SUBJ = SUBJ =
SUBJ SUBJ + SUBJ
SUBJ SUBJ + SUBJ
SUBJ × SUBJ 1SUBJ = 2 × SUBJ + SUBJ The following tables present our model’s performance against the oficial baseline for each target language. Our results demonstrate that even with a simple approach, we achieved performance above the organizers’ baselines. Key improvements are highlighted in the subsequent analysis.
For English (Table 8), our model achieved 14th place out of 24 teams, showing modest gains in accuracy (+1.27%) and recall (+0.57%) but significant challenges in subjective content detection (SUBJPrecision: -21.53%). This performance pattern suggests that while our simple approach generalized well for objective content, it struggled with English-specific subjective constructs, where more complex systems excelled.
Our Italian results secured us 9th place out of 15 teams, demonstrating remarkable improvements in precision-oriented metrics, particularly for SUBJ classification (+49.15% precision gain). While the recall decrease (-15.90%) indicates our model adopted a more conservative approach to subjective content detection—correctly identifying positives but potentially missing marginal cases—this precision-focused strategy proved efective in the competition context. The balanced performance across metrics (accuracy: +7.19%, macro-F1: +8.55%) suggests our approach successfully negotiated the trade-of between false positives and coverage.
In this research, simple models have been used for the task of identifying binary subjectivity in English, Italian, and German. After an exploratory analysis of the data, a logistic regression model was tested as a baseline.
Once the baseline was established, pretrained models such as BERT-base-uncased for English, BERTbase-italian-cased-sentiment for Italian, and German BERT for German were employed for analysis and classification. It was found that the Transformer-based models outperformed the traditional ones. Moreover, despite the simplicity of the technique used, the results obtained were competitive. Our best performance was second place in the German task.
In the future, we are plan to employ other strategies and techniques, including the treatment of corpus imbalance and the incorporation of linguistic elements related to subjectivity, such as sentiment analysis or the use of adjectives.
Additionally, we plan integrating reinforcement learning methods to the models to improve the performance of the algorithms.
I. Díaz thanks CONAHCYT scholarship program (CVU: 923309). This research was funded by UNAM, PAPIIT project IG400325.
During the preparation of this work, the authors used DeepL and Grammarly in order to: Grammar and spelling check and correct translation to English. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [15] K. L. K. T. Jacob Devlin, Ming-Wei Chang, Bert: Pre-training of deep bidirectional transformers for language understanding, Association for Computational Linguistics (2019). [16] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. [17] R. Satapathy, S. Pardeshi, E. Cambria, Polarity and subjectivity detection with multitask learning and bert embedding, 2022. URL: https://www.mdpi.com/1999-5903/14/7/191. doi:10.3390/ fi14070191. [18] M. Casanova, J. Chanson, B. Icard, G. Faye, G. Gadek, G. Gravier, P. Égré, Hybrinfox at checkthat! 2024–task 2: Enriching bert models with the expert system vago for subjectivity detection, arXiv preprint arXiv:2407.03770 (2024). [19] M. R. Biswas, A. T. Abir, W. Zaghouani, Nullpointer at checkthat! 2024: Identifying subjectivity from multilingual text sequence, CEUR Workshop Proceedings 3740 (2024) 361–368. [20] K. Salas-Jimenez, I. Díaz, , H. Gómez-Adorno, G. Bel-Enguix, G. Sierra, Jk_pcic_unam at checkthat! 2024: Analysis of subjectivity in news sentences using transformers-based models, CEUR Workshop Proceedings (2024). [21] M. Shokri, V. Sharma, E. Filatova, S. Jain, S. Levitan, Subjectivity detection in english news using large language models, in: Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, 2024, pp. 215–226. [22] B. Chan, S. Schweter, T. Möller, German’s next language model, Digital Library, Munich Digitization
Center (2020). [23] NeuralyIA, neuraly/bert-base-italian-cased-sentiment, https://huggingface.co/neuraly/ bert-base-italian-cased-sentiment, 2021. Accessed: 2024-05-24.