We present an overview of Task 2 of the seventh edition of the CheckThat! lab at the 2024 iteration of the Conference and Labs of the Evaluation Forum (CLEF). The task focuses on subjectivity detection in news articles and was oered in ve languages: Arabic, Bulgarian, English, German, and Italian, as well as in a multilingual setting. The datasets for each language were carefully curated and annotated, comprising over 10,000 sentences from news articles. The task challenged participants to develop systems capable of distinguishing between subjective statements (reecting personal opinions or biases) and objective ones (presenting factual information) at the sentence level. A total of 15 teams participated in the task, submitting 36 valid runs across all language tracks. The participants used a variety of approaches, with transformer-based models being the most popular choice. Strategies included ne-tuning monolingual and multilingual models, and leveraging English models with automatic translation for the non-English datasets. Some teams also explored ensembles, feature engineering, and innovative techniques such as few-shot learning and in-context learning with large language models. The evaluation was based on macro-averaged F1 score. The results varied across languages, with the best performance achieved for Italian and German, followed by English. The Arabic track proved particularly challenging, with no team surpassing an F1 score of 0.50. This task contributes to the broader goal of enhancing the reliability of automated content analysis in the context of misinformation detection and fact-checking. The paper provides detailed insights into the datasets, participant approaches, and results, oering a benchmark for the current state of subjectivity detection across multiple languages.
The CheckThat! lab is organized for the 7th time within CLEF 2024. This paper presents an overview of Task 2 on detecting subjectivity in news articles; the task is organized for a second time. The ability to discern between subjective and objective content has become increasingly important in the domain of natural language processing (NLP), especially in the context of news articles during the last decades. The distinction between subjective statements, which reect personal opinions, beliefs, or biases, and objective statements, which present factual information, is fundamental for various applications such as sentiment analysis, opinion mining, and fact-checking. Task 2 aims to advance research in this area by providing a platform for the development and the evaluation of systems capable of subjectivity detection across multiple languages. Specically, Task 2 challenges participants to distinguish whether a given sentence from a news article expresses the subjective view of the author or presents an objective account of the topic. This binary classication task is crucial for improving the reliability and the transparency of information processing systems, particularly in an era where misinformation and biased reporting are prevalent.
The task is designed to be language-agnostic, with datasets provided in ve languages: Arabic, Bulgarian, English, German, and Italian. Additionally, a multilingual scenario is introduced in order to evaluate the robustness of the participating systems across diverse language contexts. The datasets have been carefully annotated to ensure a high quality and consistency, following prescriptive guidelines for language-agnostic subjectivity detection. We evaluated the participating systems based on macroaveraged F1 score, which balances precision and recall across the subjective and the objective classes, ensuring a fair assessment of system performance. The task’s comprehensive evaluation framework, coupled with the diverse linguistic datasets, provides a rigorous benchmark for advancing the state-ofthe-art in subjectivity detection.
In the rest of the paper, we oer an overview of Task 2, detailing the datasets, the evaluation measures, and the submission guidelines. We also present the results and the methodologies of the participating systems, highlighting the progress and the challenges when developing robust subjectivity detection models. By fostering collaboration and innovation in this critical area, Task 2 contributes to the broader goal of enhancing the reliability of automated content analysis in the digital age.
Subjectivity detection has been approached in dierent contexts and domains. Early work on the topic
was mainly conducted in the context of sentiment analysis [
Naturally, the denition of subjectivity varies with the task and the direction it is approached from.
Previous work has developed corpora by relying on dierent assumptions to detect subjectivity, such
as domain-specic heuristics based on keyword spotting [
. Ñk. AJÖÏ@ð áKYªJË@ ÈAm.× ú¯ éAg H@Qj. ®JÖÏ@ é«AJ ú¯ ÐñJKñÓB@ H@Q K ÉgYK AÒ» OBJ Думите на Тръмп са просто думи, докато тези на Обама означават война. SUBJ Аз се почувствах се глупаво, когато разбрах фактите. OBJ But the state’s budget is nothing like a credit card. SUBJ The plan incorporates cash payments supplemented by contingent contributions. OBJ Den Gru¨nen bleibt nur, immer wieder darauf hinzuweisen, dass sie selbst gerne ein SUBJ bisschen großzu¨giger wa¨ren -sich damit aber leider nicht durchsetzen ko¨nnen.
Mitte November ku¨ndigte die Ampel-Koalition an, das zu a¨ndern. OBJ Inoltre paragonare immagini di attori paparazzati per strada a foto di studio photo- SUBJ shoppate non ha senso.
Il presidente russo, Vladimir Putin, ha visitato Kaliningrad per incontrare gli studenti OBJ dell’Universita` Kant e tenere un incontro sullo sviluppo della regione.
Our data annotations cover dierent granularity levels, including sentence [
The task of subjectivity detection was also part of the 2023 edition of the CLEF CheckThat! lab [27]. The task was oered in 6 languages [20]: Arabic, Dutch, English, German, Italian, and Turkish.
The task oered datasets in ve dierent languages with a total of more than 10k sentences manually annotated following the guidelines in [19]. Table 1 presents details on the dataset statistics. Some sample instances for each language are given in Table 2.
The dataset consists of sentences from news articles, including sources such as AraFact [28]. The complete data collection and annotation process involved several phases. In the article selection phase, we selected 1,159 news articles from AraFact [28]. Additionally, we manually searched for opinionated articles from various Arabic news outlets, eventually selecting 221 articles. These articles were parsed and segmented, resulting in 15,947 sentences. During the sentence selection phase, we selected 4,524 sentences afier various ltering steps, where dierent existing classiers were used to classify the sentences into subjective and objective, and the sentences were selected based on their decisions.
Finally, in the sentence annotation phase, we annotated the sentences using in-house and Amazon Mechanical Turk (MTurk) annotators. We adhered to standard qualication tests and ensured majority agreement among 3 to 5 annotators. A label was selected for each sentence where at least two annotators agreed. The inter-annotator agreement (pair-wise Cohen’s kappa) was = 0.538. More details about the data collection and annotation process can be found in [29].
For the lab, we used all data from CheckThat! 2023 [20], specically using the training and the development datasets as the training and development sets, respectively, and the test data as the development-test set. Based on the data collection and annotation procedure discussed above and in [29], we developed a new dataset for testing.
The Bulgarian corpus is based on a fake news dataset [30], provided by the Bulgarian Association of PR Agencies1. It was annotated by Bulgarian students of journalism and further cleaned and deduplicated by the authors. The cleaning process consisted of removing Unicode symbols and URL encoding schemes. Afierwards, we segmented the articles into sentences, and we ltered out all sentences shorter than 5 tokens. We then selected a total of 1,293 random sentences for annotation. The annotation was conducted by three native Bulgarian speakers: two annotators and a consolidator, according to the guidelines of [19]. The process consisted of multiple annotation batches, each of size 200. Afier each batch was annotated, we accepted the examples with matching labels, and we sent all conicting annotations for analysis and discussion between the annotators. In cases of disagreement afier these initial discussions, the consolidator was involved to help reach a decision. We measured Krippendor’s Alpha for the inter-annotator agreement for every batch, and it ranged from 0.6 (worse batches, lots of discussions until labels are agreed upon for every sentence) to 0.85 (good batches, very few disputes).
The English dataset is partially based on NewsSD-ENG [19], a corpus of 1,049 sentences labeled by seven annotators following guidelines for subjectivity detection tailored to an information retrieval setting [31]. We used 830 examples for training and 219 for development. The Krippendor’s alpha [32] measuring the inter-annotator agreement on the whole corpus is 0.83. The test partition of the CheckThat! lab 2023 Task 2 edition [20] is used as a dev-test set. We further developed a new test set following the same data collection methodology for NewsSD-ENG, which comprises several stages to ensure high quality. First, we sampled 15 news articles on controversial topics, with a total of 490 sentences. Then, eight annotators labeled the sentences as subjective or objective. Each sentence was annotated by at least two annotators. Then, the annotators engaged in a discussion phase to resolve any disagreements. The unresolved conicts were addressed by a third annotator who decided on the nal label. The inter-annotator agreement on the new test set measured with Krippendor’s alpha was 0.86.
The German dataset was assembled by randomly selecting sentences from the CT 2022 FAN-Corpus [33] consisting of news articles originally annotated according to the factuality of their main claim. The 800 manually annotated sentences for training and the 200 instances for development are from the 2023 edition of the task [20]. As an additional development set (dev-test), the 2023 test data is used. For the test set, 360 new sentences were randomly sampled from the CT!2022FAN-Corpus (instances not included in the other partitions). They were annotated following the guidelines outlined in [19]. We excluded all incomplete sentences as well as non German ones. We also reduced instances consisting of more than one sentence due to wrong sentence splitting to one sentence. Each sentence has been annotated by three native speakers, all co-authors of this paper. The annotators achieved a substantial agreement [34], with Fleiss’ = 0.696 ( < 0.0001, = 22.1). The nal label was obtained by majority voting.
The dataset is based on the SubjectivITA corpus [23], re-annotated using the guidelines introduced in Antici et al. [19]. The corpus contains 1,841 sentences split into 1,613 for training and 227 for development. The inter-annotator agreement measured with Fleiss’ [34] is 0.65. A novel test set is dened following the same methodology used for the English dataset. In particular, we collected 513 sentences from 14 news articles targeting controversial topics. Eight annotators labeled each sentence as subjective or objective. The inter-annotator agreement on the new test set measured with Krippendor’s alpha [32] is 0.79.
Fifieen teams participated in this task, submitting 36 valid runs. Seven teams submitted valid runs for more than one language, with three teams participating in all six language settings, including the multilingual one. All teams participated in the English subtask.
Table 3 shows the results achieved by the individual teams for each language. At least two teams improved over the baseline for all languages, except for Bulgarian. Notable performance was achieved for Italian and German, followed by English, with scores above 0.70. For Arabic, none of the teams achieved a macro-F1 score above 0.50. The team with the most stable results across all languages was Nullpointer [35]: with the exception of the English subtask, they always ranked among the top-3 teams. All teams used neural networks, with transformer-based models being the most frequent choice. Some teams used language-specic monolingual transformer models, others chose multilingual models and Checker Hacker [36] 4 § § CLaC [37] 14 § § Eevvgg [38] 8 § FactFinders 7 § HYBRINFOX [39] 1 6 3 1 4 2 § § § IAI Group [40] 3 1 4 15 2 5 § § Indigo [41] 10 § § JK_PCIC_UNAM [42] 5 1 § JUNLP 7 5 13 § § Nullpointer [35] - 2 2 1 9 3 § SemanticCuetSync [43] 4 12 § § SINAI 6 § SSN-NLP [44] 3 § Tonirodriguez [45] 5 2 § § § § Vigilantes 8 § - The run was submitted aer the oicial deadline, therefore not part of the oicial ranking. § § § § § § §
For all languages, our baseline was a multilingual SentenceBERT [46] model with a logistic regression classier on top of it. In particular, we consider paraphrase-multilingual-MiniLM-L12-v2 model card as one of the current top-performing models for semantic similarity. We regularized the logistic regression classier by applying class re-weighting to account for class imbalance. We trained the baseline model on individual language-specic training data and we evaluated it on the corresponding test set.
Multilingual Three teams submitted runs to the multilingual subtask with two beating the baseline. Team HYBRINFOX [39] ranked rst with their hybrid approach combining an ensemble of RoBERTa [47] and SentenceBERT [46] with the rule-based-expert-system VAGO [48]. The late submission by team Nullpointer [35] achieved a better score by almost 3%-points using a system tailored to English and translating all non-English instances using Google Translate.
Arabic The Arabic subtask attracted six teams with only two surpassing the baseline of 0.485. The IAI Group [40] achieved a macro F1 score of 0.495, with their XLM-RoBERTa approach, which was the top-ranked system for Arabic. The second ranked team Nullpointer [35] ne-tuned a sentiment-based transformer model using data augmentation and used a custom classier that assigned a higher weight to the objective class.
Bulgarian Four teams submitted runs to the Bulgarian subtask with none surpassing the baseline scoring 0.753. The two systems at the second and third rank – team Nullpointer [35] and team HYBRINFOX [39] – achieved similar scores of 0.717 and 0.715, respectively.
English The English subtask attracted the highest number of participants with 15 teams submitting valid runs, 10 of which surpassed the baseline. Team HYBRINFOX was ranked rst (0.744) followed by Tonirodriguez [45] and SSN-NLP [44]. The teams at positions four to nine achieved similar results in the range [0.69, 0.71]. Team Indigo [41] was ranked 10th, passing the baseline by a small margin. German Three teams submitted runs for the German subtask with only two surpassing the baseline. Team Nullpointer achieved the rst place with a score of 0.791 surpassing the IAI Group ranking second and the baseline by 6 and 10%-points, respectively.
Italian Four teams submitted valid runs to the Italian subtask with team JK_PCIC_UNAM ranking rst obtaining a score of 0.792, closely followed by team HYBRINFOX. The third-positioned team Nullpointer follows with a dierence of 4%-points still surpassing the baseline by a margin.
Below, we describe the approaches of all participating systems; see also Table 4 for an overview.
Team CheckerHacker [49] used an ensemble of two transformer-based models: BERT-base-uncased and XLM-RoBERTa-base. For the ensemble, they used average probabilities. In addition to the ensemble, they implemented data augmentation.
Team CLaC [37] approached the task by leveraging an LLM (Google’s Gemini2) for classication. They modeled the task as a multi-voter scenario where the LLM was used to create two semantically similar sentences for each sentence in the test set. Then, the same LLM predicted the subjectivity of each sentence using a single prompt. Finally, majority voting over the three labels was used to decide the nal subjectivity label. Additionally, the prompt was contextualized by providing 600 randomly selected samples from the training set.
Team Eevvgg [38] compared a number of feature-based supervised models, namely Na¨ive Bayes, SVMs, logistic regression, decision trees and random forest with transformer-based models like BERT for the English subtask, incorporating additional syntactic features deemed as stance markers.
Team FactFinders ne-tuned the Mistral-7B-Instruct-v0.2 model on the data provided for English. (No further details are available, since this submission was not accompanied by a paper.)
Team HYBRINFOX [39] evaluated an ensemble combining a RoBERTa-based encoder, a SentenceBERT encoder, and lexical features. The RoBERTa and SentenceBERT embeddings were concatenated with subjectivity scores extracted from a rule-based expert system based on the VAGO lexical database [48]. These scores covered aspects such as vagueness, subjectivity, detail, and objectivity. The enriched embeddings were then fed into the downstream classier. Regarding training, only RoBERTa was ne-tuned, while the SentenceBERT model weights were frozen. The authors used machine translation with DeepL3 for all non-English sub-tasks.
Team IAI Group [40] experimented with the multilingual XLM-RoBERTa model for all subtasks. They ne-tuned the model for each specic language.
Team Indigo [41] proposed a classier based on SetFit [50], in two steps. In the rst stage, the training data was used to ne-tune a transformer using contrastive learning. The resulting transformer generated sentence embeddings, which are fed into a Dierentiable Linear Neural Layer. This model is known to require only a few training data samples, making it suitable for few-shot learning settings. In their evaluation, the authors show the eectiveness of their model compared to other conventional ne-tuning approaches (e.g., BERT or RoBERTa), especially when trained for only a few epochs. 2https://gemini.google.com 3https://www.deepl.com/en/translator
Team JK_PCIC_UNAM [42] used a BERT-based classier for English and Italian. They ne-tuned two distinct BERT classiers, each tailored to a specic language. In each setting, they enriched BERTbased embeddings with linguistic features, including the number of quotations, the percentage of nouns, verbs, adjectives, adverbs, and sentiment probabilities from input texts provided by pysentimiento [51].
Team JUNLP ne-tuned pre-trained language models (PLMs) for Arabic, Bulgarian, and English. They used BERT for English and multilingual BERT (mBERT) for Arabic and Bulgarian. The original training data was used without tuning the values of the hyperparameters.
Team Nullpointer [35] ne-tuned a BERT-based classier for Arabic, Bulgarian, English, German, and Italian. The BERT model, initially pre-trained for sentiment analysis, was ne-tuned for each specic language, where the sentiment labels output by the model were mapped to subjectivity labels. They handled class imbalance, and translated all non-English data to English.
Team SemanticCuetSync [43] implemented and evaluated several models based on transformers and deep learning for English and Arabic. They experimented with linear regression, SVM, multinomial na¨ive Bayes, -nearest neighbors, random forest, CNN, CNN+LSTM, CNN+BiLSTM, and two LLMs that were ne-tuned. They selected a LLaMA-3-8b model for their submission, as it yielded the best results for both languages.
Team SINAI ne-tuned RoBERTa-base for English on the data provided for the task.
Team SSN-NLP [44] explored keyword- to embedding-based representations, using NN and random forests as well as neural networks, including transformers, all for English. They used a custom preprocessing pipeline, which included tokenization and part-of-speech (POS) tagging to produce additional features. Their best-performing model ne-tuned a RoBERTa-based classier enriched with POS features concerning subjectivity and objectivity.
Team ToniRodriguez [45] ne-tuned two multilingual transformer-based classiers and XLMRoBERTa for English, German, and Italian. Eventually, the mDeBERTa-v3 model was chosen as the best-performing one. Finally, they applied zero-shot cross-lingual transfer to Arabic and Bulgarian.
Team Vigilantes ne-tuned BERT for English on the data provided for the task.
We presented an overview of Task 2 from the CheckThat! lab at CLEF 2024. The task concerned the detection of subjective sentences in controversial news articles. The task was oered in ve dierent languages and also in a multilingual setting.
The majority of the submissions relied on encoder-only transformer-based architectures, either tailored to a specic language or covering multilingualism. Some approaches also evaluated popular large language models with instruction tuning to detect subjectivity or for data augmentation. The most successful solutions coupled transformer-based classiers with domain knowledge in the form of feature extraction, machine translation, and data augmentation, outperforming baselines by a large margin in most tasks, with the exception of Arabic and Bulgarian. The best macro F1 scores ranged between 0.49 and 0.79, which shows that there is a lot of room for improvement, especially for certain languages.
In future work, we plan to increase the number of languages covered and focus on multi- and cross-lingual settings.
The work related to the German data has partially been funded by the BMBF (German Federal Ministry of Education and Research) under grant no. 01FP20031J. The responsibility for the contents of this publication lies with the authors.
The work of M. Hasanain, R. Suwaileh, F. Alam and W. Zaghouani is partially supported by NPRP 14C-0916-210015 from the Qatar National Research Fund, which is a part of Qatar Research Development and Innovation Council (QRDI).
The work of D. Dimitrov, G. Pachov, and I. Koychev is partially nanced by the European UnionNextGenerationEU, through the National Recovery and Resilience Plan of the Republic of Bulgaria, project SUMMIT, No BG-RRP-2.004-0008.
The work of A. Galassi is funded by European Commission’s NextGenerationEU programme, PNRRM4C2-Investimento 1.3, PE00000013 “FAIR”, Spoke 8. based on minimum cuts, in: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), Barcelona, Spain, 2004, pp. 271–278. doi:10.3115/1218955. 1218990. [13] J. M. Wiebe, R. F. Bruce, T. P. O’Hara, Development and use of a gold-standard data set for subjectivity classications, in: Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, College Park, Maryland, USA, 1999, pp. 246–253. doi:10.3115/1034678.1034721. [14] T. Wilson, J. Wiebe, Annotating opinions in the world press, in: Proceedings of the SIGDIAL 2003 Workshop, The 4th Annual Meeting of the Special Interest Group on Discourse and Dialogue, July 5-6, 2003, Sapporo, Japan, The Association for Computer Linguistics, 2003, pp. 13–22. [15] M. Abdul-Mageed, M. Diab, Subjectivity and sentiment annotation of Modern Standard Arabic newswire, in: N. Ide, A. Meyers, S. Pradhan, K. Tomanek (Eds.), Proceedings of the 5th Linguistic Annotation Workshop, Association for Computational Linguistics, Portland, Oregon, USA, 2011, pp. 110–118. [16] I. Chaturvedi, E. Cambria, R. E. Welsch, F. Herrera, Distinguishing between facts and opinions for sentiment analysis: Survey and challenges, Inf. Fusion 44 (2018) 65–77. doi:10.1016/j.infus.2017. 12.006. [17] M. Geva, Y. Goldberg, J. Berant, Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, 2019, pp. 1161–1166. doi:10.18653/v1/D19-1107. [18] P. Ro¨ttger, B. Vidgen, D. Hovy, J. B. Pierrehumbert, Two contrasting data annotation paradigms for subjective NLP tasks, in: M. Carpuat, M. de Marnee, I. V. M. Ru´ız (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 1015, 2022, Association for Computational Linguistics, 2022, pp. 175–190. doi:10.18653/v1/2022. naacl-main.13. [19] F. Antici, F. Ruggeri, A. Galassi, K. Korre, A. Muti, A. Bardi, A. Fedotova, A. Barro´n-Cede n˜o, A corpus for sentence-level subjectivity detection on English news articles, in: N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italia, 2024, pp. 273–285. [20] A. Galassi, F. Ruggeri, A. Barro´n-Cede n˜o, F. Alam, T. Caselli, M. Kutlu, J. Struss, F. Antici, M. Hasanain, J. Ko¨hler, K. Korre, F. Leistra, A. Muti, M. Siegel, T. Mehmet Deniz, M. Wiegand, W. Zaghouani, Overview of the CLEF-2023 CheckThat! lab: Task 2 on subjectivity in news articles, in: M. Aliannejadi, G. Faggioli, N. Ferro, Vlachos, Michalis (Eds.), Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, CLEF 2023, Thessaloniki, Greece, 2023. [21] S. Rustamov, E. Mustafayev, M. Clements, Sentence-level subjectivity detection using neuro-fuzzy models, in: A. Balahur, E. van der Goot, A. Montoyo (Eds.), Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Association for Computational Linguistics, Atlanta, Georgia, 2013, pp. 108–114. [22] F. Benamara, B. Chardon, Y. Mathieu, V. Popescu, Towards context-based subjectivity analysis, in: H. Wang, D. Yarowsky (Eds.), Proceedings of 5th International Joint Conference on Natural Language Processing, Asian Federation of Natural Language Processing, Chiang Mai, Thailand, 2011, pp. 1180–1188. [23] F. Antici, L. Bolognini, M. A. Inajetovic, B. Ivasiuk, A. Galassi, F. Ruggeri, SubjectivITA: An italian corpus for subjectivity detection in newspapers, in: K. S. Candan, B. Ionescu, L. Goeuriot, B. Larsen, H. Mu¨ller, A. Joly, M. Maistro, F. Piroi, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2021, volume 12880 of LNCS, Springer, 2021, pp. 40–52. doi:10.1007/978-3-030-85251-1_4. [24] C. Banea, R. Mihalcea, J. Wiebe, Multilingual subjectivity: Are more languages better?, in: C.-R.
Huang, D. Jurafsky (Eds.), Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), Coling 2010 Organizing Committee, Beijing, China, 2010, pp. 28–36. [25] M. Abdul-Mageed, M. Diab, M. Korayem, Subjectivity and sentiment analysis of Modern Standard Arabic, in: D. Lin, Y. Matsumoto, R. Mihalcea (Eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Portland, Oregon, USA, 2011, pp. 587–591. [26] C. Banea, R. Mihalcea, J. Wiebe, Sense-level subjectivity in a multilingual setting, Comput. Speech
Lang. 28 (2014) 7–19. doi:10.1016/j.csl.2013.03.002. [27] A. Barro´n-Cede n˜o, F. Alam, A. Galassi, G. Da San Martino, P. Nakov, , T. Elsayed, D. Azizov, T. Caselli, G. Cheema, F. Haouari, M. Hasanain, M. Kutlu, C. Li, F. Ruggeri, J. M. Struß, W. Zaghouani, Overview of the CLEF–2023 CheckThat! Lab checkworthiness, subjectivity, political bias, factuality, and authority of news articles and their source, in: A. Arampatzis, E. Kanoulas, T. Tsikrika, S. Vrochidis, A. Giachanou, D. Li, M. Aliannejadi, M. Vlachos, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fourteenth International Conference of the CLEF Association (CLEF 2023), 2023. [28] Z. Sheikh Ali, W. Mansour, T. Elsayed, A. Al-Ali, AraFacts: The rst large Arabic dataset of naturally occurring claims, in: N. Habash, H. Bouamor, H. Hajj, W. Magdy, W. Zaghouani, F. Bougares, N. Tomeh, I. Abu Farha, S. Touileb (Eds.), Proceedings of the Sixth Arabic Natural Language Processing Workshop, Association for Computational Linguistics, Kyiv, Ukraine (Virtual), 2021, pp. 231–236. URL: https://aclanthology.org/2021.wanlp-1.26. [29] R. Suwaileh, M. Hasanain, F. Hubail, W. Zaghouani, F. Alam, ThatiAR: Subjectivity detection in arabic news sentences, arXiv: 2406.05559 (2024). [30] G. Karadzhov, P. Gencheva, P. Nakov, I. Koychev, We built a fake news / click bait lter: What happened next will blow your mind!, in: R. Mitkov, G. Angelova (Eds.), Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, INCOMA Ltd., Varna, Bulgaria, 2017, pp. 334–343. doi:10.26615/978-954-452-049-6_045. [31] F. Ruggeri, F. Antici, A. Galassi, K. Korre, A. Muti, A. Barro´n-Cede n˜o, On the denition of prescriptive annotation guidelines for language-agnostic subjectivity detection, in: R. Campos, A. M. Jorge, A. Jatowt, S. Bhatia, M. Litvak (Eds.), Text2Story@ECIR, volume 3370 of CEUR Workshop Proceedings, CEUR-WS.org, 2023, pp. 103–111. [32] K. Krippendor, Computing krippendor’s alpha-reliability, 2011. URL: https://repository.upenn.
edu/handle/20.500.14332/2089. [33] J. Ko¨hler, G. K. Shahi, J. M. Struß, M. Wiegand, M. Siegel, T. Mandl, M. Schu¨tz, Overview of the CLEF-2022 CheckThat! lab task 3 on fake news detection, in: G. Faggioli, N. Ferro, A. Hanbury, M. Potthast (Eds.), Working Notes of CLEF 2022—Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [34] J. R. Landis, G. G. Koch, The measurement of observer agreement for categorical data, Biometrics 33 (1977) 159–174. [35] M. R. Biswas, A. Tasneem Abir, W. Zaghouani, Nullpointer at CheckThat! 2024: Identifying subjectivity from multilingual text sequence, in: [52], 2024. [36] S. D. Zehra, K. Chandani, M. Khubaib, A. A. Aun Muhammed, F. Alvi, A. Samad, Checker Hacker at CheckThat! 2024: Detecting check-worthy claims and analyzing subjectivity with transformers, in: [52], 2024. [37] S. Gruman, L. Kosseim, CLaC at CheckThat! 2024: A zero-shot model for check-worthiness and subjectivity classication, in: [52], 2024. [38] E. Gajewska, Eevvgg at CheckThat! 2024: Evaluative terms, pronouns and modal verbs as markers of subjectivity in text, in: [52], 2024. [39] M. Casanova, J. Chanson, B. Icard, G. Faye, G. Gadek, G. Gravier, P. E´gre´, HYBRINFOX at CheckThat! 2024 - task 2: Enriching BERT models with the expert system VAGO for subjectivity detection, in: [52], 2024. [40] P. R. Aarnes, V. Setty, P. Galusˇcˇa´kova´, IAI group at CheckThat! 2024: Transformer models and data augmentation for checkworthy claim detection, in: [52], 2024. [41] S. Sar, D. Roy, Indigo at CheckThat! 2024: Using Sett: A resource ecient technique for subjectivity detection in news article., in: [52], 2024. [42] K. Salas-Jimenez, I. D´ıaz, H. Go´mez-Adorno, JK_PCIC_UNAM at CheckThat! 2024: Analysis of subjectivity in news sentences using transformers based models, in: [52], 2024. [43] A. I. Paran, M. S. Hossain, S. H. Shohan, J. Hossain, S. Ahsan, M. M. Hoque, SemanticCuetSync at
CheckThat! 2024: Finding subjectivity in news article using Llama, in: [52], 2024. [44] P. Premnath, P. Vaithiya Subramani, B. B, N. R. Salim, SSN-NLP at CheckThat! 2024: From classic algorithms to transformers: A study on detecting subjectivity, in: [52], 2024. [45] A. Rodr´ıguez de la Torre, E. Golobardes Ribe´, J. Suau Mart´ınez, Tonirodriguez at CheckThat!2024: Is it possible to use zero-shot cross-lingual for subjectivity detection in low-resources languages?, in: [52], 2024. [46] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, 2019, pp. 3980–3990. doi:10.18653/V1/D19-1410. [47] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). arXiv:1907.11692. [48] B. Icard, V. Claveau, G. Atemezing, P. E´ gre´, Measuring vagueness and subjectivity in texts: From symbolic to neural VAGO, in: Proceedings of the IEEE International Conference on Web Intelligence and Intelligent Agent Technology, IEEE, 2023, pp. 395–401. doi:10.1109/WI-IAT59888.2023. 00065. [49] K. Chandani, D. E. Z. Syeda, Checker Hacker at CheckThat! 2024: Ensemble models for checkworthy tweet identication, in: [52], 2024. [50] L. Tunstall, N. Reimers, U. E. S. Jo, L. Bates, D. Korat, M. Wasserblat, O. Pereg, Ecient few-shot learning without prompts, 2022. URL: https://arxiv.org/abs/2209.11055. arXiv:2209.11055. [51] J. M. Pe´rez, M. Rajngewerc, J. C. Giudici, D. A. Furman, F. Luque, L. A. Alemany, M. V. Mart´ınez, pysentimiento: A python toolkit for opinion mining and social nlp tasks, 2023. arXiv:2106.09462. [52] G. Faggioli, N. Ferro, P. Galusˇcˇa´kova´, A. Garc´ıa Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CLEF 2024, 2024.