In this paper, we describe our approach to CLEF 2024 Lab 2 CheckThat! Task 1 (Check-worthiness) and Task 2 (Subjectivity), which aims to evaluate how consistent Large Language Models (LLMs) can distinguish between objective truths worth fact checking and subjective opinions. Our approach was based on prompt engineering utilizing Google's pre-trained LLM Gemini. To optimize performance, we including a random subset of the training data in the input prompt while also augmenting the test data through paraphrasing. We achieved and F1 score of 0.564 for Task 1 and 0.445 for Task 2 which ranked us 24th and 14th respectively. This work contributes to understanding the limitation of LLMs and highlights one of their major pitfalls, subjective reasoning.
Since the dawn of the internet, news sources have become abundant and hence personalized to their
target demographic [
CLEF-2024 Task 1 [
The CLEF-2024 Task 2 is also a binary classification task aimed at distinguishing whether a sentence
in a news article reflects the author’s subjective opinion or presents objective information 2. Subjectivity
in natural language refers to the aspects of language used to express personal opinions. However, this
task faces an inevitable issue as the perception of subjectivity can be influenced by personal biases [
With the recent surge in LLMs, we were motivated to test the subjectivity classification capabilities
of such models. To do so, we used Google’s LLM Gemini because of its access to the information on
Google’s search engine [
Section 2 summarizes the data and presents an overview of the English corpus distribution. Section 3 presents an overview of our model’s methodology, while Section 4 describes the outcome of our approach. Finally, Section 5 analyses the results of our model. The code of the models presented in this paper is available on GitHub. 3
CLEF 2024 CheckThat! lab proposed 6 shared tasks: Check-worthiness, Subjectivity, Persuasion, Roles,
Authorities, and Adversarial Robustness [
Organisers of the CLEF CheckThat! Task 1 and Task 2 provided two diferent datasets. The data for
Task 1 was based of on debates and political speeches [
CLEF 2024 Task 1 - Check-worthiness ofered datasets in Arabic, Dutch, and English, out of which we only participated in English. As shown in Table 1, the data set consisted of three features: 1. Sentence ID: A unique identifier for each sentence. 2. Sentence: The sentence itself, which is either worthy of fact checking or not.
3. Label: A binary label: "No" if a fact check is not necessary, or "Yes" if it is.
Each instance was extracted from debates and political speeches. The English corpus consisted of a total of 23,851 instances with 22,851 for training, 1,032 for development and only 318 for testing. As shown in Table 2, the data sets were not balanced with 76% of the Train set being not worthy of a fact check versus 24% being worthy of a fact check. The Dev-Test set had a slightly diferent distribution with 66% being not worthy of a fact check versus 34% being worthy of a fact check.
CLEF 2024 Task 2 - Subjectivity provided datasets in Arabic, Bulgarian, English, German, and Italian, out of which we participated in only English. As shown in Table 3, the data set consisted of 3 features: 1. Sentence ID: A unique identifier for each sentence. 2. Sentence: The sentence itself, which is either subjective or objective. 3. Label: A binary label, "OBJ" for objective, or "SUBJ" for subjective.
Each sentence was an objective or subjective extract from a news article. The English corpus consisted of a total of 1,049 instances.
As shown in Table 4, the Train set was not balanced with 64% instances from the training set being objective versus 36% being subjective, whereas the Dev-Test set had a distribution of 61% as objective 39% as subjective.
LLMs have demonstrated a general ability to understand natural language [
Each of the LLMs involved in Vijayan experiment was fine tuned to the specific task using EasyLLM 5. The aim of their experiment was to evaluate the performance of the LLMs without changing any default hyperparameters [11]. Therefore, no hyperparameters were modified.
In terms of subjectivity detection, previous research has followed two main approaches: lexicon-based [12], and machine-learning methods (ML)[13]. In the former, researchers use a list of subjective words, and the frequency of these words determines a document’s subjectivity score. The list of words is manually compiled and is static. Alternatively, machine-learning methods also rely on the idea that certain words are inherently subjective, however, these models utilize ML algorithms to dynamically determine the list of subjective words [14].
To participate in both Task 1 and Task 2, our approach was based on three key components:
Leveraging LLMs: LLMs provide an extensive understanding of general language patterns as they
are trained on massive data sets. Because of modern processing power and transformers, LLMs are now
able to grasp a larger context window [15]. We used Gemini because of its access to information on
Google’s search engine [
Paraphrasing: In an attempt to ensure the LLM has confidence in it’s classification, we asked Gemini to generate two semantically identical paraphrases of each instance in the test set. The prompt we used to paraphrase the test set can be seen in Table 5.
Prompt "Paraphrase this sentence into two semantically identical sentences <sentence>"
Prompt Engineering: We engineered the prompt to optimize accuracy by including a random subset of the training set, the initial test sentence and its two paraphrased test sentences. We asked Gemini to classify each of these three test sentences: for task 1, as "Yes" or No"; for task 2, as "Objective" or "Subjective". The final label for the original sentence was determined by a majority vote from the three sentences (1 original sentence + 2 paraphrased sentences). Additionally, we included a random subset of 600 training instances in the input prompt to give the model an idea on how to classify the test sentence. The exact prompts we used can be seen in Table 6 and 7.
We experimented with three main methods in the development process of the model. Method 1 used Gemini solely for sentence classification. Method 2 included a random subset of the training set in the input prompt to feed the model context. To optimize accuracy, the input prompt token limit was maximized by including 600 random training sentences, along with the test sentence. Method 3 was an ensemble combining method 2 along with data set augmentation through paraphrasing. Each test sentence was paraphrased into two additional sentences using Gemini. Classification for method 3 was performed by a majority vote from the original test sentence and its respective two paraphrased sentences.
With the abundance of biases on the internet, and hence in LLM training sets, it is of utmost priority that LLM parent corporations take necessary precautions to produce ethical AI. To do so, the model’s output is often filtered to safeguard against misinformation. This practice is efective yet results in overly cautious models that do not engage in complex ethical opinions and will inadvertently hinder the model’s capacity for subjective reasoning. This pitfall is evident in the disparity between results shown in Table 9.
Many issues come to light when deciphering subjectivity and check-worthiness in text. This paper describes our approach using Googles pre-trained LLMs, Gemini, which show a significant improvement in classification when including context in the input prompt. However, dataset augmentation did not seem to improve results.
Although our model was able to classify Task 1 with an F1 score of 0.564, the model still struggled on Task 2 with a F1 score of 0.445. We speculate that Task 1 (check-worthiness classification) is inherently more objective than Task 2 (subjectivity classification), which would explain the higher scores on Task 1.
This work is just scratching the surface of LLM intricacies. Future iterations of this project can implement more concise prompts as well experimenting with other LLMs.
The authors would like to thank the organisers of the CLEF-2024 CheckThat! shared task and the anonymous reviewers for their comments on the previous version of this paper. [11] K. Vijayan, Finetuning Large Language Modles for Binary Classiifcation, EasyLLM (2023). URL: https://www.easyllm.tech/docs/blog/ ifnetuning-large-language-models-for-binary-classification.html. [12] B. Liu, Sentiment Analysis and Subjectivity, Handbook of Natural Language Processing (2010). URL: https://www.researchgate.net/profile/Bing-Liu-120/publication/228667268_Sentiment_analysis_ and_subjectivity/links/5472bbea0cf24bc8ea199f7c/Sentiment-analysis-and-subjectivity.pdf. [13] A. S. Samaneh Karimi, A language-model-based approach for subjectivity detection , Sage Journals (2016). URL: https://journals.sagepub.com/doi/10.1177/0165551516641818. [14] D. M. P. Kushal Dave, Steve Lawrence, Mining the peanut gallery: Opinion extraction and semantic classification of product reviews, Handbook of Natural Language Processing (2003). URL: https://dl.acm.org/doi/proceedings/10.1145/775152. [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Kaiserłukasz, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). URL: http://arxiv.org/pdf/2307.06435.