Subjectivity is inherently encoded in language and involves expressions of the speaker's position, attitude, and feelings towards the uttered message [17]. Thus, identification of articles written from a subjective perspective of the author involves detection of stance markers: words that express some form of evaluation or judgement (e.g. words denoting emotional valence), pronouns or modal verbs and passive constructions [16]. This work makes use of syntactic and semantic features that are fed to machine learning algorithms and large language models (LLMs) for a binary detection of news articles written from a subjective versus objective perspective of the author [4]. The datasets have been provided by the organizers of the CLEF 2024 CheckThat! lab [5], which is the international contest on challenging classification and retrieval problems. The aim of this competition is to advance the field of information retrieval from text. To this end, a new approach to subjectivity detection is proposed: a Transformer-based model that makes use of both text content and additional meta features derived from articles. Such an approach shows more consistent results than simple transfer learning (TL; adding a classification layer on top of BERT encoder) fed with textual content only. The current work describes the approach by the eevvgg team for Task 2: Subjectivity in News Articles of the CLEF 2024 CheckThat! lab in English news articles.
It is a common approach in information retrieval research to experiment with different text representation models, as in [6] where conversion of text with TF-IDF (Term Frequency Inverse Document Frequency) algorithm outperformed models with so-called Count Vectorizer, that is, a simple frequency count of particular terms in each text sample. Influence of preprocessing techniques on classification performance of deep learning models was tested in [3]. Machine learning methods are among fundamental methods used in the field of natural language processing [2]. They are used, for example, in web search engines for information retrieval [7]. Thereby, a user looking for specific information gets results relevant to the searched topic. Previous studies investigated performance of several machine learning models: logistic regression, SVM, and Naive Bayes for text classification tasks such as recognition of political affiliation of the US presidential candidates from their presidential campaign speeches [1]. Contribution of this work is three-fold: first, examination of machine learning vs. transfer learning approaches for information extraction; second, impact of feature normalisation on classification performance; third, influence of training data size on detection performance. Models for detection of subjective opinions versus objective facts could be utilised, for example, in studies on information verification (detection of fake news, which are a mixture of subjective opinion and facts [8]).
Differentiation between subjective opinions and objective facts compose a basis of proper journalism. Subjective reporting of news might reflect bias of the author which automatic solutions can identify and tag properly to reduce the spread of fake news, for example. Several work have experimented with subjectivity detection methods from text. Unified method for detection of subjectivity in multilingual text content was proposed in [12]. Fine-tuned ELECTRA large model outperformed other large language models (such as BERT and RoBERTa) for subjectivity analysis in text in the context of fake news detection, achieving 0.983 accuracy [20]. Such analyses have proved to be useful for detecting fake news with a lexicon-based approach [11] as well as in opinion mining using deep learning techniques [18]. The current work extends these studies by combining previous approaches and testing the proposed model in several experimental settings. Team eevvgg proposes a deep learning model that combines the fine-tuned BERT encoder [9] and lexicon-based method for extracting linguistic markers of subjectivity. The proposed architecture is called BERTd and is tested against other BERT-based models and machine learning algorithms in two experimental conditions: on a smaller vs. a larger set of data (n=667 vs. n=1511 training samples, and n=166 vs. n=484 test samples, respectively); and on raw vs. normalised values of linguistic meta features.
This work deals with binary detection of subjectivity from text material: whether a sentence expresses a subjective view of the author (SUBJ) or presents an objective view on the topic (OBJ). The paper proposes solutions for subjectivity extraction for English data. Initial training data comprises 833 text samples (data units) from newspaper articles: short pieces of text of up to 100 words. Development of a text classification system starts with pre-processing of the data, then splitting it into training and testing sets, extraction of features, model training and model validation. Then, the dataset was divided into two splits: 80% for training purposes and 20% for testing. Results of training and evaluation on this set (called small set) are reported in Table 1. Evaluation and training is conducted also on a bigger setofficial test set -released by the CheckThat! organizers after submission deadline and training data comprising all available train and dev sets -1511 in total; evaluation is conducted on 484 samples from the official test. Results of training and evaluation on this set (called big set) are reported in Table 2.
Usually, the first step in information retrieval tasks is to represent the text using a certain model. A common approach is to represent a document as a vector of features -most simple representations of text include Bag-of-Words (BOW) models. Regarding machine learning algorithms TF-IDF method is employed (with max_df=0.75, min_df=2). Transfer learning approaches utilise BERT encoder (BERT base uncased) as a text representation method. In order to improve the predictive performance of these models, several meta features were constructed from the textual content of news articles using the concept of feature engineering [14]. It involves the application of transformation functions on given features to generate new ones. In order to extract such features from text, it needs to be normalised. Data cleaning involved 3 steps: conversion of text into lowercase, removal of stop-words and punctuation symbols. Then, the text was lemmatised, that is, words were converted to their dictionary forms. Finally, linguistic features were extracted from the clean and lemmatised text of articles. In total, 16 syntactic features (stance markers) were extracted from text samples, specifically, frequency of occurrence of each category of terms in a given text sample. Specific terms that belong to each category are specified below:
1. Subject pronouns: I, you, he, she, it, we, you, they; 2. Object pronouns: me, you, him, her, it, us, you, them; 3. Possessive pronouns: mine, yours, his, hers, its, ours, yours, theirs; 4. Demonstrative pronouns: this, these, that, those 5. Interrogative pronouns: who, whom, which, what 6. Relative pronouns: who, whom, that, which, whoever, whichever, whomever; 7. Indefinite pronouns: all, another, any, anybody, anyone, anything, each, everybody, everyone, everything, few, many, nobody, none, one, several, some, somebody, someone; 8. Reflexive pronouns: myself, yourself, himself, herself, ourselves, yourselves, themselves 9. Modal verbs: must, shall, will, should, would, can, could, may, might; 10. Obligation verbs: need, have to, must, might, may, has to, shall; 11. Frequency adverbs: hardly, ever, rarely, scarcely, seldom, never, sometimes, often, always, usually, normally; 12. Comparison adverbs: bad, badly, worse, worst, good, better, well, best, far, farther, further, farthest, furthest, little, less, least, few, somehow; 13. Reporting verbs: advise, agree, challenge, claim, decide, demand, encourage, invite, offer, persuade, promise, refuse, remind, say; 14. Pronouns: a sum of features 1-8; 15. Emotive words: words associated with an expression of emotions marked as such in the lexicon of emotion-laden terms [13]; 16. Polarising words: words associated with inducing social polarisation (dividing the society into 'us' versus 'them' groups) marked as such in the lexicon of polarising terms [19];
The summary of preprocessing procedure is illustrated in Figure 1.
In the light of available algorithms, scikit-learn map of estimators is followed in order to chose four of them: Naive Bayes (NB), logistic regression (LR), decision trees (DT) and decision forests (DF). The Naive Bayes is one of the simplest and most popular models in the field of supervised machine learning [2] and amongst the most efficient and effective classifiers [10]. Classification with the use of this estimator is based on the calculated probabilities: probability of a certain label for a given data point is estimated through multiplying the probability of this label by a sum of probabilities of all features describing this data point given the label [7]. The NB algorithm learns probabilities based on prior distribution across classes from the training data following the assumption that all features are independent. SVM models learn to categorise data into separate classes by building a margin in the feature space that minimizes the distance between each class and that margin [21]. The logistic regression algorithm calculates the log-odds (converted into probabilities) of an event as a linear combination of one or more independent variables. The Decision Tree model predicts the value of a target variable by learning decision rules, which are inferred from the training data. DT builds a tree-like structure from these rules by splitting data into subsets based on the values of particular features until a stopping criterion is met. DTs have two main advantages of being simple to understand and interpret [15]. A decision tree forest is an ensemble learning method that combines outputs of multiple decision trees to reach the final result. Finally, the suitability of large language models for subjectivity detection is investigated. Specifically, BERT uncased model1 ) that produces contextualised text embeddings and achieves state of the art results in most information retrieval tasks. Transfer learning paradigm is utilised for training BERT-based models. Tensorflow and transformers libraries are employed for their implementation.
Machine Learning. Default settings of hyper-parameters set by the authors of the scikit-learn library are employed in machine learning estimators. TF-IDF method is utilised as a text representation method.
Textual features are combined with (normalised) meta features described in Section 2.1 and fed to these algorithms.
Transfer Learning. Regarding TL with LLMs, BERT is utilised for as a text encoder, specifically the CLS token. Then, additional layers are added on top of embeddings returned by the BERT-encoder. Two BERT-based models are proposed: BERTs comprises of two fully-connected layers2 of size 128 and a 0.5 dropout rate between them; in BERTd two fully-connected layers of size 246 and 32, separated by a 0.4 dropout layer are attached to the BERT encoder. In addition BERTb is utilised a baseline model with only a classification layer added on top of the BERT encoder. Cross-entropy is employed as the loss function and rectified linear unit (ReLU) as the activation function in all hidden layers. Classification layer comprises of two units, which in combination with the softmax function returns probabilities that a given text sample belong to class "OBJ" and "SUBJ". Learning rate is set to 5e-5 as advised by the authors of the transformers library. BERT-based models are trained for either 2, 3 or 4 epochs and the best result is reported. Tensorflow implementation of these networks and functions is employed.
Evaluation metrics. Three metrics are employed for evaluation purposes: weighted πΉ 1 (Eq. 2), macro-averaged πΉ 1 (Eq. 3) and accuracy (Eq. 4).
π πππβπ‘πππΉ 1 =
π πππππΉ 1 =
where TP: true positives, FP: false positives, TN: true negatives, FN: false negatives.
Results on a small train set, reported in Table 1, indicate that BERTb, a simple transfer learning approach, outperforms all other classifiers. Nonetheless, all tested models perform above baseline using prior label distribution. Normalisation of additional features boosts performance for ML algorithms (NB, LR, DT, RB), although BERT-based models note minor differences in performance. All models developed with a TL approach outperform all ML algorithms achieving, on average, 27% higher results (23% for normalised meta features and 30% for models without normalisation or meta features) in terms of macro πΉ 1 .
The proposed BERTd model (officially submitted to the task), consisting of two hidden layers and fed with data without normalisation of meta features, outperforms other solutions in turn on the bigger set (see Table 2). The difference in performance between ML and TL approaches decreases to 20% -due to lower macro πΉ 1 of two TL models: BERTb and BERTs. Compared to BERTb and BERTs, BERTd notes smaller differences in performance between small and big training sets. Thus, its performance is more stable across data than BERTb and BERTs. Nonetheless, all models outperform a baseline classifier using class prior distribution. All TL models and the LR classifier achieve also higher macro πΉ 1 than the baseline provided by the CheckThat! organisers.
Results: big set. Classification performance of machine learning (ML) and transfer learning (TL) models. A single asterisk signals models that perform above baseline in all three metrics (accuracy, weighted (w-πΉ 1 ) and macro (m-πΉ 1 ) πΉ 1 scores). Double asterisk signals best model in each category of machine learning (ML) and transfer learning (TL) models. A plus indicates models that achieve higher results than the baseline provider by CheckThat! organizers. In bold the best performing model is marked.
Meta features w-πΉ 1 m-πΉ
Once again, transfer learning outperformed simpler machine learning approaches for information extraction from text. All BERT-based models (with or without syntactic features) achieved substantially higher results for a binary detection of subjectivity than 4 ML algorithms. BERTd fed with additional features (syntactic features of marker) shows more consistent performance than a baseline BERTb model comprising of BERT encoder and a classification layer. Normalisation of meta features was found to boost performance for ML models. Increase of training data size had almost no impact on prediction performance for ML models and a negative influence for TL solutions in terms of macro πΉ 1 metric.
Baselines. The current work would benefit from a more thorough analysis of results of the proposed models against other systems developed on the employed dataset.
The proposed architecture notes satisfactory performance on the utilised dataset (ranking 8th in the official leaderboard of the competition on English data), however, its robustness is yet to be tested, for example, in scenarios with different training datasets or multilingual data.
Ablation studies. Ablation analysis in future work could measure the impact of individual features on the final performance of the proposed subjectivity detectors.