This work tests performance of simple machine learning algorithms against large language models (LLMs) utilising transfer learning for a binary detection of subjectivity in news articles. Second, the influence of feature normalisation on classification performance is examined. Third, the work measures impact of training data size on subjectivity extraction. The proposed BERTd model that makes use of additional information about stance markers in news articles was placed 8th in the oficial ranking of the CLEF 2024 CheckThat! lab Task 2: Subjectivity in News Articles competition for English data, achieving 0.70 macro-averaged 1. Models that distinguish subjective opinions from objective facts could be utilised in studies on information verification (detection of fake news, understood as a mixture of subjective opinion and facts).
Subjectivity is inherently encoded in language and involves expressions of the speaker’s position,
attitude, and feelings towards the uttered message [17]. Thus, identification of articles written from a
subjective perspective of the author involves detection of stance markers: words that express some
form of evaluation or judgement (e.g. words denoting emotional valence), pronouns or modal verbs
and passive constructions [16]. This work makes use of syntactic and semantic features that are fed
to machine learning algorithms and large language models (LLMs) for a binary detection of news
articles written from a subjective versus objective perspective of the author [
It is a common approach in information retrieval research to experiment with diferent text
representation models, as in [
Diferentiation between subjective opinions and objective facts compose a basis of proper journalism.
Subjective reporting of news might reflect bias of the author which automatic solutions can identify
and tag properly to reduce the spread of fake news, for example. Several work have experimented with
subjectivity detection methods from text. Unified method for detection of subjectivity in multilingual
text content was proposed in [12]. Fine-tuned ELECTRA large model outperformed other large language
models (such as BERT and RoBERTa) for subjectivity analysis in text in the context of fake news detection,
achieving 0.983 accuracy [20]. Such analyses have proved to be useful for detecting fake news with
a lexicon-based approach [
This work deals with binary detection of subjectivity from text material: whether a sentence expresses a subjective view of the author (SUBJ) or presents an objective view on the topic (OBJ). The paper proposes solutions for subjectivity extraction for English data. Initial training data comprises 833 text samples (data units) from newspaper articles: short pieces of text of up to 100 words. Development of a text classification system starts with pre-processing of the data, then splitting it into training and testing sets, extraction of features, model training and model validation. Then, the dataset was divided into two splits: 80% for training purposes and 20% for testing. Results of training and evaluation on this set (called small set) are reported in Table 1. Evaluation and training is conducted also on a bigger set oficial test set - released by the CheckThat! organizers after submission deadline and training data comprising all available train and dev sets - 1511 in total; evaluation is conducted on 484 samples from the oficial test. Results of training and evaluation on this set (called big set) are reported in Table 2.
Usually, the first step in information retrieval tasks is to represent the text using a certain model. A common approach is to represent a document as a vector of features - most simple representations of text include Bag-of-Words (BOW) models. Regarding machine learning algorithms TF-IDF method is employed (with max_df=0.75, min_df=2). Transfer learning approaches utilise BERT encoder (BERT base uncased) as a text representation method. In order to improve the predictive performance of these models, several meta features were constructed from the textual content of news articles using the concept of feature engineering [14]. It involves the application of transformation functions on given features to generate new ones. In order to extract such features from text, it needs to be normalised. Data cleaning involved 3 steps: conversion of text into lowercase, removal of stop-words and punctuation symbols. Then, the text was lemmatised, that is, words were converted to their dictionary forms. Finally, linguistic features were extracted from the clean and lemmatised text of articles. In total, 16 syntactic features (stance markers) were extracted from text samples, specifically, frequency of occurrence of each category of terms in a given text sample. Specific terms that belong to each category are specified below: 1. Subject pronouns: I, you, he, she, it, we, you, they; 2. Object pronouns: me, you, him, her, it, us, you, them; 3. Possessive pronouns: mine, yours, his, hers, its, ours, yours, theirs; 4. Demonstrative pronouns: this, these, that, those 5. Interrogative pronouns: who, whom, which, what 6. Relative pronouns: who, whom, that, which, whoever, whichever, whomever; 7. Indefinite pronouns: all, another, any, anybody, anyone, anything, each, everybody, everyone, everything, few, many, nobody, none, one, several, some, somebody, someone; 8. Reflexive pronouns: myself, yourself, himself, herself, ourselves, yourselves, themselves 9. Modal verbs: must, shall, will, should, would, can, could, may, might; 10. Obligation verbs: need, have to, must, might, may, has to, shall; 11. Frequency adverbs: hardly, ever, rarely, scarcely, seldom, never, sometimes, often, always, usually, normally; 12. Comparison adverbs: bad, badly, worse, worst, good, better, well, best, far, farther, further, farthest, furthest, little, less, least, few, somehow; 13. Reporting verbs: advise, agree, challenge, claim, decide, demand, encourage, invite, ofer, persuade, promise, refuse, remind, say; 14. Pronouns: a sum of features 1-8; 15. Emotive words: words associated with an expression of emotions marked as such in the lexicon of emotion-laden terms [13]; 16. Polarising words: words associated with inducing social polarisation (dividing the society into ‘us’ versus ‘them’ groups) marked as such in the lexicon of polarising terms [19];
3.3. Tools
In the light of available algorithms, scikit-learn map of estimators is followed in order to chose four of
them: Naive Bayes (NB), logistic regression (LR), decision trees (DT) and decision forests (DF). The Naive
Bayes is one of the simplest and most popular models in the field of supervised machine learning [
Machine Learning. Default settings of hyper-parameters set by the authors of the scikit-learn library are employed in machine learning estimators. TF-IDF method is utilised as a text representation method. Textual features are combined with (normalised) meta features described in Section 2.1 and fed to these algorithms.
Transfer Learning. Regarding TL with LLMs, BERT is utilised for as a text encoder, specifically the CLS token. Then, additional layers are added on top of embeddings returned by the BERT-encoder. Two BERT-based models are proposed: BERTs comprises of two fully-connected layers2 of size 128 and a 0.5 dropout rate between them; in BERTd two fully-connected layers of size 246 and 32, separated by a 0.4 dropout layer are attached to the BERT encoder. In addition BERTb is utilised a baseline model with only a classification layer added on top of the BERT encoder. Cross-entropy is employed as the loss function and rectified linear unit (ReLU) as the activation function in all hidden layers. Classification layer comprises of two units, which in combination with the softmax function returns probabilities that a given text sample belong to class "OBJ" and "SUBJ". Learning rate is set to 5e-5 as advised by the authors of the transformers library. BERT-based models are trained for either 2, 3 or 4 epochs and the best result is reported. Tensorflow implementation of these networks and functions is employed. Evaluation metrics. Three metrics are employed for evaluation purposes: weighted 1 (Eq. 2), macro-averaged 1 (Eq. 3) and accuracy (Eq. 4).
1 =
2 × 2 × + + ℎ 1 = 1 = ∑︀=1 × × 1
∑︀=1 × 1
+ =
+ + + where TP: true positives, FP: false positives, TN: true negatives, FN: false negatives. (1) (2) (3) (4)
Results on a small train set, reported in Table 1, indicate that BERTb, a simple transfer learning approach, outperforms all other classifiers. Nonetheless, all tested models perform above baseline using
prior label distribution. Normalisation of additional features boosts performance for ML algorithms (NB, LR, DT, RB), although BERT-based models note minor diferences in performance. All models developed with a TL approach outperform all ML algorithms achieving, on average, 27% higher results (23% for normalised meta features and 30% for models without normalisation or meta features) in terms of macro 1.
The proposed BERTd model (oficially submitted to the task), consisting of two hidden layers and fed with data without normalisation of meta features, outperforms other solutions in turn on the bigger set (see Table 2). The diference in performance between ML and TL approaches decreases to 20% - due to lower macro 1 of two TL models: BERTb and BERTs. Compared to BERTb and BERTs, BERTd notes smaller diferences in performance between small and big training sets. Thus, its performance is more stable across data than BERTb and BERTs. Nonetheless, all models outperform a baseline classifier using class prior distribution. All TL models and the LR classifier achieve also higher macro 1 than the baseline provided by the CheckThat! organisers.
Once again, transfer learning outperformed simpler machine learning approaches for information extraction from text. All BERT-based models (with or without syntactic features) achieved substantially higher results for a binary detection of subjectivity than 4 ML algorithms. BERTd fed with additional features (syntactic features of marker) shows more consistent performance than a baseline BERTb model comprising of BERT encoder and a classification layer. Normalisation of meta features was found to boost performance for ML models. Increase of training data size had almost no impact on prediction performance for ML models and a negative influence for TL solutions in terms of macro 1 metric.
Baselines. The current work would benefit from a more thorough analysis of results of the proposed models against other systems developed on the employed dataset.
Generalisability of performance. The proposed architecture notes satisfactory performance on the utilised dataset (ranking 8th in the oficial leaderboard of the competition on English data), however, its robustness is yet to be tested, for example, in scenarios with diferent training datasets or multilingual data.
Ablation studies. Ablation analysis in future work could measure the impact of individual features on the final performance of the proposed subjectivity detectors. International Conference on Information Integration and Web-based Applications & Services (pp. 15-24). [12] Karimi, S., & Shakery, A. (2017). A language-model-based approach for subjectivity detection.
Journal of Information Science, 43(3), 356-377. [13] Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word–emotion association lexicon.
Computational intelligence, 29(3), 436-465. [14] Nargesian, F., Samulowitz, H., Khurana, U., Khalil, E. B., & Turaga, D. S. (2017, August). Learning
Feature Engineering for Classification. In IJCAI (Vol. 17, pp. 2529-2535). [15] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., . . . others. (2011).
Scikit-learn: Machine learning in Python. The Journal of machine learning research, 12, 2825-2830. https://scikit-learn.org/stable/ [16] Reilly, J., Zamora, A., & McGivern, R. F. (2005). Acquiring perspective in English: the development of stance. Journal of pragmatics, 37(2), 185-208. [17] Ruggeri, F., Antici, F., Galassi, A., Korre, K., Muti, A., & Barrón-Cedeño, A. (2023). On the Definition of Prescriptive Annotation Guidelines for Language-Agnostic Subjectivity Detection. Text2Story at ECIR, 3370, 103-111. [18] Sagnika, S., Mishra, B. S. P., & Meher, S. K. (2021). An attention-based CNN-LSTM model for subjectivity detection in opinion-mining. Neural Computing and Applications, 33(24), 17425-17438. [19] Simchon, A., Brady, W. J., & Van Bavel, J. J. (2022). Troll and divide: the language of online polarization. PNAS nexus, 1(1), pgac019. [20] Vieira, L. L., Jeronimo, C. L. M., Campelo, C. E., & Marinho, L. B. (2020, November). Analysis of the subjectivity level in fake news fragments. In Proceedings of the Brazilian Symposium on Multimedia and the Web (pp. 233-240). [21] Vinodhini, G., & Chandrasekaran, R. M. (2012). Sentiment analysis and opinion mining: a survey.
International Journal, 2(6), 282-292.