Introduction

Currently, the proliferation of news sites and the widespread use of social networks have revolutionized the way news is consumed, giving rise to new forms of journalism [2]. However, these changes have introduced several challenges, including the proliferation of misinformation and fake news, the formation of "echo chambers" where news consumers limit their exposure to different points of view, and emerging concerns about the quality and integrity of journalism [3]. A common element in many of the identified challenges is the need to distinguish whether a news author is sharing objective information or expressing their own opinions, desires, or biases [4] [5]. The goal of Subjectivity Detection (SD) is to develop computational systems capable of implementing a binary classifier that can determine whether a text is objective or subjective.

CLEF2024 CheckThat! Lab Task2 [1] provides an opportunity to work on the challenges associated with subjectivity detection. This task aims to construct a binary classifier that can identify whether a text sequence, in the form of a sentence, is subjective or objective [6]. For the execution of Task2, the organizers have published five datasets in different languages (English, German, Italian, Bulgarian, and Arabic), plus an additional dataset that combines the previous five languages for the multilingual version of the task. The evaluation of the results presented will be carried out through the macro-averaged F1 between the two classes. This paper begins with the "Related Work" section, where a comprehensive review of previous research and studies relevant to the topic is conducted. This is followed by the "Data" section, which provides a detailed description of the structure and characteristics of the dataset provided for the Task2. The "Approach" section outlines the phases and techniques employed to conduct the research. In the "Results" section, the findings obtained from the implementation of the models used are presented and analysed using the metric macro F1. Finally, the "Conclusions" section provides a summary of the results, discusses the implications of the research, and suggests possible directions for future research.

Related Work

According to Liu [7], Subjectivity Detection (SD) is a field of study traditionally encompassed within a broader field known as Sentiment Analysis (SA) also referred to as opinion mining. Sentiment analysis is the field of study that analyses people's opinions, sentiments, evaluations, appraisals, attitudes, and emotions towards entities such as products, services, organizations, individuals, issues, events, topics, and their attributes. Sentiment Analysis is an area of research deeply studied in the last two decades.

Chaturvedi [8] categorizes methods for subjectivity detection into two main types: traditional syntaxcentered NLP methods and semantics-based NLP approaches. Syntax-centered NLP can be broadly divided into three main categories: keyword spotting, lexical affinity, and statistical methods. The major issue with these methods is that they are highly language-specific and require the existence of databases and resources for each language in which they are to be applied. To address this issue, solutions such as translating content between languages lacking these resources and languages like English, which have a wealth of resources, have been adopted. However, the translation of sentences can lead to the loss of lexical information, such as word sense, resulting in low accuracy [8].

On the other hand, semantic methods based on embeddings, RNNs, Convolutional Networks, and Transformers have gained significant relevance recently. They offer more accurate results than methods based on syntactic features, but they present their own challenges, as they require large datasets for each language in which we want to work. The creation of these datasets is complex and can generate problems such as ambiguity when classifying sentences [8] or annotator bias [9]. To avoid these problems, a recent paper published by F. Antici et al. [10] proposes annotation guidelines with the aim of unifying criteria and avoiding previous problems while experimenting with monolingual, multilingual, and cross-lingual Transformers scenarios between English and Italian languages.

Schumacher [11], starting with a multilingual BERT model, achieves good results for cross-language entity linking. From there, he explores Zero-Shot Cross-Lingual transfer between different languages and obtains robust results with a slight degradation when the model is applied to a language for which fine-tuning has not been performed. He concludes that although multilingual Transformer models make a good transfer between languages, issues remain in disambiguating similar entities unseen in training.

The objective of this paper is to address the question of the viability of using Zero-Shot Cross-Lingual transfer for subjectivity detection. To this end, we will fine-tune two multilingual Transformer models and analyze the results obtained within the framework of the CLEF2024 CheckThat! Lab Task2 [1]. To achive this goal, we will employ DeBERTa [12,13] and RoBERTa [14] for the monolingual approach and their multilingual versions, MDeBERTa [12,13] and XLM-RoBERTa [14], respectively for the multilingual approach. These models are evolutions built upon BERT that significantly enhance the results achieved by multilingual BERT, particularly in low-resource languages [15].

Data

The six datasets provided for the execution of Task2 exhibit varying characteristics in terms of size and distribution of objective and subjective sentences. In all datasets, objective sentences are labeled with the tag "OBJ", while subjective sentences are labeled as "SUBJ". As shown in Table 1, the Bulgarian dataset, which is the smallest, comprises a total of 1043 texts, 729 of which are included in the training dataset. In contrast, the Italian dataset contains a total of 2280 sentences, 1613 of which are in the training dataset. Furthermore, an examination of the datasets reveals a distribution bias in favour of the "OBJ" class across all datasets, although the extent of this bias varies depending on the language. For instance, while the bias is only 55.69% in favour of "OBJ" sentences in Bulgarian, this bias increases to 76.32% and 76.37% for Italian and Arabic, respectively. The multilingual dataset, the largest among all, is composed of a subset of sentences provided in each of the other datasets across all subsets (training, validation and test). However, due to its composition, it also exhibits a bias in favour of the "OBJ" class, accounting for 69.16% of the dataset.

Approach

In our research, we adopted a dual approach. Initially, we employed a monolingual approach that leveraged Transformers, placing the focus on the English language. Subsequently, we implemented a second phase, utilizing multilingual Transformers with a dual purpose: to enhance the results obtained in the first phase with the monolingual Transformers by increasing the size of the training set, and to verify the Zero-Shot Cross-Lingual transfer capabilities of the model. This means that a model that is fine-tuned in certain languages can be applied to other languages without any specific training.

Monolingual Models

The primary objective of the monolingual phase was to enhance the results provided by Task2 as a baseline. The baseline is based on a two-step approach. First, Sentence-BERT [16] is used to transform each sentence into a high-dimensional vector representation capable of capturing its semantic meaning. In the second step, a classifier is constructed by training a Logistic Regression model on the vectors generated in the previous step. To improve the results provided by the baseline, we utilized various Transformers such as DeBERTa v3 Large [12,13], RoBERTa Large [14] and BART [17] Large MNLI [17] that uses the entailment approach [18].

BART Large MNLI [17] is a Transformer encoder-decoder (seq2seq) model with a bidirectional (BERT-like) encoder and an autoregressive (GPT-like) decoder pretrained on English. BART is pretrained by ( 1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. BART is particularly effective when fine-tuned for text generation tasks (e.g., summarization, translation) but also performs well for comprehension tasks (e.g., text classification, question answering). In this study, we selected the checkpoint for bart-large after it had been trained on the MultiNLI (MNLI) dataset. Yin et al. [18] proposed a method for using pre-trained NLI models as ready-made Zero-Shot sequence classifiers. The method works by posing the sequence to be classified as the NLI premise and constructing a hypothesis from each candidate label.

Multilingual Models

In the second phase of our study, we utilized multilingual Transformers. Although these models have architectures and training procedures similar to their monolingual counterparts, they differ in that the corpus used for their pretraining consists of documents in many languages. The multilingual transformer models used in this study were MDeBERTa Base and XLM-RoBERTa Base. These models use masked language modeling as a pretraining objective and are trained jointly on texts in over one hundred languages. By pretraining on vast corpora across numerous languages, these multilingual Transformers enable Zero-Shot Cross-Lingual transfer. This implies that a model fine-tuned on one language can be applied to others without any additional training. The characteristics of these models MDeBERTa V3 Base: [12,13] mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was trained with CC100 multilingual data. The mDeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has 86M backbone parameters with a vocabulary containing 250K tokens which introduces 190M parameters in the Embedding layer. This model was trained using the 2.5T CC100 data as XLM-R.

XLM-RoBERTa: XLM-RoBERTa is a multilingual version of RoBERTa. It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. Following the work of XLM and RoBERTa, the XLM-RoBERTa or XLM-R model takes multilingual pretraining one step further by massively upscaling the training data [19]. Using the Common Crawl corpus, its developers created a dataset with 2.5 terabytes of text; they then trained an encoder with MLM on this dataset. Since the dataset only contains data without parallel texts (i.e., translations), the TLM objective of XLM was dropped. This approach beats XLM and multilingual BERT variants by a large margin, especially on low-resource languages [15].

The objective pursued through this cross-lingual approach is to utilize the same model across different languages, as the resulting linguistic representations can be well generalized across languages for various subsequent tasks, such as classification in our case. To this end, we have fine-tuned the multilingual models in English, German, and Italian, and applied them to the rest of the languages available in Task2, Arabic and Bulgarian.

Results

In the initial phase of this research, we focused on the English language, applying fine-tuning to various monolingual models with the aim of achieving optimal results as measured by the macro F1 metric, as outlined in the guidelines for Task2. We selected three distinct Transformer-based models for this purpose: DeBERTa Large, RoBERTa Large, and BART Large MNLI. We used Kaggle as the platform for training these models. The results of this process are presented in Table 3.

The models DeBERTa v3 Large and RoBERTa Large yield very similar results for the English language, with the best result being achieved by RoBERTa Large, scoring 0.74 on the test dataset. A much larger model, BART Large MNLI, which in principle should be capable of a greater understanding of language, performs worse, likely due to the dataset size not allowing it to generalize the characteristics of subjective language. As this model does not have an equivalent multilingual model, we have discarded it for the subsequent phases of the research. In any case, all trained models significantly outperform the baseline result provided for Task2 in English.

In the second phase of the research, we fine-tuned the multilingual models equivalent to the models selected in Phase 1 on a training dataset composed of the union of the data provided in Task2 for English, Italian, and German languages. Given the increased size of the training dataset, we used the base models, which are smaller in size, instead of the large models. Therefore, we replaced DeBERTa v3 Large with MDeBERTa v3 Base, and instead of RoBERTa Large, we used XLM-RoBERTa Base. As we can observe in Table 4, in all cases, the MDeBERTa v3 Base model outperforms the XLM-RoBERTa Base by a wide margin. In the case of the English language, we narrowly missed surpassing the result obtained by RoBERTa Large in the previous phase, but we matched the result obtained by DeBERTa v3 Large with a base model. The results obtained in the German and Italian languages are noteworthy, where we achieved scores of 0.85 and 0.83 respectively, significantly surpassing the baseline provided by Task2 for these languages.

In order to ensure the reproducibility of the results obtained with both the monolingual and multilingual approaches, Table 6 displays the models, training dataset, and hyperparameters used to train the models that achieved the best results when applied to the Final Test Dataset.

Finally, we sought to verify the Zero-Shot Cross-Lingual properties of both models by applying the models trained with the English, Italian, and German language datasets to the test datasets for the Bulgarian and Arabic languages without any specific fine-tuning for them.

We can observe in Table 5 that for both Arabic and Bulgarian languages, the results obtained in each case are worse than the baseline provided for both languages by Task2. Therefore, we must conclude that for subjectivity detection, there is no significant transfer of learning from one language to others without having seen examples in the second language during training. Consequently, we cannot rely on this feature of multilingual models for subjectivity detection in low-resource languages.

We believe that there could be several reasons why cross-lingual transfer has not worked, which should be analyzed in greater depth in subsequent studies. Lauscher [20] highlights the pretraining corpora size of the target language and the structural language similarity between languages as the main factors for the success of cross-lingual transfer.

In the final ranking for Task2, we achieved the second position out of a total of 15 participating teams in English language, with a final result for the Macro F1 score of 0.7372 and a SUBJ F1 score of 0.58. In Arabic, we obtained the fifth position out of a total of 7 participating teams, with a Macro F1 score of 0.4551 and a SUBJ F1 score of 0.25.

Conclusion

Our contribution to Task2 of CheckLab!2024 Subjectivity [1] aimed to determine, based on the provided datasets, whether it is possible to use the Zero-Shot Cross-Lingual feature of multilingual models to detect subjectivity in low-resource languages. The conclusion we reached is that it is not possible. However, given that this is a widespread problem that applies to all languages, we believe it would be interesting to continue investigating other non-multilingual Transformer-based approaches to help detect subjectivity in low-resource languages. Although the answer to our research question was negative, during the research process, we fine-tuned an MDeBERTa v3 Base model that achieved second place for English in Task2, with a score of 0.7372. It also achieved excellent results for German and Italian, with scores of 0.85 and 0.83 respectively, although we did not actively participate in the competition for these languages. As future lines of work, we propose adding Bulgarian and Arabic datasets, which we have not used to train the MDeBERTa v3 Base model, to see if adding more languages improves the model. It would also be relevant to analyze the use of Downsampling and Oversampling techniques to mitigate the bias present in the available datasets between objective and subjective sentences.