The wide-spread use of social networks has given rise to subjective, misleading, and even false information on the Internet. Thus, subjectivity detection can play an important role in ensuring the objectiveness and the quality of a piece of information. This paper presents the solution built by the Gpachov team for the CLEF-2023 CheckThat! lab Task 2 on subjectivity detection. Three diferent research directions are explored. The first one is based on fine-tuning a sentence embeddings encoder model and dimensionality reduction. The second one explores a sample-eficient few-shot learning model. The third one evaluates ifne-tuning a multilingual transformer on an altered dataset, using data from multiple languages. Finally, the three approaches are combined in a simple majority voting ensemble, resulting in 0.77 macro F1 on the test set and achieving 2nd place on the English subtask.
Subjectivity is a feature of language and a form of bias, in which whenever a person is sharing information, it comes out skewed by the speaker’s own personal preferences, beliefs and views. In today’s interconnected world, where opinions and biases travel fast and far, subjectivity detection can be a very important piece in order to ensure information reporting is done in a clear, objective and unbiased fashion.
Specifically, subjectivity in news and media articles can be nuanced, subtle and dificult to identify. Detection of subjectivity in such texts can play an important role in identifying potentially misleading or malicious texts and in detecting fake news online.
In Task 2 of CheckThat! Lab at CLEF 2023 [
This paper explores the efects of fine-tuning a large pre-trained language model on the subjectivity task. Additionally, we have examined the angles of few-shot-learning and finetuning a sentence embedding model. Finally, an ensemble method is proposed to unify all three into one solution.
While sentiment analysis can be regarded as a classic NLP task with lots of research already
available on the subject, subjectivity classification is deemed to be a slightly less popular research
topic. [
In [
In [
In [
For the subjectivity detection task, datasets in 6 diferent languages were provided - Arabic, Dutch, English, German, Italian and Turkish. A total of 7 datasets were available - one for each language, and one for the multilingual version of the task. The English dataset contained a total of 1019 examples. 800 of the provided examples were labeled as training, the other 219 as validation. A baseline solution1 is provided by competition organizers, which consists of a sentence encoder model, producing sentence embeddings, which are then classified with Logistic Regression.
An interesting insight was that most of the sequences were relatively short. For the English dataset, the average number of words in a sequence was 23, while 90% of sequences consisted of 40 words or less. 1https://gitlab.com/checkthat_lab/clef2023-checkthat-lab/-/tree/main/task2/baseline
The English training set is imbalanced, with 64% of samples labeled objective and 36% subjective. This imbalance is not present in the validation set.
Three research directions were explored, each of them resulting in a separate solution. The final program used for submission is a simple majority voting ensemble of the three solutions. The ifrst research direction explores what is achievable using sentence embeddings. The second one looks at a few-shot-learning model and dual-stage fine-tuning. The third is based on fine-tuning a pre-trained transformer model, also utilizing training data from the other languages available for the task.
All evaluations of experiments are done on the English validation set, provided by the organizers. All research directions will be described in more detail in the next subsections.
Multiple experiments with sentence embeddings were conducted. All of them were based on
using a pre-trained sentence embedding encoder model [
A potential place for improvement was related to sentence embeddings dimensionality. Due to only having 800 training examples, embeddings of dimensionality 384 could prove challenging for a classifier. Using dimensionality reduction, information from embedding vectors can be further compressed in a way that could make it easier for classifiers to find a proper decision boundary.
Best performance was achieved using PCA with 110 remaining components (out of 384), which explained 92.5% of total variance. Experiments with diferent classifiers and dimensionality reduction are summarized in the first column of Table 1.
While dimensionality reduction can help classifiers, the sentence embeddings themselves were generally created for a very broad category of NLP tasks. Fine-tuning the embeddings encoder for subjectivity detection proved to be helpful for all of the tested classifiers.
Embeddings were fine-tuned using cosine similarity loss in a contrastive learning manner. The new similarity label of a pair of sentences consisted of two equally weighted components their original similarity and their label-based similarity. Label-based similarity is defined as 1 if the two sentences are from the same class and 0 otherwise. This can be summarized with the following equation:
Classifier∖Embedding Baseline (SBERT + LR)
SVM
ElasticNet
PCA + ElasticNet __(, ) = 0.5 * (, ) + 0.5 * (() == ()) (1)
To generate training samples, N objective and N subjective sentences were randomly selected. Training pairs were generated - each sentence was paired with every other sentence and a similarity label was generated using Formula 1. In total, 2N*(2N-1) training pairs were generated.
All of the tested classifiers performed better using the fine-tuned embeddings (with N=100) instead of original embeddings. All experiments with classifiers, dimensionality reduction and ifne-tuned sentence embeddings are summarized in Table 1. For both the original and the ifne-tuned embeddings, best performance was achieved through PCA and Linear Regression with elastic net penalty. All macro F1 scores are measured on the (English) validation set.
The second research direction explored what can be achieved with few-shot learning.
Experiments are based on the SetFit model from HuggingFace [
Experiments and results for this approach are outlined in Table 2. Results are similar to finetuning sentence embeddings encoder, but achieved while using a lot less information (samples) from the task-specific dataset, which showed promise of low variance and good generalization capabilities.
The third research direction was based on the idea of fine-tuning a transformer model to the
dataset provided. Lots of transformer models are available in the HuggingFace hub [
Table 3 summarizes the experiments and results when using the English dataset. Fine-tuning a DeBERTa-v2-large achieved the best performance on the validation set.
Additional experiments were performed using data from other languages. Multiple training
datasets were created. The first one consisted of all the available data for all languages. The
second one consisted of English, Arabic and Turkish - the languages who had published baseline
solutions with higher F1 than English. The third one contained training samples from English
and German translated to English using a neural model [
For all three datasets, multilingual transformer models were used - bert-base-multilingual
[
Table 4 summarizes the results. An xlm-roberta-base model was fine-tuned using all available data, achieving a macro F1 score of 0.83, which showed slight improvement over using Englishonly dataset. The best results were obtained by using English and German translated to English, iftted to xlm-roberta-base model.
Best performing solutions from all three research directions were chosen and formed into an ensemble (Figure 1). The final solution was a simple majority voting ensemble from the following solutions: • A fine-tuned sentence embeddings encoder, producing improved sentence embeddings (in the context of this task). The encoder output passes through dimensionality reduction (384 dimensions reduced to 110). The reduced embeddings are then classified through LogisticRegression with equally weighted L1 and L2 penalties, ‘saga’ solver, balanced class weights and 0.5 regularization constant. • A few-shot learning SetFit model trained using dual-stage fine-tuning procedure with
N=20 sentence pairs.
• An xlm-roberta-base model, fine-tuned on English and German translated to English.
The final submitted ensemble was able to achieve 0.85 macro F1 on English validation set.
The final solution achieved macro F1 of 0.85 on validation set and 0.77 on test set. This indicates generalization issues, likely stemming from overfitting the solution to the validation set.
Each of the three separate methods was analyzed on the test set, to identify possible causes for the significantly lower performance on the test set. All components of the solution perform worse on the test set than on the validation set, indicating a more dificult test set. The first method however incurs a bigger loss of performance than the others. This is likely caused by an issue in the fine-tuning procedure, resulting in the encoder model producing highly specialized embeddings, which lose a bigger part of their pretrained semantics than optimal and thus generalize poorly. Less examples used for contrastive learning, smaller learning rate, or using a holdout set to check for generalization issues could have mitigated this problem.
Results also indicate that the best performer is an xlm-roberta-base model trained on English and translated German. All test and validation F1 scores are summarized in Table 5.
Table 6 indicates that the solution is biased to expect more frequent ‘objective’ examples, hence the higher recall but lowered precision. This is likely stemming from the initial class imbalance in the English training set, where 64% of examples are labeled objective. While this imbalance doesn’t seem to have a major impact on final F1 scores, a more balanced solution could have been achieved by using any of the well-known class-imbalance techniques, e.g. sample weighing or choosing a diferent threshold.
Overall, xlm-roberta model trained on English and translated German seem to be the best performer. The few-shot-learning approach yielded decent test results. The fine-tuned SBERT encoder method seems to not be general enough and is reducing the performance of the full solution. Submitting predictions from only the transformer model would have resulted in 0.79 macro F1, which could have won the English subtask challenge.
As indicated by Table 5, the transformer-based solution proved to be the most efective and
robust out of the ones used in the ensemble. Further experiments can be conducted with newer
and more promising versions of existing transformer models. For example, DeBERTa-V3 [
Additionally, due to resource constraints, transformer models with ’xlarge’ and ’xxlarge’ architectures were not used for this research. For the same reason, very little hyperparameter tuning was performed. Using a larger model and exploring bigger hyperparameter space can potentially improve the results.
Finally, the transformed-based approach used for subtask 2A (English) can also be attempted and used for the other 5 languages in the task - Arabic, Dutch, German, Italian and Turkish.
This research is partially funded by Project UNITe BG05M2OP001-1.001-0004 funded by the OP "Science and Education for Smart Growth", co-funded by the EU through the ESI Funds, and partially financed by the European Union-NextGenerationEU, through the National Recovery and Resilience Plan of the Republic of Bulgaria, project No BG-RRP-2.004-0008.