This paper discusses the CLEF CheckThat! Lab Task 2 on Subjectivity in News Articles, and our approach on using back-translation to augment the minority classes in Arabic, English, Turkish, German, Italian, and Dutch to distinguish subjective and objective statements. While we find that back-translation works well for other tasks in the fact-checking pipeline, we find that it does not work as well for subjectivity detection. This paper begins to examine several reasons why back-translation as an NLP data augmentation strategy could inhibit subjectivity detection.
children, including one entitled What Is White Privilege?" is labeled as ’Objective’, rather than ’Subjective’. As the sentence contains specific, falsifiable claims, this seems to be a reasonable labeling. However, the characterization of the books as tools of ‘Leftist indoctrination’, is clearly a subjective editorialization on the part of the author. This highlights the inherent ambiguity present in the task and underscores a core challenge that the annotators, and the models both face in learning a clear decision boundary.
In this work, we describe the back-translation augmentation strategies and models employed
by Team Accenture’s submissions to Task 2. Team Accenture’s back-translation and transformer
approach yielded the 3rd highest submissions in Arabic, 4th in Turkish, 5th in Dutch, and 8th
in German and English. While back-translation has been shown to be an efective means of
NLP data augmentation to improve checkworthiness identification [
Table 1 shows the number of samples and unique word counts for each of the datasets provided. We see that Italian had the largest number of samples in training (1,613). However, Arabic had the highest count of unique words (12,181), while German (4,622) and Dutch (3,944) had the lowest. Assuming consistent data collection methodology and annotation standards across languages, we would hypothesize that a larger quantity of unique words would yield higheraccuracy models. The sample size of all languages in this task is relatively small compared to the other tasks in the CheckThat Lab.
As shown in Figure 1, all of the datasets provided by the CheckThat! organizers had label bias which skewed each dataset towards sentences labeled as ’objective’.
Transformer models utilize WordPiece tokenization schemes that are dependant on the model being evaluated. At the time of pre-training, the WordPiece algorithm determines which pieces of words will be retained, and which will be discarded. An Unknown (UNK) token is utilized as a placeholder in the lexicon, and used to represent WordPiece tokens received in novel input that did not get utilized at model creation.
The proportion of out-of-vocabulary tokens are have been shown to inversely correlates to
overall accuracy [
In this work, we utilize BERT and RoBERTa models. The Bidirectional Encoder Representation
Transformer (BERT) is a transformer-based architecture that was introduced in 2018 [
For the Arabic Dataset, we used lanwuwei/GigaBERT-v4-Arabic-and-English [
For experimentation and comparison to roberta-large, we also fine-tune the pre-trained model
on subjectivity/style classification task, cfl/bert-base-styleclassification-subjective-neutral [
For each language, augmentation and training were done via back-translation into the respective
language using AWS translation. We back-translated the minority class in each dataset, which
is always the subjective documents. We appended back-translated subjective documents to the
training set. In our 2021 experiment [
In this work, we fine-tune lanwuwei/GigaBERT-v4-Arabic-and-English at diferent levels of data augmentation and compare performances on the gold test set provided by the organizer.
Table 3 shows the BLEU score for each back-translation scheme. Table 4 show training sample size before and after data augmentation and Table 5 shows the number of new tokens acquired after back-translation for each language. The higher the score, the more consistent or similar the translation to the original text. For Arabic and Italian, BLEU scores decrease as more pivot languages are used for back-translation, as we would expect. As a perfect translation would not provide variation in the training samples, and a low BLEU score may not provide consistent variation, this may suggest there is a sweet spot to BLEU score in a NLP data augmentation task to provide diverse word selection but consistent translations.
For all BERT and RoBERTa models utilized across all languages, we added an additional meanpooling layer and dropout layer on top of the model prior to the final classification layer. Adding these additional layers has been shown to help prevent over-fitting while fine-tuning. We used an Adam optimizer with a learning rate of 2 − 5 and an epsilon of 1.5 − 8. We use a binary cross-entropy loss function, 4 epochs, and a batch size of 32.
Arabic Dutch English German Italian Turkish
SUBJ 280 OBJ 905 SUBJ 311 OBJ 489 SUBJ 298 OBJ 532 SUBJ 308 OBJ 492 SUBJ 382 OBJ 1231 SUBJ 378
OBJ 422
Table 6 and 7 contains all model performance on the test set provided by the organizers. We find that our Arabic model has an accuracy of 0.800 with a weighted average F1-score of 0.816. Our English model had an accuracy of 0.696 with a weighted average F1-score of 0.687. For Turkish, we had an accuracy of 0.788 and a weighted average F1-score of 0.784. German received an accuracy of 0.337 and an F1-score of 0.174. Italian had an accuracy of 0.689 and F1 of 0.706. Finally, our Dutch model had an accuracy of 0.646 and a weighted F1-score of 0.618.
Table 8 and 9 shows Arabic model’s performance on the gold test set with diferent level of data augmentation.
Unique tokens in source
Unique tokens in MT
New Tokens
in MT
We observe that a specialized style-classification model outperformed the RoBERTa-large model quite significantly as seen in Table 10 and 11. This is likely because for a subjectivity classification task there is a heavy emphasis on vocabulary and terminology, which is a lacking in the relatively small training set provided. The raw RoBERTa did not have enough training vocabulary to outperform a specialized model. We also observe a diminishing return when over augment with the Arabic training set. As mentioned before, vocabulary plays a key role and augmenting with several pivot languages may have afected the data quality, potentially removing keywords that determine subjectivity. Look at the example below of a document labeled subjective after only one translation from Arabic to English: "Are there any resolutions that the Security Council may issue to ensure that Egypt’s water The second round of back-translation (Arabic > English > Spanish > English) then produces: "Is there a resolution that the Security Council can issue to ensure that Egypt’s water quota in the Nile River is not afected?" And the third (Arabic > English > French > English) produces: "Are there resolutions that the Security Council could adopt to ensure that Egypt’s share of water in the Nile is not afected?" By the second or third translation, the tone of the statement has shifted towards much more objective. This results in much lower model performance. We can see the results of these experiments in Table 8.
Due to extremely low sample size on the subjective class, we augmented Arabic and Italian
training data three times. Table 12 shows the average cosine similarity score between each
translation results to the original and the weighted average sentiment score of the pivoting
English back-translation based on the Vader Lexicon [
Our paper suggests there may be a ’sweet spot’ in BLEU score for data agumentation for back-translation, where a perfect translation would not add suficient noise to the training data and a poor translation would not add suficient context. We would recommend exploration of the BLEU score space as an optimization problem in future work.
We have described the back-translation augmentation strategies and models employed by Team Accenture’s submissions to Task 2. Team Accenture’s back-translation and foundation model approach yielded the 3rd highest submissions in Arabic, 4th in Turkish, 5th in Dutch, and 8th in German and English. In future work, we hope to explore in more detail to what extent back-translation data augmentation can inhibit subjectivity detection systems.