Over the years, psychologists have examined human personalities to understand their behaviours. This analysis has demonstrated the existence of several factors that are highly correlated to their conduct that supply clues about thought patterns. Consequently, in recent years the automatic prediction of personality traits has received considerable attention from the community. Thus, EVALITA proposes an evaluation campaign of Natural Language Processing, which suggests the PoliticIT task, which aims to recognise Twitter users' political polarity by analysing their comments. In particular, this task is composed of two subtasks focused on binary and multiclass classification problems. In this work, it is proposed a system which utilises Deep Learning architectures to address these subtasks. Three diferent pre-trained versions of the BERT transformer model are employed for each task. The outcomes of each generated model reached acceptable scores on the binary classification problems, but its performance dropped slightly on the multiclassification problem. In binary classification, 0.77 and 0.72 were achieved in gender and ideology tasks, respectively, and 0.56 in the multi-class.
nizing the political polarity in Italian texts [
Social Media has revolutionized the way that people the datasets’ content delivered for each subtask and the
interact, providing an open and direct path for commu- proposed system’s architecture. Section 4 analyses the
nicating with each other anytime and anywhere and results achieved on each subtask in the challenge.
Fifor other varied purposes like enjoyment or simply in- nally, Section 5 compiles the findings gained facing this
formation access [
This article describes the approach submitted to the challenge proposed in the EVALITA campaign for recog
is strongly related to their personal traits [
As a result, it provides three diferent outcomes, the resulting CSV file, the main classification metrics and the confusion matrix.
The dataset was composed of tweets harvested from
Twitter user accounts, whose political afiliation could be
easily deduced since they have a clear link to the party they
belong to [
The distribution analysis shows the imbalance of the dataset on each task. The highest is on the practice
This section details the outcomes obtained on each subtask delivered in the challenge. The metrics selected to assess the performance of the proposed systems in the challenge are precision, recall, and F1-score. Table 2 scrutinizes the results achieved by the proposed architecture on each classification task.
2https://huggingface.co/dbmdz/bert-base-italian-xxl-cased 3https://huggingface.co/m-polignano-uniba/bert_uncased_L-12_H768_A-12_italian_alb3rt0 gender
ideology_binary
ideology_multiclass Total male female Train 23 25 48 37 11 48 10 27 1 10 48
Test 23 25 48 37 11 48 10 27 1 10 48
Evaluation Train 810 488 1298 578 720 1298 131 447 558 162 1298
Test 318 135 453 205 248 453 51 154 148 100 453
The table was divided into three zones, one per each that the text annotated by these labels, ’right’ and subtask delivered. The best results were achieved in ’moderate_left’, contains similar features to ’left’ classifying the political polarity, primarily on tagging since the model can not difer between them. Consetweets as a ’right’ in binary and multiclass classifica- quently, a more exhaustive study of the dataset is required tion problems, where both employed models obtained to identify precisely the most discriminant features bean F1-score of 0.8 and 0.81, respectively. If we look tween these labels and reduce the ratio of mistakes made. at the dataset distribution in Table 1, both cases did At the close of this section, it is compared the results obnot contain the highest number of samples. Therefore, tained by the architecture proposed against the three first from this behaviour, it can be deduced that, even when participants and the baseline delivered. Table 3 shows the dataset is non-balanced and the amount of train- an excerpt of the oficial leader board, which depicts the ing data is not abundant, both models could recognise F1-score obtained by these proposals. the correct label with a high percentage. Moreover, With regard to the baseline, which combines the Bag these high results suggest that the samples found in the of Words (BoW) and the logistic regression, it achieves dataset and used during the training phase contained better results than our proposal on the binary task of more discriminate features, improving the models’ dis- ideology, reaching 0.81 against 0.77. However, it is obcrimination capacity. Conversely, the low results ob- tained much better results in the other tasks, doubling tained on ’ideology_multiclass’, where the preci- the performance in the multi-classification task. This sion dropped to 0.54 and 0.65 in ’moderate_right’ imbalance between the two approaches reveals the inand ’moderate_left’, respectively, reveals that de- fluence of the choice of features on the behaviour of the spite the highest number of training data, the outcomes techniques since it is compared two ways of extracting were not extremely high, indicating the characteristics features, a set of isolated words versus embeddings, in extracted from tweets were not discriminant enough to other words, words versus semantics. The higher results teach the model how to recognise these labels during obtained by the baseline show that the selected features the classification tasks. Finally, to have a deeper under- are highly discriminative and influence the highest restanding of the models’ behaviour, Figure 2 shows the sults. The lower results show exactly the opposite. There confusion matrices of each classification task. are similar features assigned to these classification tags
Starting for the gender task, the confusion matrix re- that cause the approach to make mistakes, which has a veals that the model had great dificulties in character- negative impact on the results obtained. On the other izing tweets written by males since it misclassified 68 hand, the results obtained by the first three proposals in against 42 mistakes made in female classification. For the ranking are very far from the results obtained by the the political polarity identification, in the binary clas- other proposals. Only in the multiclassification task, the sification problem, the models made more mistakes in proposed architecture can approximate the performance recognizing right than left. In the multilabelling task, of the third classified opponent, which means there is the errors were in the label ’left’ when the model still work to be done in fine-tuning the architecture and tried to diferentiate this against ’moderate_left’ extracting features. and ’right’, making 44 and 24 mistakes, respectively.
From this amount of elevated errors, it can be deduced
F1Ideology Binary
F1Ideology Multiclass TuebingenPoliticIT INFOTEC-LaBD extremITA ...
NLP_URJC UMU_TEAM
an evaluation campaign focused on Natural Language Processing and Speech challenges for the Italian language in 2023, promoted by the Italian Association of Computational Linguistics (AILC). In particular, this work faced the PoliticIT challenge, which motivates the development of smart systems capable of recognizing the ideological polarity in italian texts. The challenge included three subtasks that address three diferent classification problems, two binary, where it is demanded to categorize gender and political polarity and one multiclass, where the political polarity was extended from two to four diferent classes. To deal with these tasks, a system based on the Transformers models. The system includes three versions of the BERT model pre-trained using three datasets. The best performance obtained by the system was on the binary political polarity classification subtask. Not too far are the remaining tasks, in which the system gains relatively similar scores. In spite of the results obtained being quite elevated, there is still a significant range of improvement.
In future work, several lines would be interesting to explore for extending the architecture presented here. Firstly, it would be interesting to use augmentation techniques to study their efects on the Transformers models selected. A more detailed dataset study would be interesting to address, primarily to identify discriminant features that can boost the system’s performance.
grants: PID2021-125709OA-C22, funded by MCIN/AEI/10.13039/501100011033, and “ERDF A way of making Europe”; P2018/TCS-4566, funded by Comunidad de Madrid and European Regional Development Fund, and “Programa para la Recualificación del Sistema Universitario Español 2021-2023”.