at PoliticIT-EVALITA2023: Evaluating Transformer Model for Detecting Political Ideology in Italian Texts

Abstract

This paper describes the participation of the UMUTeam in the PoliticIT shared task organized at EVALITA 2023. It is an automatic document classification task on clusters of texts, which consists of extracting self-assigned gender as a demographic trait, and ideology as a psychographic trait through a set of texts written in Italian by several authors sharing these traits. For this task, we used the fine-tuning approach of a pre-trained transformer-based masked language model for Italian called dbmdz/bert-base-italian-cased to carry out the identification of diferent features. After several submissions for these tasks, our team ranked sixth out of 7 participants, with an average F1 score of 70.426% of all classification models. However, our binary political ideology classification model obtained the fourth-best result with an F1 score of 86.63%. Natural Language Processing, Transformers, Politic ideology detection, Large Language Model, Multiclass classification [1]. The relationship between personality traits and po- Italian by several authors sharing these traits. This task can guide our daily decisions, just like other psycho- by researchers at Google to address the language translaEVALITA 2023: 8th Evaluation Campaign of Natural Language ProCEUR

1. Introduction

Political ideology is considered to be a psychographic trait that can be used to understand individual and social behavior, including moral and ethical values, as well as inherent attitudes, appraisals, biases, and prejudices litical ideology was demonstrated in [ 2 ], in which the authors collected data from 21 countries and concluded that political ideology is related to the big five personality traits and that the results of this relationship vary across countries, especially as a function of the level of prosperity.

Political ideology has a great influence on society and graphic traits such as personality. However, these decisions are made both consciously and unconsciously, as the social and cultural environment influences our ideology. In general, however, people tend to be more reluctant to follow advice and directions from politicians who do not coincide with their ideology [ 3 ]. In some extreme cases, people may be strongly biased in favor of one political party and, at the same time, strongly sions and put people’s lives at risk by ignoring specific recommendations of the authorities. nEvelop-O

Thus, the aim of PoliticIT at EVALITA 2023 [ 4 ] is to extract information about users’ political ideology through text clusters in Italian. For this purpose, the task is divided into several objectives such as extracting selfassigned gender as a demographic trait, and ideology as a psychographic trait through a set of texts written in builds on a previous task called PoliticES presented at

IberLEF 2022 [5] where the dataset was an extension of

the PoliCorpus 2020 dataset [ 3 ].

Transformer models are a neural network architecture used in various fields of NLP and other machine learning domains. The idea of a Transformer network originated from a paper called Attention is all you need [ 6 ], developed tion problem. Unlike recurrent neural networks (RNNs) and convolutional neural networks (CNNs), which were widely used in NLP before the advent of Transformers, this model relies on attention mechanisms to process sequences more eficiently and capture long-range relationships between words in a sentence or the order of words in a sequence [ 7 ]. Pretrained models based on Transformers are language models generated through massive learn language patterns and features from the training data. This enables them to capture semantic, syntactic, and contextual information, developing a deep understanding of language in general. For example, BERT (Bidirectional Encoder Representations from Transformers) is a Transformer-based model that achieved remarkable accuracy and advanced the state of the art in various disagree with others, which may lead to irrational deci- training on large amounts of unlabeled text, aiming to © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Natural Language Processing (NLP) tasks [ 8 ]. Another advantage of pre-trained models is the ability to add specific layers to adapt them to particular tasks. In other demographic and psychographic features of the cluster words, the process of taking a pre-trained model on vast is shown in Table 1. amounts of data and continuing the training on a smaller aadnvdatnatsakg-espoefcfinifice-dtuatnaisnegt iiss tkhnaot witnalalos wfinse-lteuvnerinaggi.nTghtehe Gender Trait FMeamleale 84T18r08aining 31T13e85st 16T12o23ta8l ignegnearnadl alapnpglyuianggeitktnooswpleecdificgetaasckqsuwiriethd rdeulartiinvgelpyrsemtraaliln- Ideology binary LRMeiogftdhetrate left 755257088 221404858 977680836 amTohuinstpsaopfetrrapirneisnegndtsattah.e UMUTeam’s participation in Ideology multi-class LRMeiogftdhetrate right 141346172 15151040 242896582 this shared task, based on the fine-tuning of a pre-trained Italian Transformer model to detect the gender and politi- Table 1 cal ideology of users through their texts written in Italian The distribution of demographic and psychographic traits of (document-level classification). The rest of the paper is the clusters. organized as follows. Section 2 presents the task and dataset provided. Section 3 describes the methodology of our proposed system for addressing the task. Section 4 3. Methodology shows the results obtained. Finally, Section 5 concludes the paper with some findings and possible future work.

This task involves identifying the gender (demographic

trait) and political ideology (psychographic trait) of users 2. Task description in a given set of texts (document-level classification). The task addresses gender traits as a binary classification The shared task PoliticIT 2023, organized at EVALITA [ 9 ], problem and political ideology as both a binary and a is a document classification problem that aims to extract multi-class classification problem. The approach used to self-assigned gender as a demographic trait, and politi- solve the challenge of classifying both demographic and cal ideology as a psychographic trait from a given text psychographic features at the document level consists cluster [ 4 ]. The organizers propose political ideology as in creating a phrase-level classification model by finea binary classification problem and a multi-class classifi- tuning a pre-trained Italian model based on BERT [ 8 ] cation problem. The organizers created text clusters by called dbmdz/bert-base-italian-cased [ 10 ]. The system mixing some of these extracted tweets to avoid ethical architecture is depicted in Figure 1 and the pipeline used and privacy concerns about author profiling on Twitter. to participate in this task can be described as follows. Consequently, all clusters are composed of texts written First, the dataset has been processed, and the emoji in the by diferent users sharing all the features under evalu- text have been transformed into text through the emoji ation. The users of the dataset are labeled with their library1. Second, the training set has been divided into gender (male, female), and political spectrum on two two parts: training and validation. Third, classification axes: binary (left, right) and multi-class (left, moderate models are created for each feature using the approach left, moderate right, right). Regarding the tweets col- explained above. Fourth, having the classification models lected from each user, the organizers discarded retweets at the sentence level, two strategies have been evaluated and tweets that contain headlines from news sites and to identify the demographic and psychographic traits of removed tweets written in languages other than Italian. the users (at document level): (1) mode, which consists Moreover, they anonymized them by replacing all men- in predicting each user’s tweet individually and selecting tions of the politicians and other Twitter accounts men- the most repeated label among the results obtained with tions with @user. Furthermore, other entities, such as the classifier, and (2) highest probability, which selects political party references, are also replaced with @politi- the label with the highest probability. Next, some more cal_party token. The tweets that belong to each cluster details about the preprocessing and fine-tuning stages are selected favoring diversity, including texts from dif- are provided. ferent dates and topics. For this shared task, the dataset is divided into training and test sets (80%-20%). For these 3.1. Dataset preprocessing diferent classification problems, we have used the same approach, which consists in fine-tuning a pre-trained Tweets from the user have the same demographic and Italian model based on Transformer for diferent demo- psychographic features, so to carry out the task of idengraphic and psychographic trait classification tasks at tifying these features, we created a sentence-level classithe tweet level. Thus, the training set consists of 103,840 ifcation model for each feature with all tweets from all tweets from 1,298 clusters and the test set consists of 453 users as the training set. The distribution of demographic users with a total of 36,240 tweets. The distribution of 3.2. Fine-tuning approach

We utilized the fine-tuning approach of a transformer

based masked language model for Italian called dbmdz/bert-base-italian-cased2 to carry out the identification of diferent features. Based on the BERT base model, dbmdz/bert-base-italian-cased has been pre-trained using a recent Wikipedia dump and various texts from the OPUS corpora collection [ 11 ]. The final training corpus has a size of 13GB and 2,050,057,573 tokens. The ifne-tuning process consists of adapting and adding a classification layer to the model to perform the training of the full model. In this way, the model takes advantage of the pre-trained linguistic knowledge of dbmdz/bertbase-italian-cased and adapts it specifically for a particular classification task, which can significantly improve performance on that task. To do this, we have used

2https://github.com/dbmdz/berts

4. Results

This section describes the systems presented by our team in each run and the overall results obtained in this shared task. It should be noted that each participating team could submit ten runs.

We submitted a total of 2 runs for this task. The results and a brief description of each are shown in Table 3. The ifrst run is based on running the diferent classification models on the user texts and then using the “mode” strategy, which consists in selecting the most frequent label obtained in the set of user texts for each feature. This run obtained an average F1 score of 0.70426. The second run has the same structure as the first one but with a diferent post-processing strategy; in this case, the “high probability” strategy has been used, which consists of making a decision based on the highest probability of all the predictions obtained on the user texts. In this case, it performed worse than with the “mode” strategy, with an average F1 score of 0.49232.

To perform the error analysis and check in which case the models give wrong predictions, a normalized confusion matrix with truth labels has been used, which consists of a table showing the distribution of the predictions of a model with respect to the truth label of the

3https://huggingface.co/docs/transformers/index

Ideology binary M-R 5. Conclusion data. The confusion matrix of the system using the mode strategy is shown in Figure 2. It can be observed that our model tends to confuse the female gender with the male This paper summarizes the participation of UMUTeam in gender and right-wing with left-wing political ideology, the PoliticIT shared task (EVALITA 2023). We achieved and this may be due to the fact that there are fewer ex- a 6/7 on the mean of all F1 scores (70.42%) for the demoamples in the training set (see Table 2), which makes the graphic and psychographic feature identification modmodel predict the wrong case. In the multi-class classifi- els. For this, we used the fine-tuning approach of a precation model of political ideology in our model, it fails trained transformer-based masked language model for mainly in the left-wing prediction, which tends to predict Italian called dbmdz/bert-base-italian-cased4 to carry out moderate left with a probability of 58%, and the moderate the identification of diferent features. right-wing prediction, which tends to predict moderate As future work, we are planning to improve our left with a probability of 74.51%. pipeline using an expanded transformer-based masked

Figure 3 shows the confusion matrices of the system language model with political speech, i.e., extend a with the highest probability strategy. It can be observed Masked Language Model (MLM) model with political that this strategy tends to confuse right and left ideology text and later fine-tuning this model for detecting politin the identification of binary ideology, so in Figure 3b ical ideology. In addition, we are planning to fine-tune it is shown that there are 78.05% cases that have been other pre-trained Italian models to see if they improve mispredicted. As for the identification of multiclass ide- the dbmdz/bert-base-italian-cased performance. ology, the model performs better than the mode strategy in identifying left and moderate right with a probability Acknowledgments of 40% and 52.94%, respectively. However, it tends to misidentify moderate left and right ideology poorly, with This work is part of the research projects hit probabilities of 9.45% and 45.45% and a diference of LaTe4PoliticES (PID2022-138099OB-I00) funded 89.86% and 40.91% with respect to the mode strategy (see by MCIN/AEI/10.13039/501100011033 and the EuroFigure 2c and 3c). pean Fund for Regional Development (FEDER)-a

The oficial leaderboard for this task is depicted in Ta- way to make Europe and LaTe4PSP (PID2019ble 4. We achieved the sixth position in the ranking with 107652RB-I00/AEI/ 10.13039/501100011033) funded an average F1 score of 0.7042. However, our binary politi- by MCIN/AEI/10.13039/501100011033. This work is cal ideology classification model obtained the fourth-best also part of the research projects AIInFunds (PDC2021result with an F1 score of 0.8663. (b) Ideology binary

References

[1] B. Verhulst , L. J. Eaves , P. K. Hatemi , Correlation not causation: The relationship between personality traits and political ideologies , American Journal of Political Science 56 ( 2012 ) 34 - 51 . doi:https://doi. org/10.1111/j.1540- 5907 . 2011 . 00568 .x.

[2] M. Fatke , Personality traits and political ideology: A first global assessment , Political Psychology 38 ( 2017 ) 881 - 899 . doi:https://doi.org/10. 1111/pops.12347.

[3] J. A. García-Díaz , R. Colomo-Palacios , R. ValenciaGarcía , Psychographic traits identification based on political ideology: An author analysis study on spanish politicians' tweets posted in 2020, Future Generation Computer Systems 130 ( 2022 ) 59 - 74 . doi:https://doi.org/10.1016/j.future. 2021 . 12 .011.

[4] Russo, Daniel and Jiménez Zafra, Salud María and García Díaz , José Antonio and Caselli, Tommaso and Guerini, Marco and Ureña López , Luis Alfonso, and Valencia García, Rafael, Overview of PoliticIT2023@EVALITA: Political Ideology Detection in Italian Texts, 8th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2023 ) ( 2023 ).

[5] García-Díaz , José Antonio AND Jiménez Zafra, Salud M. AND Martín Valdivia , María Teresa AND García-Sánchez , Francisco AND Ureña López, Luis Alfonso AND Valencia García , Rafael, Overview of politices 2022: Spanish author profiling for political ideology , Procesamiento del Lenguaje Natural 69 ( 2022a ) 265 - 272 .

[6] A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit , L. Jones , A. N. Gomez , L. Kaiser , I. Polosukhin , Attention is all you need , CoRR abs/1706 .03762 ( 2017 ). URL: http://arxiv.org/abs/ 1706.03762. arXiv: 1706 . 03762 .

[7] T. Lin , Y. Wang , X. Liu , X. Qiu , A survey of transformers , AI Open 3 ( 2022 ) 111 - 132 . URL: https://www.sciencedirect.com/science/ article/pii/S2666651022000146. doi:https: //doi.org/10.1016/j.aiopen. 2022 . 10 .001.

[8] J. Devlin , M. Chang , K. Lee , K. Toutanova , BERT: pre-training of deep bidirectional transformers for language understanding , CoRR abs/ 1810 .04805 ( 2018 ). URL: http://arxiv.org/abs/ 1810 .04805. arXiv: 1810 .04805.

[9] M. Lai , S. Menini , M. Polignano , V. Russo , R. Sprugnoli , G. Venturi, Evalita 2023 : Overview of the 8th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Eighth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian . Final Workshop (EVALITA 2023 ), CEUR.org, Parma, Italy, 2023 .

[10] S. Schweter , Italian bert and electra models , 2020 . URL: https://doi.org/10.5281/zenodo.4263142. doi: 10 .5281/zenodo.4263142.

[11] J. Tiedemann , Parallel data, tools and interfaces in OPUS , in: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) , European Language Resources Association (ELRA) , Istanbul, Turkey, 2012 , pp. 2214 - 2218 . URL: http://www.lrec-conf. org/proceedings/lrec2012/pdf/463_Paper.pdf.