This paper describes the participation of the UMUTeam in the PoliticES shared task organized at IberLEF 2023 within the SEPLN conference. It is an automatic document classification task on clusters of texts, which consists of extracting the self-assigned gender and profession as demographic traits, and political ideology as a psychographic trait from a set of texts written in Spanish by several authors that share those traits. For this task, we have fine-tuned the MarIA, a transformer model proposed for Spanish, to create classification models for each feature. After several submissions for these tasks, our team ranked seventh out of 12 participants, with an average F1 score of 69.225% of all classification models.
In general, individuals tend to be reluctant to heed the advice and instructions of politicians
who do not coincide with their ideology. In extreme cases, individuals may show a strong bias
towards a particular political party while roundly rejecting others’ ideology. In this line, [
With the aim of detecting political ideology information from Spanish texts, the shared-task
PoliticES [
This work presents the participation of the UMUTeam in this shared task, which is based on exploring the fine-tuning of diferent pre-trained and Transformers-based Large Language Models (LLMs) to detect gender, profession, and political ideology. The rest of the paper is organized as follows. Section 2 presents the task and dataset provided. Section 3 describes the methodology of our proposed system for addressing the task. Section 4 shows the results obtained. Finally, Section 5 concludes the paper with some findings and possible future work.
The shared task PoliticES 2023, organized at IberLEF workshop, aims to extract political ideology information from a set of texts written in Spanish by several authors. Specifically, the organizers propose a task of identifying demographic traits (gender and profession) and political ideology as psychographic traits from a given cluster of texts (document classification). In the case of the identification of political ideology, the organizers have defined it as a binary problem and as a multiclass problem.
The dataset provided is an extension of the PoliCorpus 2020 dataset [
The training set consists of 180,000 tweets from 2,250 clusters and the test set consists of 547 clusters with a total of 43,760 tweets. The distribution of demographic and psychographic traits of each cluster is shown in Table 1.
This task involves identifying gender and profession (demographic traits) and political ideology (psychographic traits) of users in a given set of texts. The pipeline used to participate in this task can be described as follows. First, the dataset has been processed, and emoji features have also been added. Second, the training dataset was split into training and validation at the tweet level. Third, a classification model is created for each of the features using the fine-tuning approach. Fourth, having the classification models at the sentence level, two strategies have been evaluated to identify the demographic and psychographic traits of the users (at document level): (1) mode, which consists of predicting each user’s tweet individually and selecting the most repeated label among the results obtained with the classifier, and (2) highest probability, which selects the label with the highest probability. The system architecture is depicted in Figure 1
As mentioned above, tweets from the same user will have the same demographic and psychographic traits, Therefore, to carry out the problem of identifying these traits, we have created a sentence-level classification model for each of them with all the tweets and the distribution of demographic and psychographic traits of the dataset as shown in Table 2. In this case, we have divided the training set into two subsets (80%-20%): training and validation. The customized validation split is created using stratified sampling, in order to maintain a balance between the labels.
We utilized the fine-tuning approach of a transformer-based masked language model for Spanish
called MarIA [
This section describes the systems submitted by our team in each run and the overall results obtained in this shared task. It should be noted that each participating team was allowed to submit ten runs.
We sent two runs for this task. The results, and a brief description of each, are depicted in Table 3. The first run is based on the mode strategy, which consists of selecting the most frequent label obtained in the user’s text set for each feature, achieving an average F-score of 69.225%. The second run, which entailed the highest probability strategy, achieved an average F-score of 55.889%. It can be observed that the macro F1-score obtained with the mode strategy applied in diferent classification models for both demographic and psychographic features performed better than the highest probability strategy. In addition, the binary classification models (gender and binary ideology) outperformed the multiclass models (profession and ideology multiclass).
The oficial leaderboard for this task is depicted in Table 4. We achieved the seventh position in the ranking with an average F-score of 69.225%. The teams ELiRF VRAIN and HiTZ-IXA achieved the best position, outperformed our best run with an average F-score of 81.131% and 79.348%, respectively.
In order to perform the error analysis and check what kind of wrong predictions our system makes, a normalized confusion matrix with truth labels have been used, which consists of a table showing the distribution of the predictions of a model with respect to the truth label of the data. The confusion matrix of the system using the mode strategy is shown in Figure 2. It can be noticed that our model tends to confuse the female gender with the male gender, with a percentage of 37.95%. As for the prediction of users’ professions, the model usually confuses celebrities with journalists with a percentage of 92.86% because there are few celebrity texts in the training set (see Table 2). Regarding the identification of political ideology, it can be observed that the model has a good accuracy in the identification of left-wing with a percentage of 99.39%. However, it tends to confuse right-wing and left-wing political ideology, with a percentage of 30.45%. In the case of the identification of political ideology at diferent levels (multiclass classification), it is observed that by having more moderate left cases in the training set, the model has an accuracy of 90% in the identification of this class. However, the model tends to confuse left-wing ideology with moderate left (52.99%) and right-wing with moderate right ideology (58.21%).
These working notes summarize the participation of the UMUTeam in the PoliticES shared task (IberLEF 2023). We achieved a 7/12 on the mean of all F1-score (69.225%) for the demographic and psychographic feature identification models. For this, we used the MarIA fine-tuning approach with the processed dataset and emoji features.
As future work, we are planning to improve our pipeline using an expanded LLMs model with political speech, i.e., fine-tuning a Masked Language Model (MLM) model with political text (a) Gender (b) Profession (c) Binary ideology (d) Multiclass ideology and later fine-tuning this model for detecting political ideology. In addition, we are planning to ifne-tune other pre-trained Spanish models to see if they improve MarIA’s performance. The source code is available via GitHub.
This work is part of the research projects AIInFunds (PDC2021-121112-I00) and LT-SWM (TED2021-131167B-I00) funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR. This work is also part of the research project LaTe4PSP (PID2019-107652RB-I00/AEI/ 10.13039/501100011033) funded by MCIN/AEI/10.13039/501100011033.