In this paper, we present our approach to the PoliticES 2023 task. This shared task aims to extract user information from tweets in Spanish. We have developed systems based mainly on Deep Neural Networks (Transformers) to address the problem of binary and multiclass classification. Using pre-trained Transformer-based language models in this shared task poses an input length challenge because the amount of text per user significantly exceeds the input capabilities of common Transformer-based models. Our systems deal with input length problems by dividing the input into subsamples and performing the classification using a voting scheme. The results show the adequacy of our systems for the proposed task.
PoliticES 2023 is a shared task that aims to extract user information from tweets in Spanish.
There is a growing interest in this type of analysis of user profiles and the correlation between
some personality traits and political ideology, especially in the field of social networks. This task
was initiated last year in IberLEF 2022, and was called PoliticES 2022 [
The dataset[4] defined for this task is an extension of the PoliCorpus 2020 dataset [
The politicians’ accounts were selected from: (1) members of the government of Spain, (2) members of the Congress and Senate of Spain, (3) mayors of some important cities in Spain, (4) presidents of the autonomous communities, (5) former politicians, and (6) collaborators afiliated with political parties.
Journalists were selected from diferent Spanish news media, such as ABC, El País, ElDiario, El Mundo, or La Razón, among others.
The organizers created clusters of texts, mixing some of these extracted tweets. Each cluster groups 80 tweets written by diferent users that share all the traits under evaluation and are selected, favoring the diversity, including texts from diferent dates and topics. Every cluster is labeled with gender (male, female), profession (politician, journalist), and two types of political ideology: binary (left, right) and multiclass (left, moderate_left, moderate_right, right).
The dataset is composed of approximately 2800 diferent clusters. The training and test sets will be released (80%-20%), that is, 2250 clusters for training and 547 for testing. Cluster size approximately varies between 6000 and 21 600 words for the training set and 8100 and 21 600 words for the test set.
In this work, we wanted to evaluate the capabilities of Transformers-based [6] systems for the task of user profiling. However, as mentioned in the previous section, each training sample is made up of the text of 80 tweets from users with the same traits. This involves much more text than pre-trained models based on Transformers can handle. To deal with this problem, it was decided to divide each training sample into a set of subsamples with a size that would ift in the input layer of the Spanish pre-trained models that have shown better performance, MarIA [7] and BETO [8] available at HuggingFace [9] public hub. This can lead to mislabeling of some training subsamples, but we thought the benefits of using the whole dataset outweigh the drawbacks. In the inference stage, from a test sample a set of subsamples are generated that are individually labeled by the system. The final decision is made using a simple voting scheme. For each task, the most voted class is selected. Due to time limits, it has not been possible to test other more prominent voting techniques.
The decision of which pre-trained model to use and the strategy to generate the subsamples was made through a previous validation process.
For choosing the pre-trained model for each classification subtask, we nfie-tuned MarIA and BETO pre-trained models for the diferent classification subtasks, and measured the performance of each classification model with the validation set. All the BETO-based models obtained slightly better results than MarIA-based ones. For this reason, it was decided to use BETO for all the ongoing experimentation. Regarding the strategy to split each sample, to generate each subsample, the largest number of complete tweets that do not exceed a previously determined maximum number of words are grouped sequentially. We tested the maximum word length in the range of 300 to 500. The best results were achieved with subsamples that did not exceed 450 words.
In most of the models trained for this shared task, a search process for the best hyperparameter configuration was carried out. To do this hyperparameter optimization, the optuna library was used. Table 1 summarizes the hyperparameters search space.
During the challenge, we published a total of 7 runs. Most of those runs followed the system architecture detailed in Section 3. However, Run2 was created with a classic approach using a Linear Support Vector Machine (SVM), and the input text was vectorized as a Bag of Words of up to 4-grams weighted with TF-IDF. This run was developed using the library Scikit-learn [10] and can be seen as a baseline for classical machine learning approaches.
To perform the fine-tuning of the pre-trained models, a stratified split of the training corpus was made. 90% of the corpus was used to fini-tune the models and the remaining 10% was used for the selection of the best epoch and the best hyperparameter configurations. Due to the size of the corpus, the stratified split could only be done taking into account the gender, profession and ideology-binary labels. In addition, to have greater variability in the learned models, two diferent random partitions were performed using the same stratified strategy but two diferent random seeds. In this way, it was possible to learn diferent models with two diferent partitions. • Run1: BETO models fine/tuned individually for each task with hyperparameter search only in multiclass tasks, profession and ideology-m, using the first random partition. • Run4: BETO models fine-tuned individually for each task with hyperparameter search in all tasks using the second random partition.
• Run6: For each task, the best model from Run1 and Run4 is selected.
Table 2 shows the hyperparameters used for Run1 and Run4. Since only these two runs introduce diferent fine-tuned models, we chose only to list them for simplicity. The rest of the runs were based on selecting a set of models from Run1 and Run4.
The use of diferent models for the ideology-binary and ideology-multiclass tasks can produce discrepancies in the labeling. For instance, for the same sample, the binary classifier can label a sample with the right label while the multiclass classifier can label this sample as left or moderated_left. To avoid this, we create an Ideology Discrepancy Correction (IDC) procedure, where the political wing (left or right) is synchronized between the binary and the multiclass labels. Thus, the binary classifier was prioritized on the ideology identification, and the multiclass label was overwritten by the binary one when there were discrepancies between them. For instance, if a sample if binary classified with the right label, and multiclass identified as left, IDC procedure changes the multiclass label to moderated_right. Three additional runs were obtained by applying the IDC procedure to the previous runs. By doing it this way, we can quantify the impact of the IDC on the system’s performance. The additional runs were: • Run3: Applying the IDC procedure to the output of Run1. • Run5: Applying the IDC procedure to the output of Run4.
• Run7: Applying the IDC procedure to the output of Run6.
Overall, our solution achieved the best performance of the challenge in two of the four classification tasks and in the average system performance. Interstingly, our approach had the best results in the binary classification tasks (Genre and Ideology) but achieved second or third place in multiclass tasks (Profession and Ideology). In the Ideology task, our best run achieved nearly the same score as the best run. However, the diferences between our run and the best run were more significant in the Profession task. Only by the results, we can not conclude whether our approach has some limitations in multiclass classification tasks posed by this challenge or whether it is just circumstantial.
In this paper, we have presented a user profiling approach to infer gender, profession, and political ideology from tweets written in Spanish. Our system contains Spanish Transformerbased models that were fine-tuned for each classification subtask. The fine-tuning process was done with a hyperparameter search, which helped to increase the overall system performance. A dificulty posed by the challenge was the overall text length of the user’s text when Transformersbased models are used due to the input length limitations of these models. We addressed these limitations by splitting the input into subsamples, classifying them, and performing a voting process to determine the final label for each sample. Also, we resolved discrepancies between the binary ideology classification and the ideology multiclass one, which further increased the system’s performance. Our solution took first place in most classification tasks and reached the best overall performance in the challenge, indicating our approach’s adequacy for the proposed tasks. In future work, we will explore some variations of the architecture, for example including adapter, and in the fine-tuning process. It is also interesting study other possibilities for tackling with the length of the input texts, for example, by modifying the voting mechanism.
This work is partially supported by MCIN/AEI/10.13039/501100011033, by the "European Union" and “NextGenerationEU/MRR”, and by “ERDF A way of making Europe” under grants PDC2021120846-C44 and PID2021-126061OB-C41. It is also partially supported by the Spanish Ministerio de Universidades under the grant FPU21/05288 for university teacher training.