This paper presents the PoliticIT 2023 shared task, organised at EVALITA 2023 workshop. The task aims to extract politicians' ideology information from a set of tweets in Italian framed as a binary and a multiclass classification. The task is designed to be privacy-preserving and it is accompanied by a subtask targeting the identification of self-assigned gender as a demographic trait. The PoliticIT task attracted 7 teams that registered for the task, submitted results and presented working notes describing their systems. Most of the teams proposed transformer-based approaches, while some of them also used traditional machine learning algorithms or even a combination of both.
The study of the political discourse on Social Media
Platforms is of paramount importance in order to understand
where society is heading. Political discourse is by
definition ideologically based and political ideologies are
spread with discourse [
right wing, whereas openness to experience and agreeability were notably more correlated to the left wing.
Moreover, political ideology has a great influence in the
daily lives of each citizen. For example, [
biases, and prejudices [
0009-0006-9123-5316 (D. Russo); 0000-0003-3274-8825
(S. M. Jiménez-Zafra); 0000-0002-3651-2660 (J. A. García-Díaz);
0000-0003-2936-0256 (T. Caselli); 0000-0003-1582-6617 (M. Guerini);
[
The rest of the paper is organized as follows. Section 2 describes the PoliticIT shared task. Section 3 presents 3.1. Data Collection the dataset provided in the competition. Section 4 summarises the participant approaches. Section 5 shows the The dataset was collected from the Twitter timelines2 of results and a discussion thereof. Finally, Section 6 con- Italian politicians using the UMUCorpusClassifier [ 8], folcludes the paper with a discussion of the most interesting lowing a strategy similar to the one adopted in PoliticES outcomes of this task and possible future works. 2022 [7] and in [9]. In particular, the data refer to the politicians from the legislature XIX of the Italian Republic. The list of deputies, senators, and ministers was taken 2. Task Description from the institutional websites of the Italian Parliament3 and Government.4 All the politicians’ Twitter accounts The PoliticIT task is structured along three subtasks: were manually retrieved, as they are not reported on the institutional websites. We discarded politicians that did • Subtask A - Self-assigned Gender : Given a mes- not have a Twitter account or that were highly inactive on sage, the system must assign a value for the gen- this social media (i.e. whose accounts present very few or der of the author. The set of labels has been de- old tweets). The time window for the corpus compilation termined according to the personal web pages of was December 2022 as the oldest date, but no start date the politicians of the Italian Parliament. The task was set. In the first iteration, we compiled 371,822 tweets has been framed as a binary classification task from 468 politicians between November 2010 and Decemwith M for men and F for women. ber 2022. The average number of tweets per politician • Subtask B - Political Ideology (binary): systems is 794.49 but with a large standard deviation of 847.12, are required to determine the political orienta- which suggest that not all politicians are equally active tion of a message; the binary version of the task on Twitter. Thus, we decided to remove from the dataset presents two macro-categories: Left and Right. those politicians with less than 25 tweets, leaving a total • Subtask C - Political Ideology (multi-class): this of 408 politicians.
subtask presents a more fine-grained set of la- To balance the number of tweets per politician, we
bels for the political orientation expressed by a ifrst removed those tweets that are not written in Italian.
given message. In this case, we employed four To detect the language, we employed FastText language
labels: Left , Moderate-Left , Moderate-Right, identification model [
PoliticIT was organized through CodaLab.1 The run of tained mentions of news websites, by detecting linguistic
the task is divided into three phases: (i) Practice, (ii) Eval- clues within the text, such as the pipe symbol, which
uation, and (iii) Post-evaluation. In the Practice phase, is commonly employed by news websites to categorise
the participants were initially provided with a subset their content. Thirdly, we selected tweets based on topics.
of the training data in order to familiarise themselves An initial list of topics was extracted with BERTopic [
This section provides the reader with an overview of the dataset proposed for the PoliticIT 2023 shared task along with a comprehensive description of the modalities employed for creating it.
timelines/api-reference/get-statuses-home_timeline
3Chamber of Deputies: https://www.camera.it/leg19/28 Senate of the Republic: https://www.senato.it/leg/19/BGT/Schede/ Attsen/Sena.html 4https://www.governo.it/it/ministri-e-sottosegretari
We enriched the dataset by assigning to each politician a label indicating their political ideology. Political ideologies have been directly derived from the politicians’ afiliation party. In particular, the mapping from the politician to the political ideology was obtained through a two-step procedure: 1. Automatic labelling of politicians with their current political party afiliation. The party afiliation has been inferred from the parliamentary group to which the parliament party belongs. The data were extracted from the Italian institutional websites on October 31, 2023, thus they do not relfect changes in parliamentary groups following this date. 2. Mapping of the political parties to specific political ideology labels. The set of labels has been identified using Wikipedia. 5 We used four political ideology labels, i.e. left , moderate left , right, moderate right. Parties that are mapped in the centre, or cross-party, were nevertheless assigned one of the four aforementioned labels on the basis of their political alliances and the programme they presented during the 2022 Italian election campaign. The decision to “force” this classification was made to avoid excessive imbalance within each class. Therefore, we labelled “Movimento 5 Stelle” as left , whereas “Azione” and “Italia Viva” as moderate left . replaced with the @user token. Consequently, the text traits cannot be guessed trivially by reading a politician’s name and searching for personal information on the Internet. We also replaced the name of the political parties and of their Twitter accounts with the POLITICAL_PARTY token. • Clustering procedure - Subsequently, we created clusters of texts by mixing some of the extracted tweets in order to prevent ethical and privacy issues related to author profiling on Twitter. All the clusters are composed of tweets written by diferent politicians that share the same traits under evaluation, i.e. political ideology and selfassigned gender. For this, we divide the politicians into training and test in order to prevent that tweets from the same politician from appearing both in training and validation. To generate a cluster, we first set their demographic and psychographic traits, and then randomly pick tweets from users that share these traits. Thereby, each cluster represents “virtual users”, with their selfassigned gender (male, female) and political spectrum. For the latter, we labelled the data according to two axes: binary (left, right) and multiclass (left, moderate left, moderate right and right). At the end of this process, we obtained 1751 clusters with 80 tweets per cluster. It should be noted that the clusters from the training and test sets are independent to prevent machine learning approaches from identifying the authors rather than the demographic and psychographic traits.
approaches, depending on the source of the data. for the 3.3. Data Formats Italian deputies, gender was directly extracted from the institutional website, which allows the filtering of mem- The training and test sets are produced in a ratio of nearly bers according to this trait. The website of the Senate 75%-25%. Table 1 presents a summary of the distribution of the Republic does not clearly states the gender of the of labels per subtask. In no case, the labels are evenly members. In this case, employed linguistic cues present distributed. Male politicians are almost double the numon the personal page of each senator to infer the gender. ber of female politicians, and more than 200 politicians Specifically, we looked at the Italian verb “nascere” in are from the left wing than from the right. As regards its past participle form as “nato” for the male label and the multi-class ideology, moderate left and right are the “nata” for the female label. Finally, for the ministers, gen- most represented labels. der was manually assigned as the oficial Government Ultimately, each entry of the PoliticIT dataset comwebsite does not comprise this information and does not prises four elements: a cluster id, the self-assigned gender present helpful linguistic cues. In this case, ministers label and the political ideology labels for binary and multiwere labelled according to their biological sex. class classification. The dataset is organised at tweet level.
Subsequently, to build a privacy-compliant approach This means that each row represents one tweet. Each we took a two-step procedure including anonymization line also contains a cluster id to identify the cluster and clustering: to which the tweet belongs, as well as the demographic and psychographic traits of the cluster. Examples are provided in Table 2. The full dataset, including the gold labels, is available on Codalab 6. • Anonymized references in text - References to politicians within Twitter mentions were anonymized by replacing them with the token @user. The rest of the in-text mentions were also
A total of seven teams participated in the PoliticIT task, with all teams involved in each of the subtasks. The majority of the participants represented academic institutions. An overview of the system approaches can be found in Table 3, while Section 4.1 provides further details on the systems proposed.
ExtremITA [
ifers with linear and nonlinear kernels. Specific attention was given to the data representation. Indeed, the authors employed low-dimensional projections to concisely represent the dataset and its associated labels. These vectors were used for training an SVM classifier.
INGEOTEC. The team implemented diferent configurations of Bag of Words (BoW) classifier in all the subtasks. In particular, for gender and political ideology multiclass classification, INGEOTEC employed a stack generalization approach leveraging three BoW classifiers: two BoW classifiers pre-trained on 5M Italian tweets, and a BoW classifier trained on PoliticIT training set. Instead, for the political ideology binary subtask, the team proposed a BoW classifier trained on the training set.
proaches: an SVM-based approach and a Transformerbased approach. The former was the best-performing, i.e.,.e simple linear SVMs with sparse word/character n-gram features, trained separately for each task, only
Focusing on each subtask, it appears evident that bi- of the errors afect the classification of male politicians nary classification of political ideologies is relatively eas- into females. The best system, INFOTEC-LaBD, obtains ier when compared to the other two subtasks, with the 0.824 macro F1, with a positive Δ from the second best best results being 0.928 obtained by Tübingen. The more system (Tübingen) of 0.032 points. ifne-grained the distinction of political ideologies is the more challenging the task. This is not just an efect of having multiple classes and the distribution of the data, 6. Conclusions but it involves also the subtleties and nuances in policies across the four groups. In Figure 1 we display the In this paper, we have summarized the outcomes of confusion matrices obtained by the Tübingen team for the first edition of the PoliticIT task at EVALITA 2023. the three classification subtasks. Focusing on the errors PoliticIT targets the identification of the political ideolof Subtask C, multi-class ideology classification, we can ogy and gender of the author of a tweet. Political ideology notice that most of the errors concern misclassification is a psychographic trait that can be used to understand of the “extremes” (i.e., Left and Right) into the Moder- individual and social behaviour, and thus contribute to a ate Left category) rather than of the moderate positions better understanding of the society. The task introduces an innovative method concerning the anonymization of (tMhaot d-eartalteeasLteift n tvhse.irMcoodmemrautneicRaitgiohnt)o.nTThwisiitntedric-adt eifs- users to preserve privacy, allowing the investigation of ferences in the moderate political areas are stronger. A these sensitive topics in a fair and ethical way. further noticeable result is the fact that messages from PoliticIT has seen the participation of seven teams, politicians on the Right spectrum tend to be assimilated ifve of whom submitted a full report describing their approach. The results indicate that fine-grained political mostly with the Moderate Left and Moderate Right, suggesting that the narratives of moderate groups tend ideology distinction is more challenging than binary clasto assimilate issues and expressions of the Right. sification between two extreme values. This appears to
The identification of the gender of the author from a be due to the absorption of the narratives of the extremes tweet is also quite challenging even if it is framed as a into the moderate positions, especially from the Modbinary task. In this case, as illustrated by Figure 1a most erate Left . This is a datum that could provide political scientists and sociologists additional insights into the evolution of the Italian political systems. A further as- the Junta de Andalucía (DOC_01073). pect that raises interest concerns the errors in Subtask A, gender classification. In this case, most of the errors are male politicians classified as females. A deep exploration References of the communication styles of the two genres and their correlation with political afiliations is a promising path to better understanding this behaviour.
As expected, approaches based on Transformers are the trend solutions presented by participating teams, but some of them also used feature-based linear machine learning systems. It is quite impressive that the best performing team, Tübingen, uses linear SVMs with word and character n-grams as features weighting each feature using TF-IDF. This indicates that simple methods are still competitive and far from being fully overcome by neural approaches.
PoliticIT has been fully run on CodaLab. The task is now in the Post-evaluation phase and it is still open to submissions from other teams, making this resource freely available to the NLP community for research purposes.
Future work will investigate the addition of an extra subtask related to stance detection, to determine which authors are in favor of certain topics and which users are against. We can use this information to define clusters of users and to observe whether there is a relationship between the topics and the political ideology.