This study focuses on multilingual parliamentary speech analysis, specifically identification and classification of the ideology of the speaker's party and their governing status. The approach used here provides valuable insights into the political dynamics of parliamentary debates, enhancing the understanding of legislative discourse. A Logistic Regression model (combined with Count Vectorizer) is employed, trained on a dataset comprising of diverse multilingual parliamentary speech. The model achieves an F1-score of 0.59 for ideology classification and 0.69 for determining governing status. The efectiveness of the model is demonstrated in the context of evaluating parliamentary speeches from multiple countries.
Analysing political ideologies has traditionally been a challenging task due to the lack of a detailed
dataset representing individual views. A commonly employed approach which has shown remarkable
prowess in capturing nuanced linguistic patterns by utilizing advanced language models (LLMs) like
BERT and GPT-4, as outlined in this study which analyzes parliamentary representatives’ ideological
positions [
Inspired by the success of these approaches in various previous studies, the work done here aims to extend and refine existing methodologies for multilingual parliamentary speech analysis. Building upon the foundation laid by these prior investigations, the accuracy and robustness of classification models can be enhanced by incorporating additional linguistic features and optimizing model parameters. By leveraging logistic regression as a reliable framework for binary classification, the aim is to deepen the understanding of the intricate interplay between political ideology, governing status, and parliamentary discourse, thereby contributing to the broader discourse on computational approaches to political analysis.
The dataset comprises a collection of speeches along with metadata that includes the speaker’s gender and a classification label. The following are the dataset’s attributes: • id: A unique identifier for each speech record. This attribute helps in referencing and tracking specific speeches within the dataset. • text: The original speech text in 28 diferent European languages. This attribute is crucial for analyses that require the original language. • text_en: The translated speech text in English. This attribute is useful for analyses where English is the preferred language for processing or interpretation. This attribute has been used as the primary source for analysis and was fed into the machine learning model for further processing. • sex: The gender of the speaker, indicated by ’M’ for male and ’F’ for female. This attribute allows for gender-based analysis and comparisons. • label: A classification label for the speech, with possible values ’1’ and ’0’. In the context of Sub-Task 1, this label indicates the the speaker’s party’s ideological stance: left-wing (0) or right-wing (1). In the context of Sub-Task 2, this label indicates where the party is in the present political structure: ruling party (0) or opposition (1).
Furthermore, on analyzing the dataset for the two sub-tasks, for sub-task 1 (orientation), an average of 10422 speeches per dataset were present, with an average of 5784 instances as left-wing and 4638 as right-wing. For sub-task 2 (power), an average of 8370 speeches were present, with 4445 instances as ruling party as 3925 for opposition party. This illustrates that the dataset for both tasks was welldistributed and balanced, which is crucial for the model to efectively understand the characteristics of the data.
Preprocessing involves extracting the relevant fields (text_en and label) from the dataset, and converting
the text data into a suitable format for the model by employing vectorization using CountVectorizer.
CountVectorizer is a class in scikit-learn [
The parameters used for the CountVectorizer tool are as follows: • lowercase: Convert all characters to lowercase before tokenizing. Set to True. • ngram_range: Range of n-values for diferent n-grams to be extracted. Set to (1, 1): only unigrams.
• analyzer: Level at which the input text will be tokenized. Set to ’word’.
The first model which used for experimentation was Bidirectional Encoder Representations from
Transformers (BERT) uncased classifier [
Hence, a diferent approach was chosen. The second model that was used for experimentation was
logistic regression. Logistic regression is widely applied across various domains, often demonstrating
superior accuracy compared to classifiers such as random forest and K-nearest neighbor in numerous
empirical studies [
The initial approach involved combining the training datasets of all the languages and using this aggregated dataset to train the logistic regression model. However, this method resulted in a notably low average training F1-score. Subsequently, the model was trained separately for each language, and upon analyzing the outcomes, it was found that this method provided a higher average training F1-score. Hence, the latter method was used for subsequent analysis and evaluation.
Logistic regression is a process of modeling the probability of a discrete outcome given an input variable. The most common logistic regression models a binary outcome- something that can take two values such as true/false, yes/no, and so on. Logistic regression is a useful analysis method for classification problems, where the goal is to determine which category a new sample is most likely to belong to.
The logistic function is represented by the following formula:
The model was trained on speeches in around 25 diferent languages, which were translated into English for the purpose of evaluation. In the training phase, the model achieved an average F1-score of 0.99 for sub-task 1 (orientation), 0.98 for sub-task 2 (power).
In the testing phase, the model achieved its highest F1-score of 0.83 for the Power task for the Greek
language, and 0.72 for the Orientation task for the Italian language. Additionally, the model’s F1-score
surpassed that of the baseline for several languages. On average, the metrics measured were 3 to 5
percent higher than those of the baseline model [
One possible reason for the improvement in results over the baseline model could be the incorporation of the hyperparameter C into the model. The value of C was set to 0.3, determined through random search, where the model was trained and evaluated across various C values. This value of 0.3 yielded the best performance. Therefore, proper regularization helped improve the model’s generalizability to new, unseen data by reducing overfitting.
The model was also analysed based on the parliamentary languages, and the top 4 highest performing languages of the two sub-tasks have been listed in tables 2 and 3.
In conclusion, the research demonstrates the eficacy of logistic regression as a reliable technique for binary classification in the nuanced domain of multilingual parliamentary speech analysis. Through meticulous analysis of the dataset and model training, the proposed model demonstrated the utility of the approach in interpreting key attributes of political discourse, namely party ideology and governing status.
The findings revealed compelling F1-scores, averaging at around 0.59 and 0.69 respectively for the two tasks of identifying party ideology and governing status. This highlights the reliability of Logistic Regression in capturing the inherent complexities of parliamentary debates, even in diverse linguistic contexts.
Looking ahead, further refinements and extensions of the approach hold promises in enhancing the predictive capabilities and applicability across a broader spectrum of parliamentary contexts. This includes exploring the integration of advanced language models (LLMs) such as BERT or GPT. By leveraging LLMs, it is possible to delve deeper into the complexities of parliamentary discourse, uncovering subtle semantic nuances and contextual cues that traditional methods may overlook. Additionally, the plan is to investigate novel techniques for fine-tuning LLMs on parliamentary speech data, as well as exploring ensemble methods that combine the strengths of multiple models. Through these endeavors, the aim is to develop a more comprehensive understanding of the intricate dynamics of legislative language and its implications for governance and policy making.
Due to the lack of computational resources and time, the model was trained with same features for both the sub-tasks. This work can be improved and extended by using diferent features for both the sub-tasks.