Political debates are vital in shaping public opinion and influencing policy decisions. However, understanding the complex linguistic structures used by politicians to ascertain their orientations and power dynamics can be challenging. In this paper we explore Natural Language Processing techniques for identifying political orientation and power structures in parliamentary debates. We introduce a Located Missing Labels-loss in order to train jointly to predict both power and ideology. Furthermore, our proposed method also trains to predict a third synthetically generated polarity label. Finally, we combine this training method with pre-processing steps including back-translation and meta data inclusion. Our results show that our method manages to improve upon conventional methods of fine-tuning.
Parliamentary debates play a vital role in political communication and society [
The ability to detect and classify political motives from speech may also be utilised when the speaker
is not forthcoming with their political agenda. Detecting hidden political motives in media such as
news reporting and advertisements may benefit society by providing transparency [
The complex nature of politics makes these debates challenging to understand [
The challenge of analysing and making classification on political speech may be approached from the
perspective of Natural Language Processing (NLP). NLP is a field in artificial intelligence that focuses on
analysing, understanding, and processing natural language data using computers [
Conference and Labs of the Evaluation Forum (CLEF) [
Research goals. The research goals1 for this project are inspired by and correspond to the two
sub-tasks of the Touché competition Ideology and Power Identification in Parliamentary Debates [
RQ2 Investigate what the best methods and practices are for identifying whether a parliamentary speech is made by a speaker in opposition or in power.
Evaluation The results of both research questions are evaluated against a test set provided by Touché
using a macro-averaged F1-score [
Here, we discuss previously explored techniques used for political classification within NLP and their relevance to our project. We also examine a range of studies showcasing the performance of models such as RoBERTa and Bidirectional Encoder Representations from Transformers (BERT) in tasks such as multi-lingual political orientation classification and stance classification of political tweets. Additionally, we explore domain-specific pre-training, back-translation, multi-label learning with missing labels, and other methodologies relevant to our research objectives.
Fine-tuned RoBERTa has demonstrated significant superiority over zero- and few-shot GPT-3 for
multilingual political orientation classification [
It has been shown that even when only fine-tuning a classification head, BERT has approached the
performance of few-shot GPT-3 models for stance classification of political tweets [
In a similar manner, a comparison of the performance of “Small Language Models” and modern LLMs
for sentiment analysis tasks has been performed [
RoBERTa has been shown to be state-of-the-art for propaganda classification [
New and large decoder-only models have been compared to encoder-only models for the tasks of
intent classification and sentiment analysis [
Domain-specific pre-training refers to the process of training a model on domain-specific texts before
ifne-tuning for a specific task within that domain [
For the task of multi-lingual political orientation classification, however, some authors recently
showed that domain-specific pre-training does not greatly impact results [
Back-translation involves the process of translating one text into another language, and then translating
the new text back into the original language [
Multi-label Learning with Missing Labels (MLML) has shown great utility for image classification tasks
[
Our two sub-tasks can be combined into a single multi-label classification problem, as will be shown
in Section 4. Similar approaches of combining tasks have shown to be beneficial. For the task of
peer-assessment evaluation, multi-task learning BERT has been shown to outperform its single-task
counterparts [
In 2021, amongst other years, CLEF organised a competition called EXIST [
R M L X a T R E B o R • Task 1: Identifying Sexist Content In this task, the system is supposed to perform binary classification. It must determine whether a given text (tweet or gab) exhibits sexism, whether directly, by describing a sexist scenario, or by criticising sexist behaviour. • Task 2: Categorising Sexist Content Following the identification of sexist content, the subsequent task involves categorising the content based on the type of sexism present.
The results, and techniques used in the two sub-tasks in the competition are compiled in Table 1. Some takeaways from the approaches in the competition comes from the datasets including Spanish and English text. When participants used Beto (Spanish version of BERT), it was exclusively used to analyse the Spanish texts which means that it had to be combined with other models for English. The same is true for BERT, almost all participants that used BERT for the English texts also ended up using other models for the Spanish texts. Lastly, the most common and best performing LLMs for handling multiple languages in this competition were mBERT and XLM-R.
# of speeches % of task data
% of all data
Opposition
Touché provides a dataset [
The organisers altered the dataset to provide less information than the original one, but also includes
an automatic translation to English for most non-English texts. The provided training dataset consists
of 6.5GB of text files [
is a unique (arbitrary) ID for each text. speaker is a unique (arbitrary) ID for each speaker. There may be multiple speeches from the same speaker. sex text is the (binary/biological) sex of the speaker. This information is collected from varying sources (typically data published by the respective parliament), and in some cases it may be unspecified or unknown. is the transcribed text of the parliamentary speech. Real examples may include line breaks, and other special sequences escaped or quoted. text_en is an automatic English translation of the corresponding text. This field may be empty for speeches in English. There might be missing translations for a small number of non-English speeches. label is the binary/numeric label. For political orientation, 0 is left and 1 is right. For power identification 0 indicates coalition (or governing party) and 1 indicates opposition.
Data imbalance There is an uneven distribution of data where some countries have more data than others. In addition, the label distribution among the countries is also skewed. For instance, in Figure 1 we can see that Serbia has more data for speeches connected to right wing parties while for the power dataset Serbia has more speeches from speakers in power. Table 2 represents the overall label distribution of the dataset.
The text lengths of each country also vary and are displayed in Figures 2 and 3 for the orientation dataset. Shorter texts may contain less helpful information for the predictions and thus, decrease the performance. Moreover, the limited context length of the models may not be able to capture all relevant information in the longer texts. For instance, BERT has a context length of 512 tokens which means that it can not process the entirety of most texts at once.
15000 10000 5000
0 15000 10000 5000
0 Croatia
Czechia
Denmark
Estonia
Finland
France Galicia
Great Britain
Greece
Hungary
Iceland
Italy Latvia Spain
Norway
Poland
Portugal
Serbia
Slovenia Sweden
The Netherlands
Turkey
Ukraine Power
Opposition
Left
Right
Shared information Even though the datasets are split, 47.5% of the speakers that appear in one of the datasets appear in both datasets, as shown in Figure 4. Moreover, 51.4% of speeches in the power dataset are made by a speaker who also appears in the orientation dataset. This amount equates to 72.2% of the total number of speeches in the orientation dataset.
Isolated speeches The dataset does not include the date of when the parliamentary speeches occurred. This means that modelling changes in party ideas over time or fully encapsulate how politics shift is infeasible. Moreover, simply connecting a speech to a certain ideology will not be as useful when it comes to predicting whether the speaker is currently in a governing position or opposition. This is because a country with a left leaning government one year could have a right leaning government the next.
Even though speeches are originally part of debates and exchanges in parliament, the dataset contains no data for connecting multiple speeches to a single conversation or exchange. This prevents approaches which would model an entire debate and label the participants in the debate rather than the individual speeches themselves.
Privacy in the dataset The dataset uses arbitrary codes for people’s names. This makes it dificult to check if our results match up with real-world politicians. Additionally, the test set for the orientation task does not contain any speakers in the original orientation dataset. As a result, a solution which attempts to connect specific speeches to the correct political parties using the speakers identity is infeasible.
Test set The test set for the orientation sub-task contains randomly sampled speeches by speakers who do not appear in the training set. The test set for the power sub-task does, to a large extent, contain speakers that also appear in the training set. However, speakers recurring in the test set will tend to
BasaqAnuudesHCtreioaruznetgryovBinealgBiuumlgCaartiaalonCiaroaCtziaeDcheinamaErsktonFiianlanFdranGcGreeaalitcBiaritaGinreHecuengaIcryelandItalLyatNvioarwaPyolPanodrtugSalerbiaSpSaTwihneeSdNleoenvtehneiralandTsurkUekyraine osnia B
Country have a diferent label distribution compared to the training set. For both sub-tasks, the test set contains approximately 2,000 speeches for each parliament, whose general label distributions resemble those in the training set. The test data follows a similar structure to the training data apart from the speaker_id and label field being hidden.
In the following, we describe how we processed and prepared the data as well as how we selected and trained models. An overarching view of our method is illustrated in Figure 5.
We decided not to extend the dataset using external sources. This was partly due to other parliamentary debates in our selection of countries either being unavailable or already in the original ParlaMint corpora. It would also require a lot of work in order to create properly labelled datasets in the same format as our base dataset. Finally, the amount of available data is of substantial size and a larger amount would put further pressure on the need for computational resources.
We generated the training dataset for each task by sampling 70% of the provided datasets, leaving
the remaining 30% of the datasets for the validation set. Note that the 70/30 split is a commonly used
rule-of-thumb which has shown some empirical optimality [
We used this split for evaluating the individual and combined parts of our model. Once we got the most efective techniques, we switched to a diferent split for our final ensemble model as explained in Section 4.6.
Missing translations Firstly, some English translations were missing in the provided Finnish dataset. Specifically, there were 271 translations missing in the training dataset and 88 in the test set. These 0.004 0.003 y it s n e D0.002 0.001 0.000 Finland Estonia Ukraine Spain Greece Galicia 500 0 500 1000 texts had to be manually machine translated. For the translations we used a Python package called mrTranslate. 4.1.1. Back-translation To address country distribution imbalance, we applied back-translation to the data from countries with fewer than 15,000 entries in the power and orientation datasets combined. We chose this threshold in order to strike a balance between improving the amount of speeches available for parliaments with less representation in the dataset and not increasing the training time excessively. The back-translation process involved translating English text to the original language and back, followed by translating the original text to English and back to the original language. We also used mrTranslate for the translations and appended the resulting data to the dataset, keeping all other fields unchanged from the original entry. Sometimes, however, the translations would fail. In these cases, we manually translated the texts using Google Translate. 4.1.2. Meta data inclusion By prepending each text with the corresponding country and gender of the speaker, i.e., “Germany, Female”, models got access to the available contextual information not included in the speeches themselves. We hypothesise that since parliaments and contexts of debates vary, so should their analyses. By giving the model access to all available metadata, i.e., all available context, we suspect that models might be able to better adapt their predictions. An example of adapting to such a context might be adapting the prediction of someone advocating for a certain law depending on what the law is currently in a given country. There might also be useful information in the meta data itself, such as gender making a politician more likely of belonging to a certain ideology in some countries. These examples are of course purely speculative and therefore part of the reason why we chose to include all meta data instead of selecting fields based only on our own speculations.
We merged the datasets by combining them where the label column was split into orientation and power labels. Despite being separate, both datasets contain shared elements, allowing the extraction of some data in the power dataset data into the orientation dataset, as explained in Section 3. We first verified that each speaker in the orientation dataset consistently had the same label. Once a speaker was confirmed as label in the orientation dataset, all texts by that speaker in the power dataset were also classified as label for orientation. When the datasets shared a text, the text from the orientation dataset was removed (since the text from the power dataset already had received the orientation label). This increased the number of orientation entries by 72.2% which equates to 51.4% of the original power dataset. 4.2.1. Located missing labels loss function Since the combined dataset has many missing labels, we needed to create a custom loss function. We calculated a filter tensor for each batch and label, marking rows with true labels as one and rows without true labels as zero. Then, we used this filter tensor as weights for the entries in the batch when computing the CE-loss. To account for the number of incomplete labels, we divided each label-specific loss by the sum of its filter tensor. By summing the loss for our two labels, we had created a multi-label loss function which could account for Located Missing Labels (LML).
Let us consider this loss function for a single label, i.e., orientation. Formally, let be the set of
classes (i.e., left and right) and the output predictions for all classes ∈ for entry in the batch.
, is then the predicted probability of class for entry in the input batch such that , ∈ [
1
∑︁ CELoss(, ) LMLLoss(, ) = ∑︀ = {︃1, ∈ {} 0, = − 1 (1)
As each label-specific loss is divided by the sum of its filter tensor, the resulting loss maintains a consistent size regardless of the number of samples with a true label in the batch. Consequently, the ifnal summed loss represents the combined loss for each task, regardless of its prevalence in the batch.
If training to predict both the power and orientation labels at the same time yields increased performance, then training to predict a third label might yield further improvements. We chose polarity as the label to add to our dataset, i.e., whether a text carries a positive, negative or neutral sentiment. We chose polarity since it is an efective metric for identifying trends in parliamentary speeches.
To obtain polarity labels for our dataset, we used version 0.2 of the instruction fine-tuned Mistral-7B,
as available under mistralai/Mistral-7B-Instruct-v0.2 on Hugging Face. We chose this model since it
outperforms other open source LLMs of similar or larger size, such as 7- and 13-billion parameter
versions of Llama-2 [
We double-quantised the Mistral model to a 4-bit normal float with a 16-bit float compute type to ift the model in memory and for faster inference. To fit the entire base prompt along with the text to label, we defined a context length of 4,096. To generate from the model, we used sampling beam search with 3 beams along with forcing the output to contain “Positive”, “Negative”, or “Neutral”. Finally, we assigned the polarity label as 0, 1 or 2 depending on whether the first word of the output was “Negative”, “Positive” or “Neutral” respectively, assigning − 1 otherwise.
To perform the polarity classification, we added three output nodes to our classification head, corresponding to each polarity class. To account for holes in the polarity data caused by failed generations, we also used our LML-loss to calculate the loss from the polarity predictions. We only added half of the polarity LML-loss to the base LML-loss in order to prioritise our two core tasks.
We restricted the models to those we could efectively train. Therefore, we excluded more extensive
and capable models, such as the 70-billion parameter version of Llama-2 [
We compared diferent modern transformer-based models to find which models performed the best for our task. Diferent models necessitated diferent hyperparameter values due to their difering sizes and designs. For all models, any implemented classification heads took the place of the last layer of the model as provided by their Hugging Face sequence classification implementation, leaving the method of pooling as implemented in the base model.
BERT and mBERT We evaluated the uncased version of BERT [
We also evaluated the uncased version of multilingual BERT, available under bert-base-multilingualuncased, on the speeches in their original languages. We chose this model because it is a multilingual version of BERT and because it demonstrated excellent results for multilingual tasks in similar competitions (see Section 2.5).
RoBERTa and XLM-RoBERTa We evaluated the large version of RoBERTa, available under
FacebookAI/roberta-large, since it has been shown to be state-of-the-art for similar tasks (see
Section 2.1). We also evaluated XLM-RoBERTa, available under FacebookAI/xlm-roberta-large, which is
RoBERTa pre-trained for multi-lingual tasks [
DeBERTa V3 DeBERTa V3 is an improvement upon the original DeBERTa model [
Gemma The 7-billion parameter version of Gemma outperforms similar models of equal size such
as Mistral-7B and Llama-2-7B [
Due to our limited computational resources, unfortunately, no experiments could be exhaustive. When we discovered that a certain hyperparameter value worked well, for instance using a warm-up period, we could not then aford to repeat experiments for all previous models to include this choice. Eforts where instead directed as to for each model balance the necessity to fit to the data in a manageable time frame with the desire not to cause excessive unlearning in the base model.
The main parameters which had to be adapted depending on the size of the models was the learning rate and the warm-up ratio. For instance, if a large model was trending downwards by the second epoch, then the learning rate might be lowered and/or a warm-up period added, in this way experiments where exploratory.
We set specific hyperparameters to increase training speed and reduce memory consumption to accommodate larger models. For instance, all models except BERT and mBERT used 16-bit floating point mixed-precision training to accelerate the training process. BERT and mBERT used regular full-precision training. Additionally, Gemma required LoRA and 4-bit quantisation to make fine-tuning feasible for us.
All models trained using a maximum sequence length of 512 tokens. Whilst we would have preferred to train with longer sequence length for the models which can handle it (DeBERTa-V3 and Gemma), this was not computationally feasible. However, to still utilise the longer context lengths DeBERTa-V3 can handle, it was re-evaluated using a maximum sequence length of 4096 tokens after training was complete. In this way we could train and evaluate the models more eficiently with 512 tokens and then afterwards leverage longer context lengths for only the final version of the model. We would have preferred to also re-evaluate Gemma using a longer context length, however, due to unforeseen limitations on computational resources, only DeBERTa could be re-evaluated using a longer sequence length.
Test data for orientation task: id, text, text_en, sex se1, txt1, txt1_en, M id, se6, se5, se8,
Training data for power task: speaker, text, text_en, sex sp1, txt1, txt1_en, M sp1, txt4, txt4_en, M sp1, txt3, txt3_en, M
Prediction:
After identifying the best performing models and training methods using our validation set, we created new training and validation sets which we used to re-train the selected models for ensemble modelling. These new validation sets contained disjoint selections of 10% of the available data, with a minimum of 5 samples for each country and label.
We chose this approach to increase the amount of data available to our models. By each model in the ensemble having a separate validation set, each model could be monitored for over-fitting whilst the ensemble as a whole had still trained on the entire dataset. We speculate that by using this approach, our ensemble will be able to leverage the entire training set. This is since if a single model has not been able to learn something useful due to the required data being in its validation set, the other models of the ensemble will have had access to that data.
We also chose to decrease the ratio of the validation set as we deemed the necessity of it being representative and reliable to be diminished once we had already determined and validated our method.
We created the ensemble by selecting the best performing multilingual and English-only models. We then trained two instances of the best performing of the two models as well as once instance of the other model using our newly created ensemble training and validation sets. For a given prediction, we ran each of these models and averaged their output logits before applying the sigmoid function and rounding to receive a final prediction.
Roughly 23% of the texts appearing in the provided test data for the orientation task also appear in the training data for the power task. Using this information we can extract the speaker id for the overlapping texts. Then, since the orientation label is always the same for each speaker, we can use these additional speeches to influence our prediction on the test set. For a given speech in the orientation test set, we averaged the logits of the examined model on that speech and the logits of our best performing model on all other speeches by the same speaker. In other words, predictions on the test data were averaged with those produced by our best model on speeches by the same speaker. This way if the text in the test data lacks clear ideological signals, we can instead rely on other texts by that speaker to make our prediction. The process is visualised in Figure 6.
In the following, we present the results of our method application to the provided dataset and evaluated models.
Sub-task
Macro-average F1-score
We fine-tuned the models using the transformers library for Python as provided by Hugging Face, running on a NVIDIA V100 with 32GB of memory. Also, if not mentioned, we set the hyper-parameter values to their default values as provided by the library.
The competition organisers provided a baseline of a simple linear logistic regression model. When we fitted this model to our training set and then applied this model to our validation set, we achieved the macro-average F1-scores as shown in Table 3.
To understand the efects of each component of our method, we fine-tuned BERT multiple times with diferent combinations of the components in our base method. These components, as presented in Section 4, were combined training, back-translation, polarity label extension and meta data inclusion. All runs used the same hyper-parameters, which can be found in Appendix C.1. Results are illustrated over the training epochs in Figure 7 and summarised in Table 4.
The results as a whole show that the combined training beats both the conventionally trained BERT and the baseline. The other components all individually improve performance of the combined training further. Finally, all components together yielded the best performance.
Combined training (CT) Training to predict both labels at once using our LML-loss showed significant improvements in comparison to when only training for a single label at a time. Back-translation (BT) Our results indicate that back-translation yielded an improvement in the orientation task over all epochs whilst only yielding an non-marginal improvement in the first epochs of the power task when training for both tasks using CT. To further investigate if back-translation helped improve the performance of countries with less data, we also visualised the results for each baseline
+CT +CT+BT +CT+PLE +CT+MDI +(all) parliament individually. Results can be found in Appendix D.2 and show that, on average, parliaments with back-translation saw a significant improvement whilst the remaining parliaments did not. Polarity label extension (PLE) Extending the combined training by adding a third label yielded an increase in performance over all epochs.
Meta data inclusion (MDI) Including the available meta data by prepending it to each text resulted in a improvement by the second epoch and onwards for both tasks. However, for the first epoch it caused a decrease in performance for the power task whilst not impacting the orientation task. Method components conclusions The examination of the components in our method seems to indicate that all components of our method are beneficial for BERT. This is especially clear due to the combination of two factors. The first factor is that the conventionally trained model had seemingly started to stagnate or over-fit whilst our proposed method was still improving through out all training epochs. The second factor is that our method exceeds the conventional fine-tuning already by the first epoch. In combination then, we may reason that our method provides an intrinsic advantage since it both converges faster, by the second factor, and does seems to cause less over-fitting or unlearning, by the first factor. These factors may also be reasoned as to guaranteeing that we actually make better update steps instead of just smaller (by the second factor) or larger (by the first factor).
In order to to validate that the improvement in performance on the orientation task from our method is not only due to the increased amount of data, we conventionally fine-tuned RoBERTa on the full set of available orientation data. We compare this to RoBERTa fine-tuned using the same hyperparameters, as found in Appendix C.2, but using our full method. Results are illustrated in Figure 8 and show that even when using the same training data, our method exceeds conventional fine-tuning over all epochs for orientation classification. As discussed in Section 2.1, fine-tuned RoBERTa has been shown to outperform very capable models and to be state-of-the-art for similar tasks. It is therefore very encouraging to note that our method managed to significantly improve upon the performance of ifne-tuned RoBERTa for political orientation classification.
The highest attained scores resulting from the application of our method on various models are shown in Table 5. Corresponding hyperparameters can be found in Appendix C.2. Results indicate that DeBERTaV3 was the best performing model, with XLM-RoBERTa being the best performing multilingual model. Gemma, which trained using LoRA and quantisation, manages to exceed the performance of BERT and mBERT but falls short of the other models.
To investigate the impact of re-evaluating DeBERTa-V3 using a longer sequence length, we also evaluated diferent sequence lengths. The results are shown in Table 6 and indicate that there was a significant improvement in performance from increasing the sequence length initially but that these increases diminish. The improvement by going from 512 to 1024 tokens was noticeable (+0.0059 and +0.0084) whilst the improvement by going from 2048 to 4096 tokens was minor (+0.0001 and +0.0003). These findings are not surprising, after all, successive increases in maximum sequence length add fewer and fewer tokens since more speeches become fully covered.
To investigate whether models pre-trained for multilingual tasks outperform their mainly Englishcomprehending base models, we compared BERT and mBERT as well as RoBERTa and XLM-RoBERTA. Each pair used the same hyperparameters internally (see Appendix C.2). The multilingual models processed the original texts whilst their counterparts processed the automatic translations. Results indicate that the multilingual models lag behind by a consistent amount. The macro-average F1-scores over the training epochs can be found in Appendix D.1.
Ensemble
In order to validate that ensemble modelling was a beneficial approach, we averaged the output logits of DeBERTa-V3 and XLM-RoBERTa, fine-tuned on our base training set, on our base validation set. The macro-average F1-scores of the yielded predictions, as seen in Table 7, show that the predictions of our best performing stand-alone model could be improved by also considering the outputs of our best performing multilingual model.
Baseline The macro-average F1-scores attained by the baseline model on the competition test set are shown in Table 8. This baseline model was fitted to the entirety of the original provided datasets. When comparing this baseline to the baseline on our validation set, we see a decrease in macro-average F1-score of 0.1152 and 0.0748 for the orientation and power task respectively. This indicates that the test set is much more challenging, which is not surprising due to the nature of its construction. For the orientation task the test set contains speakers that do not appear in the training set, and for the power task it contains speakers which appear with a diferent role than they do in the training data. The test set also does not share the same distributions in parliament representation, for further details see Section 3.
Final ensemble model Our final ensemble consisted of two DeBERTa-V3 models and one XLMRoBERTa model, fine-tuned using disjoint selected validation sets, as detailed in Section 4.6. We have made these models available on Hugging Face under oscpalML/DeBERTa-political-classification , oscpalML/DeBERTa-political-classification-alternative and oscpalML/XLM-RoBERTa-political-classification . Our final ensemble, averaging the logits of these models, yields macro-average F1-scores as seen in Table 8.
Additional data extraction The additional data extraction improved the performance on the orientation task. Our ensemble, without considering the other available speeches by a speaker, yielded a macro-average F1-score for the orientation task of 0.7854 whilst when utilising the other speeches the score increased to 0.7945.
In the following chapter, we discuss and reason about the efectiveness of the proposed solution. We further discuss the task itself and the limitations of our project.
Method efectiveness We show that our method improves performance for BERT and RoBERTa. Whether or not these findings translate to other models and tasks is, of course, a prudent question. The limitations of this project, enforced upon us by our limited computational resources, prevent us from examining this quandary fully. However, due to the relatively similar architectures of our examined models and the nature of our method, we hypothesise that the benefits of our method does translate. This is because our method does not closely depend on the internals of a model, instead aiming to provide a more representative loss function and better input data. We leave more extensive empirical confirmation for future research.
Multi-task training The combined training yields an increase in performance for the orientation task, which is not surprising since we extract additional orientation labels and therefore provide the model with more data. On the other hand, the fact that there is also a significant improvement for the power task is very intriguing. We speculate that this improvement is due to our LML-loss providing a more representative loss, which incentivises extracting features which are useful for both tasks and discourages over-fitting. This seems to be supported by the additional increase in performance provided by also adding the prediction of a polarity label. This increase in performance is even more impressive when considering that the polarity label was synthetically generated and likely to add at least some amount of noise. Since polarity likely shares some important similarities with features useful for our tasks, we speculate that our LML-loss was improved as to further incentivise cross-task useful features. Data preprocessing The observed impact of prepending available meta data to each speech, as shown in Figure 7, is not inexplicable. We suspect that the prepended sentence is very diferent from the pre-training material of the base model, since it is simply two words and does not follow the form of a regular phrase or sentence. This disruption, we speculate, might essentially confuse the model until it is able to learn it in later epochs. Once the model has understood how to interact with the prepended sentence however, it is able to leverage it into making better predictions.
It would be interesting for future research to compare the diference between adding new tokens representing the meta data and prepending the meta data in English, as we did. It might be the case that base models are able to leverage prior understanding of countries, be it their general political environment or some other aspect. On the other hand, it might also be the case that prior bias hurts the models ability to predict accurately and fairly.
Nature of tasks On the validation set, it is interesting to note that the linear baseline performed better on the power task than the orientation task given that the models utilising our method show the opposite behaviour. We may also note that whilst the diference is small, the power task benefited more from a longer sequence length. It is therefore not entirely unreasonable to suggest that the power task might rely more on specific words and phrases, as the linear baseline does. In other words, it might be the case that specific words are more important for predicting political power whilst how you speak in general is more indicative for predicting political orientation.
Diference between validation and test scores Another notable aspect is the discrepancy between the achieved scores on the validation and test sets. Without using additional data extraction, our top-performing single model achieved a macro-average F1-score on the validation set that was 10.1 percentage points higher than that of our best ensemble model on the test set. In contrast, the gap for the power task was just 3.6 percentage points. We attribute the majority of this gap to two factors: the diference in the distribution of the amount of parliamentary data and the nature of the test sets’ construction.
Since all parliaments have the same amount of data in the test sets whilst having greatly difering amounts of data in the training and validation sets, it is not surprising that the achieved scores difer. In the likely case that the models perform better on parliaments with more data, then the test set represents an increased presence of the harder-to-predict parliaments and a decreased presence of the easier parliaments. Regardless of this factor, just the diference in distribution itself likely also introduces a challenging condition. In other words, that there is a drift in label and parliament distributions is likely detrimental, even if the nature of that drift was not suspected to be particularly damaging. This is because the model likely to some extent relies on the statistical trends, such as favouring to predict the more common label when a speech is ambiguous.
The nature of the test sets’ construction may also account for the diference in discrepancy between our two tasks. The power test set largely contains the same speakers as our training data, just with a diferent power label. Given that we saw a relatively small decrease in performance, it seems that our models have been able to avoid over-fitting to a specific speakers power label. Such a case was likely aided by speakers exhibiting multiple power labels in the training data.
This behaviour is not shared with the orientation task. Since the orientation test set largely consists of speakers who do not appear in the training data, the gap in performance may indicate that our models have overfit to specific speakers. An additional factor is that the new speakers may cover topics our models have previously not encountered. If a speaker is mainly exhibiting a political idea which the model has not previously learnt the ideological connotations of, then the classification likely becomes much more challenging. It would therefore be interesting to investigate whether these previously not encountered speakers are contemporary to and covering the same topics as the speakers in the training set.
Weak ideological signals Some parliamentary speeches might not indicate strong political beliefs. They could solely cover practical proceedings, not expressing any opinions or making any arguments. These types of speeches are likely more challenging to classify, especially if the provided dataset does not exhibit clear rhetorical or linguistic diferences for labels within a given class. This likely introduces an upper limit on the performance of any model performing this task with similar data. Limitations As previously discussed, our limited access to computational resources determined what methods and models we could examine. This prevented us from examining large models such as the 70-billion parameter version Llama-3 or even the the 7-billion parameter version of Gemma. Not only were we limited in the selection of models, but also in the number of experiments which we could perform. More time and computational resources would have allowed us to attempt more techniques and further search for optimal hyperparameters. Techniques which could not be examined include using diferent learning rates for diferent layers and balancing the loss-function.
We were also limited by the data which we had access to. In real life, these speeches are not standalone but most often parts of exchanges and debates. The problem of weak ideological signals could likely be mitigated by considering all the speeches a speaker makes in an exchange together. By representing speeches as parts of a larger debate, not only could a model base its prediction on all of the speakers speeches, but also on the speeches made by the other participants in the debate.
In this study, we proposed a method for improved fine-tuning of LLMs for ideology and power identification. Our research questions where as stated below.
RQ1 Investigate what the best methods and practices are for identifying the political orientation in a parliamentary speech.
RQ2 Investigate what the best methods and practices are for identifying whether a parliamentary speech is made by a speaker in opposition or in power.
In answering our research questions, we found that modern LLMs are an efective approach for identifying both ideology and power in parliamentary debates. We further found that ideology and power likely share useful features and therefore fine-tuning to predict them jointly yields improved performance for both tasks. This improvement also extends to fine-tuning to predict synthetic labels, in our case polarity.
We also note that performance can be improved by making the context of the speech, as available by meta data, available to the models. Furthermore, back-translation can be utilised to boost performance of countries with a smaller presence in a given dataset. Finally, we found that English models predicting on automatic translations tend to outperform multilingual models predicting on the original languages but that an ensemble of both types of models is the best approach.
We would like to acknowledge our supervisor, Pablo Picazo-Sanchez, for his continuous guidance, feedback and assistance in this project. Furthermore, we want to express our gratitude for the computational resources provided by the Data Science and AI division at Chalmers University of Technology and University of Gothenburg. Finally, we would like to thank our examiner Moa Johansson for providing feedback on early and intermediary versions of this paper.
A. List of countries in the dataset
Label the polarity of the following text, similarly to the provided examples. Your answer needs to start with “positive”, “negative” or “neutral”, followed by a short justification for your answer. It is important that you only assign a positive or negative label if you are sure of your answer. Here is your ifrst example.
Text: The south-west was cut of from the UK last winter and Network Rail performed miracles in getting that line back up and running. I therefore find it extraordinary that reasons such as the weather have been used to excuse the chaos and incompetence of this debacle, particularly out of King’s Cross. Why did the Secretary of State feel that it was not necessary for Ministers to ask for a basic reassurance that an overrun on any of the big programmes could be managed? Why were contingency plans not in place, and why was the rail regulator warning not adhered to?
Negative. The text expresses frustration and criticism towards the handling of infrastructure issues, particularly the failure to address problems with the rail system despite previous incidents. It highlights perceived incompetence and lack of planning, suggesting a negative sentiment towards the situation.
Here is your second example.
Text: We are committed to ensuring that claimants receive high-quality, objective, fair and accurate assessments. The Department monitors assessment quality through independent audit. Assessments deemed unacceptable are returned to the provider for reworking. A range of measures, including provider improvement plans, address performance falling below expected standards. <p> I do agree with the hon. Lady, which is why we have been trying to work more strategically with Motability, thrashing through the issues I am very aware of on appeals and on matters such as when an individual leaves the country. We are looking to reduce the amount of time that appeals take and at what we can do with the running of the scheme so that the precise scenario she outlines does not happen.
Neutral. The text describes the commitment to ensuring quality assessments for claimants and outlines measures taken to monitor and address assessment quality. Additionally, it mentions eforts to work with Motability to improve processes and reduce appeal times. The tone is informative and focused on addressing issues, without expressing overt positivity or negativity.
Here is your third example.
Text: I congratulate the hon. Gentleman on bringing this much needed debate to the Floor of the House. Will he join me in paying tribute to local MND associations across the United Kingdom for the invaluable support they provide? I know of the excellent work of my local Leicestershire and Rutland association, having heard at first hand from a constituent and friend of mine, Ruth Morrison, about her tragic personal experience. The support that is available is of immense value and I hope the hon. Gentleman will join me in paying tribute to the work of those associations.
Positive. The text expresses gratitude and admiration for the eforts of local Motor Neurone Disease (MND) associations, highlighting the invaluable support they provide. It also encourages acknowledgment of their work, suggesting a positive sentiment towards their contributions.
Now here is your text to label:
The hyperparameters which where not left to their default values as provided by the transformers library are shown in table 9.
Hyperparameter
Value
3e-5 5 40 0.001 512
Eval. input length *Gradients where accumulated for 2 steps to simulate batch size 20 for DeBERTa-V3
The best performing hyperparameters used for models which did not utilise LoRA can be found in table 10, and the ones for models which did utilise LoRA in table 11. Not mentioned hyperparameter values where left to their defaults as provided by the transformers library.
The comparison of models and their multilingual counterparts is illustrated in fig. 9.
The diference in macro-average F1-score when fine-tuning combined training BERT with and without back-translation is visualised for each parliament in fig. 10.
The power dataset is illustrated inf fig. 11 and fig. 12.
BERT vs. mBERT, Power
BERT vs. mBERT, Orientation 1 f 0.02 0.020 0.015 0.010 e r o c s e g a r e v o r c a f10.005 -a0.000 m e h T e h
T IcelandGalicia LatviaEstoniaCataloneirazegovinaHungaryBelgiumFinlandSwedenDenmarkBulgariaCzechiaNorwayPortugal ItalySlovenia SpainGreeceFranceetherlandsUkrainePolandCroatiaAustria SerbiaTurkreeyatBritain
N
G
Country (a) Orientation
Delta macro-average f1 score from backtranslation on the power dataset H d n a ia n s o B
Back-translated data Non back-translated data e u q s a B
H d n a ia n s o
B translation on the orientation task. The countries are ordered by the number of speeches before back-translation, from left to right in increasing order.
AuesCtreioaruznetgryovBinealgBiuumlgCaartiaalonCiaroaCtziaeDcheinamaErsktonFiianlanFdranGcGreeaalitcBiaritaGinreHecuengaIcryelandItalLyatNvioarwaPyolPanodrtugSalerbiaSpSaTwihneeSdNleoenvtehneiralandTsurkUekyraine asqnudH B a
Country