Machine translation (MT) models have become increasingly accurate and widely accessible for multiple languages in recent years. They can potentially lift the barriers to applying NLP tools and methods to previously unsupported languages and boost comparative cross-lingual research in digital humanities. This study empirically contrasts results obtained with source and target Slovenian ParlaMint corpus of parliamentary debates on topic modelling. It qualitatively compares three steps in topic interpretation: topic description, topic significance in subcorpora, and marginal topic distribution. The results indicate that the topic modelling on the target corpus only partially replicates the topic modelling on the source corpus, but the overlap is suficient to provide a starting point for the cross-country comparison.
[
Fortunately, machine translation (MT) models are increasingly accurate and freely available to the research community. They are a cost-eficient and fast method for converting almost any corpus to a language the researcher could understand. With state-of-the-art models approaching or sometimes even surpassing human accuracy 2[], machine translation helps alleviate language barriers for comparative research in multilingual text collections.
The main research question of this paper is to what extent do bag-of-words results, specifically
topic modelling, on machine-translated corpora correspond to the results on the native corpora.
Given that topic modelling relies on word distributions and not on the order of words, proper
grammar, and correct pronouns, in-context word-to-word translation accuracy is the most
important requirement for MT, which is already relatively high with existing approaches.
Similar research has already shown that machine translations can successfully capture topics,
similar to the source corpus3[], and that target corpora can be used in comparative research
[
We chose to work with a recently published ParlaMint corpus, which provides rich linguistic annotation and metadata. Moreover, we chose the Slovenian ParlaMint corpus as we are native speakers of the language and can accurately interpret the results. Slovenian generally has fewer language resources than English and is morphologically rich, resulting in poorer machine translation.
The paper qualitatively compares the outcomes of source topic models with their counterparts obtained from the target corpus. Topic overlap is estimated, not in terms of topic-term similarity, but on how similar the analytical results would be if using target corpus. Evaluation is done by comparing: a) topic interpretation in the source and target topic model, b) significant topics for pre-COVID and COVID period, and c) marginal topic distribution of both models.
Various techniques can be applied for a comparative analysis of topic models of multilingual
corpora. Mimno et al. 6[] propose Polylingual Topic Models (PTM), which can extract topics
for corpora in many languages, but they require an initial set of comparable documents. PTM
can be extended to unaligned documents, but not all corpora contain comparable documents.
Boyd-Graber and Blei 7[] further this idea by proposing multilingual topic models for unaligned
documents. When the documents in diferent languages do not cover the same topics, which
is often the case, Yang, Boyd-Graber and Resnik [
The alternative is using machine-translated corpora and computing topic models on those.
However, the results depend heavily on the quality of the translation. There are numerous
approaches to automatically estimating machine translation quality. Most rely on quantitative
assessment against a reference text, such as BLEU9[] and NIST [
However, such studies are few. Reber [
This contribution extends the paper from de Vries, Schoonvelde and Schumacher3][, which compare a gold standard human translation oenuparl data set with a machine-translated corpus. They use topic modelling with LDA to compare both text sets via the generated term-document matrices, which explicitly shows that target corpora can be successfully used for extracting topics.
We likewise estimate the machine translation quality empirically, but in contrast to the above paper, we focus on a qualitative perspective. Instead, we leverage our native knowledge of the Slovenian language to estimate how close the interpretation of topic modelling of the target corpus would be to the topic modelling of the source corpus.
The first data set 1 is ParlaMint-SI, a linguistically annotated corpus of parliamentary speeches
from the Slovenian parliament from 2014 onward1[
We took the data from 2019-01-01 onward, encompassing about a year of pre-COVID and a year of COVID speeches. We parsed the corpus into 18,476 utterances, each representing a single speech given in a session. We kept only speeches given by regular MPs, as these would correspond best to topics discussed in the parliament. We also removed speeches (utterances) shorter than 50 words, as these would typically be procedural remark1s5][. In the end, the ifltered corpus contained 6861 speeches.
The second data se2t is a machine-translated version of ParlaMint-SI version 2.0. We will refer to it as the “target corpus”. Machine translation was performed wiothpus-mt-zls-en model 3.
The source corpus already contains lemmas and POS tags attained with the CLASSLA pipeline
[
We kept only lemmatised nouns, thus removing a large portion of tokens. The reasoning is
that nouns suficiently reflect topics in parliamentary speeches, and they are easier to interpret
than, say, verbs [
In the end, the source corpus retained 3695 types, while the target corpus had 5127. More than 30% diference in types in the target corpus is already a significant discrepancy. The diference can be attributed to the diferent lemmatiser (personal names were lemmatised correctly in the 110.6084/m9.figshare.19248812 210.6084/m9.figshare.19258814 3https://huggingface.co/Helsinki-NLP/opus-mt-zls-en 4A manual inspection of token diferences between the source and target corpora revealed the dificulties of the lemmatiser to deal with Slovenian proper nouns and acronyms source corpus, but not in the target one), diferent POS tagger (which tagged certain words erroneously), and, indeed, faulty machine translation model (which sometimes creates random repetitions of words)5.
Compared with de Vries, Schoonvelde and Schumacher, we were stricter with the
preprocessing. We kept only nouns from the source corpus, which empirically gave the best resu6lts
and is similar to related work1[
We compare the practical eficiency of machine translations for comparative research of multilingual corpora on topic modelling resu7l.tsThe choice of topic modelling is in line with related work. However, we extend this with a qualitative comparison of the results. Namely, we wish to determine whether machine-translated corpus would give similar results to the native corpus. We use the Latent Dirichlet Allocation (LDA) topic model, a generative model that extracts topics based on word distributions. We compare the topic interpretation of the source and target topic model, the ranking of topic significance for pre-COVID and COVID subcorpus, and the marginal topic distribution of the two topic models.
The tasks generally correspond to typical analytical workflows in topic modelling research,
namely topic identification, contrasting topic frequencies in diferent periods or between parties,
and estimation of topic importance 1[
Furthermore, we aim to explore the quality of the target topic model for a language with
lesser resources, namely Slovene. With the proliferation of freely available yet high-quality
MT models, such as the OPUS collection from Helsinki NLP grou2p0[] and Facebook’s MBart
models [
We extracted 20 topics with Latent Dirichlet Allocation on TF-IDF weighted bag-of-word matrix.
Twenty topics is a suficiently large number to cover a wide array of topics that can be discussed
in the parliament while also being sensibly moderate to allow interpretation. Zhao et2a2]l. [
and Rosa, Gudowsky and Repo 2[
The results of topic modelling with the top 10 words describing each topic are detailed in Table3 for the target corpus and in Table4 for the source corpus (see Appendix). We manually 5Machine translation accuracy cannot be estimated on the ParlaMint corpus due to the lack of a gold standard. However, authors report a BLUE score of 25.6 and character n-gram F-score of 0.407 for Slovenian to English translation on Tatoeba corpus.
6We tried topic modelling on all tokens, NOUN+VERB+ADJECTIVE, NOUN+VERB and only NOUN and compared the results for 5, 10, 20 and 50 topics. The pipeline that yielded the best results was only nouns with 20 topics.
7The Orange data mining workflow for reproducing the analysis is available1a0t.6084/m9.figshare.19248806 .
Topic 13 Topic 17
Topic 8 Topic 15
Topic 8TopicT2o0pic 19ToTpoicp2ic 2
Topic 4Topic 18
translated the results into English and assigned topic names. We ussmeall caps to denote topics from the target corpus andbold to denote topics from the source corpus. 5.1. Comparison of topic modelling results Topics extracted from the source data are semantically more cohesive, as topics 1, 3, 18, and 19 from the target corpus represent a mix of two subtopics. For exampleT,opic 3 (Table3) from the target corpus mixes discussions on family policy (child, family, parent, allowance) with those on electoral process (election, constituency, voter). AlTsoop, ic 6 is a “junk” topic with unspecific words (i, t, something, someone).
However, topic modelling on the target corpus is able to identify certain overarching topics, namely the discussions on Sunday working hours of shops, issues on education, and taxes. Other topics have partial overlap (Figur1e), such asepidemic (Topic 14 and Topic 4), judiciary (Topic 11 and Topic 1), agriculture (Topic 5 and Topic 13), and credit management (Topic 17 and Topic 7). There are two pairs of topics which display high similarity in t-distributed Stochastic Neighbour Embedding (t-SNE) projection, but we were unable to determine why they would be deemed similar, namelyTopic 2 and Topic 20, and Topic 8 and Topic 19. 5.2. Evaluation of topic significance in subcorpora Apart from determining topic overlap, we were interested in how the target topic model can replicate a more complex analytical result. Such a task can be comparing the diferences between
Target
Topic 14: epidemic Topic 7: disabilities act
Topic 11: judicial
Topic 15: migration Topic 18: pensions & transport
Source Topic 4: epidemic Topic 3: health care Topic 11: firefighters Topic 17: migration
T1: judicial two subcorpora. The ParlaMint data set contains rich metadata with pre- and post-COVID speeches annotated. Thus we chose to compare topic prevalence in these two time periods. We determined which topics were more significant for the pre-COVID (label Reference) period and the pandemic period (label COVID) with the source corpus.
We used Student’s t-test to compare the diferences in the topic distribution in the reference and COVID subcorpora. Topics with the highest test statistic denote more strongly represented topics in a specific period. For example, Figure 2a shows that in the target corpus Topic 14, which is about the epidemic, was more frequent in the COVID subcorpus compared to the reference subcorpus. The same is true for the source corpus, where Topic 4 represents the epidemic and is also the highest-ranked topic. The two topics even share a similar test statistic, which shows that, at least for this topic, relative word frequencies were successfully retained in the machine translation.
Topic ranks are listed in Tabl1e. Besides the epidemic, judiciary and migration topics were among the five highest-ranked topics. The overlap of topic ranks is partial. Three out of five top-ranked topics were identified in both the target and the source corpus. Health care formed a separate topic from the epidemic in the source corpus, while in the target corpus, pensions and transport were collated in a single topic—certainly, even small shifts in word distributions afect topic models and the results extracted from them. (a) Topic 7 from target corpus (b) Topic 3 from source corpus 5.3. Comparison of marginal topic frequencies Finally, we observed marginal topic frequencies, showing which topics appear with greater probability in each corpus (Tabl2e). The target corpus sufers greatly from an overestimation of meaningless topics. At the top isTopic 16 containing procedural words, such as “law”, “article”, “amendment”, and “draft”. Topic 6 includes uninformative words, such as “t”, “thing”, “something”, and “someone”. Certain topics seem to be afected greatly by shifting word frequencies, such asepidemic, judicial and budget allocation, the latter completely disappearing from the target topic model.
Topic frequencies show a less promising picture of target topic models. As this particular target topic model seems to mix two topics often, topic frequencies will overlap less. The most represented topics in the source corpus correspond well to the parliamentary agenda, namely budget allocation, the COVID epidemic, infrastructure issues and pensions. Target topics do not reveal the same agenda, giving a less-than-clear picture of parliamentary discussions.
While certainly imperfect, machine translation can help researchers explore corpora in their non-native languages to some extent. The findings imply that machine-translated corpora can be used by researchers who are not fluent in a specific language but with limited success. In terms of topic modelling, LDA extracted topics, generally comparable with the source corpus, thus enabling a cross-country semantic comparison of parliamentary data sets in the English language.
On the example of ParlaMint-SI corpus, the topic model of the target corpus identified three topics, identical with the source topic model, while many topics at least partially overlapped. Certain topics in the target model were still relevant parliamentary topics, such as railway infrastructure, migration, and bank audits, even though they were not identified in the source corpus. Machine-translated word frequencies were diferent enough that LDA could not capture the same topics, but it did ofer other relevant sub-topics.
That said, the target topic model reveals a high bias towards topics with generic words, overestimating the importance of procedural words in topic identification. It also merges
Target Topic 16: 0.0990851 (procedural) Topic 6: 0.0767897 (- no topic -) Topic 17: 0.0706852 (housing)
Topic 15: 0.0610067 (migration) Topic 10: 0.0589544 (Sunday work)
Topic 9: 0.0589093 (media)
Topic 4: 0.0516202 (economy)
Topic 18: 0.049224 (pension and transport) Topic 8: 0.0482725 (infrastructure and ecology)
Topic 12: 0.0481855 (bank audit) Topic 13: 0.0461726 (health care)
Topic 19: 0.0438473 (army) Topic 7: 0.0429779 (disability act)
Topic 11: 0.0422667 (judicial)
Topic 5: 0.0385346 (agriculture) Topic 1: 0.0362681 (sport and education)
Topic 14: 0.0361004 (epidemic) Topic 2: 0.0314906 (regional development)
Topic 20: 0.0288681 (railways) Topic 3: 0.0268099 (family and election) specific topics into one, which makes the identified topic dificult to interpret (i.e. family and election, sport and education). Stronger preprocessing could be applied to the target corpus to remove key MT errors, such as duplicating words and erroneous translation of personal names.
In the future, we plan to provide annotated machine-translated ParlaMint corpora for all 16 languages (the UK corpus is already in English). The annotated corpus will enable a more accurate comparison of the results with identical preprocessing. Nevertheless, machine translation can be a viable first option for non-fluent researchers even in its imperfect current form.
The work described in this paper was funded by the Slovenian Research Agency research programme P6-0436: Digital Humanities: resources, tools and methods (2022-2027) and the Research Infrastructure CLARIN ERIC flagship project ParlaMint (2020-2023).
n r ,,iiitsleeccdhnnaoo ,,iiittrrseeebvdnaaa ,,itseepcccnnaooom ,,rseeccyynnaaoom ,,ittrsecvyuhnnnaoo ,itgyhnno ,iittfrcbunnaoo ,,ittsleeeepvvdnnomm ,,jtrrzeeebvnam ,,jrseebdyuaao ,,,iirrrsepccbhnnaom ,,iittrrssseepbduunao ,,iittslpyhaaow ,,iiiitttfseeecnnnnom ,,iiittrzleeeccvynno ,iiirrsseepcvvnoo ,,iitrrsseecgunaa ,,iiittfrssleebgdyyyaa ,,irrrsleebddoo ,,irsslcgdnnao ,,ittrrsleeepghnnaaao ,,,itrleeeuhnnooommm ,,,trrllepvnnnaaaaow ,,,iittrspppynaoooom ,,trlepcbgkkhnaoooo ,,,itreeeevgyhnnomm ,,,ttrrssleeeecvdnaaa ,iiiitlppccggkynnaaa ,,,jiittrršseunnaaomm ,,rlleeeeeppyyoommm ,,ittrrrsseeppccuuoooo ,,,tttrreeccbduuunom ,,,tttrrrssseeegyuham ,,iitreccvvgnnnaaoom ,,,itfrrreeecgduhnoom ,,,rrrsleepppcduaooo ,,,itttttrseeeeecbgdda ,,,iirrrrseeevddnnaoo ,,iiiittllepppccynaaoo ,,itrrrsssleeppgnnaao t i p , n , u c , o n d e d n r a ,,,,,iittttrsslleecccddduuhhnnaaooo ,,,,iiittslleeeeeppccvvdkyhnnoooom ,,,,iiittttfslleeeecccvdyyunnnaooom ,,,,iittttrrrsleeeecvggddhnnnaaoooom ,,,,iittfrrrllleeccgdduuuunaaoooomw ,,,,,iiiitttrssseevgguhhnnooomm ,,,,iiittfrrsleeexcbgduhnnaaoommm ,,,,iitttttfrrrrrssseeecccuuunnnaaoomw ,,,,iiiittttrrrrseeeeppcvgdnnnaaooomm ,,,,trrrrseeepcgddkkkyunaaaaooww ,,,,jjiiiitttttrseeecccggddduuunnaaaoo ,,,,iiiiiitttrrsssepcvgdunnnaaoooomm ,,,,iitttrrrsleeeepcccduhhnnaaaaoo ,,,,,tfrrrrssseeeeepccvkuhhnnaaaamm ,,,,iiittrrrebggdunnnnaaaooommmmw ft,,,,,ittrrrrlleeepcgdduunnaaaaommw ,,,,iittttfrsseeepbvgduuunnnnnaaomm ,,,,iitttrrrssleeepppccvghhhnnaaaooo ,,,,,frrseeeeepppccdyunnaaaaommw fi,,,,,iitttttrrrrsllpcdynnaaaaaaoow spo sepc ilch bu a ta se kn ipm fre l h en p 1 2 3 4 5 6 7 8 9 01 11 12 13 14 15 16 17 18 19 20 ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc ipc t t s ,rpo ,sed iten lppy rea itry jbo ifna em r dna iitc itsec enam irsu ,lpa ,ssu ,dy no tuho ,ino ,tren law koo icno ilym edna eomm ,lepo ,,,,,jj()iiiiiittrssleeccggddkyuuuhnnnaaaaooomw ,,,,,()()iiittfffrrrrssssleeepccgduuuhnnnaoooomm ,,,,,,iiiiirssseeeepcccvddyuuhnnaoooTVmmmmmw ,,,,,()iiiittttrrsssslleeeeppckuhhhhnnaaaaaoommm ,,,,,iiiiiittttttrrsssleeepxccccvyuuunnnnnaaooomm ,,,,,iiiittrrrrsssleeeeppccbyyyuunnnnaaaooooomm ,,,,,iiiitttrllleeepcccbggdynnnnaaaaaaooommmmw ,,,,,,ittttttrrrrssseeepppccccggdyyuuhhnnaaaaoooo ,,,,,iiiiiiitttttfrrrseepcvvuhnnnnnnaaaaoooooomm ,,,,iifrrrrsssssssllleeeeeppppcbdyuunnaaoooooom ,,,,,,iiiitttttrrrseeeeeexcccccggdyynnnnaaaaoom ,,,,()iitttttrrrrssleeeeeepcggddkyuhnnnaaaaaoomm ,,,,,,,iitttfrrrrrlleeeeepppccbgggdyuunnnaaoooow ,,,,,,iiiiitttrrrrrsssleepcbvyhnnnnaaaaaoooTVmAw ,,,,,,iiiiiiitrsllllleeeppcccbkyyuuhnnnnaaaooomm ,,,,,itttfrssleeeeepppcccddkyyyuhnnnnaaaaooomm ,,,,,,iiitttrrrrlleepcbgdyunnnnnnaaaaaaoooomm ,,,,,()iiiitttttffrrsssllleeeeeeepccyuhnnnaaoommmm ,,,,,iiiitttrrrssleeeccbbvgddduunnnnaaaoooommm ,,,,iiittttrrrsllleeecccggddyyuuuhhnnnnaaooooom ,,j()iiittrrleeepccddduuunaoo ,,()()iittfreecgdddnnaaaamw ,,,iittrrsseeeevguhnnnaoomm ,,iiittttrsleeeepcquuhhnnaam ,,itttfrrrrrseepcduunnaaaaom ,,,iitrrrssseeeepcuunnnnnaao ,,,,itttfssseeebvdkunnnnaam ,,,iittrrsleeppccddkunaaaom ,,,,iiittrssleepppccuhnnnaom ,,,,itttleeeeecdnnaaommmmw fifi,,,iiiitrrrrrleeeeecvvvggddhhn ,,,()ittrrssseepppccddunaooo ,,()iiitfrrleepccvddduaooom ,,,iitttrrssecccvgknnaaaaao ,,,tttrsleeeppxbgdyuunaamm ,,,,ttrrrssleeeeppSkuunoomw ,,,ittrrrreebgduhnaaommw ,,,iiifrrsleeeeeecbdnnommm ,,,iittrrrrseepbgnnaaoooomm ,,iiiiittsslllegddyuhnaaaooo itscno icddna ppoo ,trcoo ,jtreco isepno w epaow itrlca ,illch na pa tebgdu rkow ,rdyam ,teeem ,irsno ,tru ,iev d p hoo itmm secc co tc sc o a e c p s r e p