Testing Pre-trained Transformer Models for Lithuanian News Clustering

Abstract

TA recent introduction of Transformer deep learning architecture made breakthroughs in various natural language processing tasks. However, non-English languages could not leverage such new opportunities with the English text pre-trained models. This changed with research focusing on multilingual models, where less-spoken languages are the main beneficiaries. We compare pre-trained multilingual BERT, XLM-R, and older learned text representation methods as encodings for the task of Lithuanian news clustering. Our results indicate that publicly available pre-trained multilingual Transformer models can be ifne-tuned to surpass word vectors but still score much lower than specially trained doc2vec embeddings.

Keywords

• eol>Document clustering
• document embedding
• Lithuanian news articles
• Transformer model
• BERT
• XLM-R
• multilingual

1. Introduction

classification, clustering is universal in that it can handle unknown categories [11, 12]

Therefore it is well suited for the quickly changing news articles data.

© 2020 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmUmoRns WLiceonrsekAsthtriobuptioPnr4o.0cIneteerdnaitniognasl ((CCC EBYU4R.0)-.WS.org) 1https://www.internetworldstats.com/stats7.htm

2. Literature Review

In this section we review the two first consecutive pha

ses of common natural language processing (NLP) tasks: text preprocessing and text representation [10].

These stages recently are of the most active research

and culminated in the development of the Transformer architecture. We also examine relevant NLP contributions for Lithuanian language and our task of news clustering.

2.1. Text Preprocessing

Text preprocessing involves selection of features that will bear the understanding of text. The most elementary approach is tokenization into simple characters or words. The finer the tokenization, the smaller is the resulting vocabulary and the more challenging task is given to the NLP model. On the other hand, coarser tokens drastically increase vocabulary size and induce other problems such as sparseness. The middle ground is statistically significant n-grams of both words and chars. Examples of this type of tokenizers are SentencePiece [14], BPE [15], and WordPiece [16]. They are often used in the state-of-the-art (SOTA) Transformer models and are shipped together with publicly available pre-trained models. This way manual tokenization step is skipped.

There are number of methods to filter word level to

kens. It includes lowercasing, stemming, lemmatization, filtering by maximum and minimum document frequencies (ignoring tokens that are too rare or too common throughout the documents).

However, it was shown in [17] that such filtering

benefits only the classical text representation approaches such as tf-idf, while shallow neural network models, doc2vec [18], benefited of not using any such filtering.

2.2. Text Representation

Although tokenized text remains meaningful to us, models still can not operate on it directly. They need it in a numerical form. The preferable way is to derive vector representation for each text sample. Cosine similarity is the simplest example of models operating on (comparing) these embeddings. The classical approach to text representation uses a Bag of Words (BoW) model. As the name suggests, the order of tokens is lost here and each document is represented by bare counts (histogram) of its tokens. Therefore token weighting such as tf-idf is involved. The higher weight of tf-idf is, the more descriptive token for a given document is. Given the number of word

occurrences in a document as tf a n, d total number of documents , tf-idf , number of documents containing word as df

is given by , tf-idf , = tf, ⋅ log( /df ). (1)

BoW approaches sufer from several problems. The

vector length for each document is the same as the size of the vocabulary. Typically, the vocabulary size is huge and this induces major memory constraints.

The embedded vectors are also very sparse as each document uses only the small subset of the vocabulary. Various methods, such as Latent Semantic Analysis (LSA) using Singular Value Decomposition (SVD), are employed to reduce the dimensionality. Nevertheless, SVD has to operate on the same high dimensional documents × tokens matrix.

Work of [ 19 ] revolutionised word embedding calculations. Previous word embeddings, known as cooccurrence vectors, were superseded. They were calculated as direct probabilities of surrounding words in a context window of a given length. The new word2vec [ 19 ] algorithm uses the same training inputs, except the goal is not to calculate the word distribution, but to derive such embedding weights that context words would be predicted with maximum accuracy. Such setup significantly reduced the word vector size and eliminated problems of high dimensionality and sparseness. 2.3. Related Work on Lithuanian Later, the next word embedding model, Global Vectors Language (GloVe) for word representation was presented [ 20 ].

It merged advantages of both the matrix factorization There are several works on Lithuanian text clusterand the shallow window-based methods. Currently it ing. [ 25 ] used internet daily newspaper articles from is the most used method for independent word vectors. the Lrytas.lt news website and information from the

Original word2vec word vectors were extended to largest internet forum for mothers – supermama.lt. paragraph vectors by doc2vec model [18]. Here each means and Expectation Maximization (EM) algorithms sequence of tokens has its own embedding in the same were compared on BoW data representation. It was space as that of words. CBOW and Skip-gram archi- found that optimal clustering algorithm is -means with tectures from the original word2vec algorithm were cosine similarity. Other work analysed unsupervised applied to documents correspondingly as PV-DM and feature selection for document clustering [ 26 ]. AuPV-DBOW. Distributed Memory model of Paragraph thors found that tf-idf weighting with 3 000 features Vector (PV-DM) is tasked to predict the next context and spherical -means clustering algorithm works best. word given the previous context and document vector, A similar observation is expressed in [ 27 ]. Here thus the vector has to sustain memory of that is miss- means was compared to various hierarchical clustering. Distributed Bag of Words version of Paragraph ing algorithms and was not superseded. Authors found Vectors (PV-DBOW) is forced to predict context vec- that tf-idf together with stemming is superior to other tors randomly, given only the document vector. The approaches. original work [18] used a concatenation of the both [17] compared BoW and doc2vec (PV-DBOW) Lithuamodels. However, in [17] it was found that PV-DBOW nian news article representations for document clusalone gave the best results. tering. It was shown that PV-DBOW representation,

The most recent text representation models produce trained on the whole dataset is superior to the BoW contextualised token vectors. Models like ELMO [ 21 ] method. Authors also investigated various PV-DBOW and various Transformers have each of their inputs to hyperparameters and outlined recommendations for interact with the other ones. This leads to each to- training method weights. ken vector being aware of the others. Contextuali- There is also other NLP work on Lithuanian lansation solved the problem of word polysemy seen in guage. [ 28 ] and [ 29 ] compared traditional and deep word2vec or GloVe models. learning approaches for Lithuanian internet comments

The Transformer architecture presented in 2017 [1] sentiment classification. Authors demonstrated that excels other contextualised models due to several rea- traditional Naïve Bayes Multinomial and Support Vecsons. Firstly, only after a single layer each input repre- tor Machine methods outperformed LSTM and Convosentation becomes aware of the other ones. For Recur- lutional neural networks. Other work [ 30 ] compared rent Neural Networks (RNN) like ELMO it took lay- CBOW and Skip-gram word embedding architectures ers, where is the sequence length. Another advan- and found the first one to be superior. We can add that tage over recurrent architectures is that Transformer in this current work we noticed a similar tendency for is very parallelizable. It does not need to wait for a document vectors: equivalent version of PV-DBOW in hidden state of the previous word as is the case with our initial experiments outperformed PV-DMM archiRNN. This particular feature led to creation of multi- tecture. billion-parameter Transformers such as GPT-2 [ 22 ], T5 We have not found any previous work on Lithua[ 23 ], Megatron [ 24 ], and T-NLG2. Despite huge suc- nian language using Transformer models. cess of Transformer models, it can not process long sequences as its complexity per layer is ( 2 ⋅ ) where 3. The Data is representation dimension. For example, the maximum input length of the popular BERT model is just 510 tokens. This and other problems of Transformer architecture currently are researched very actively.

We followed methodology of [17] and expanded their

dataset from 82 793 up to 260 146 articles. Although average number of characters in each our text sample is 2 948, several scraped articles were very small and resulted in empty vectors during averaging the GloVe vectors. Due to this reason we filtered all articles with less than 200 characters and this resulted in a final dataset of 259 996 texts.

2https://www.microsoft.com/en-us/research/blog/ turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ • Lithuanian news (60 158 articles); • World news (68 635 articles); • Crime (30 967 articles); • Business (19 964 articles); • Cars (6 313 articles); • Sports (14 910 articles); • Technologies (4 438 articles); • Opinions (9 728 articles); • Entertainment (2 462 articles); • Life (3 811 articles); • Culture (7 967 articles); • Other (30 643 articles that do not fall into the

previous categories).

During most of experiments we employed a smaller subset of the dataset described above. We sampled randomly 125 news articles from each of the 12 categories. That results in a total of 1 500 articles equally distributed among the categories and corresponds to the data required for one clustering. We planned to make 50 independent clusterings to average results and enhance their reproducibility. Thus we repeated inde- 4.3. PV-DBOW pendent sampling of 1 500 equally distributed articles and used a subset of 55 487 unique articles (with repetitions it would be 75 000). During each experiment we calculated embedding vectors only for those 55 487 news articles.

4. Methods

The data consist of Lithuanian news articles scraped

from lrt.lt, 15min.lt, and delfi.lt websites. The number of texts are correspondingly 26 344, 133 587, and 4.1. Clustering 100 065. Due to the absence of sitemap in lrt.lt website, we did not scrape more articles from this site than is We use -means clustering algorithm. Due to its high already scraped in [17]. speed, it is suitable for large corpora [ 25 ] and outper

Evaluation of clustering requires existing knowledge forms other clustering algorithms [ 27 ]. During experof the potential clusters. For this task we leveraged ar- iments we feed vectorized document representations ticle category labels extracted from each article URL. and the expected number of clusters to -means and Following the mappings of [ 27 ], the labels were unified receive document assignments to clusters. We set each from over a hundred categorical descriptions down to of 50 -means initialisations the same. 12 distinct categories. The resulting categories of the articles are: 4.2. Evaluation

First, we calculate the following confusion matrix elements: • TP – pairs of articles that have same category

label and are predicted to be in the same cluster; • TN – pairs of articles that belong to diferent categories and are predicted to be in diferent clusters; • FP – pairs of articles that belong to diferent categories but are predicted to be in the same cluster; • FN – pairs of articles that have same category

label but are predicted to be in diferent clusters.

We chose to evaluate clusters by Matthews Correlation Coeficient (MCC) score due to its reliability as described in [31]. It is calculated as

TP ⋅ TN − FP ⋅ MCC = √(TP + FP)(TP + FN)(TN + FP)(TN + FN) . (2)

The MCC score ranges from -1 to 1. Scores around 0

value correspond to random clustering, while close to 1 indicate perfect matching. For the sake of completeness, it must be said that there are other Correlation Coeficients that provide reliable evaluations [32].

This doc2vec version was trained on all our dataset – total of 259 996 Lithuanian news articles. We preprocessed the dataset by lowercasing and tokenizing it into words. The same vector size (100), number of epochs (10), window (12), and minimum count (4) parameters were used as in [17]. PV-DBOW returns a single embedding for each document so no further aggregation is required. 4.4. GloVe 4.6. XLM-R We performed our own text preprocessing during ex- XLM-R [7] is one of the recent multilingual language periments with GloVe [ 20 ] type Lithuanian word vec- models, much bigger than multilingual BERT. It is trained tors [33]. It combined lowercasing and word level tok- on 2 terabytes of filtered text from which 13.7 GB is enization. Out of whole unique 1 028 816 tokens from Lithuanian. The huge size of this model limited our exour whole dataset 311 470 were also present in Lithua- periments. We only calculated outputs of the first 512 nian GloVe vectors. This unique tokens intersection tokens for each news article. It took approximately a amounts to 30 % of our tokens and up to 94 % of GloVe. total of 40 hours.

We tried several ways of aggregating GloVe vectors: 5. Results 5.1. GloVe • calculating an average of all the word vectors in

the article; • weighting all tokens with tf-idf and calculating an average of the 20 word vectors with the highest weight;

Results with aggregation of GloVe vectors are presented in Table 2. It is clearly seen that applying tf-idf weighting to select the best tokens to average can signifi• weighting all tokens in an article with Softmax(tf- cantly surpass the simple average of all vectors. idf) and calculating a weighted average of all the word vectors in the article.

5.2. Multilingual BERT

4.5. Multilingual BERT The efect of multilingual BERT fine-tuning on Lithuanian news articles is depicted in Fig. 1. One can clearly BERT [6] outputs the same number of vectors as it is see that (1) the fine-tuning improves the clustering refed inputs. The first is a special [CLS] token which is sults, (2) the average of all tokens is much better than designed to be used in sentence level tasks. The fol- only the [CLS] vector, and (3) the uncased model verlowing are text data tokens, ending with the last [SEP] sion outperforms the cased one. token. Optionally, one can add a second [SEP] token in To our surprise, the best results were obtained with a middle of an input sequence to separate two text seg- limiting the number of tokens to only the first 144 (see ments. In our experiments we input only one segment Fig. 2). This can be attributed to the more important and always separately try the [CLS] token output vec- information being in the beginning of news article. We tor and the averages of all token vectors. Pre-trained observed the same tendency with the XLM-R model. models like multilingual BERT are supposed to be finetuned for the desirable task. 5.3. The Best Models

We performed Masked Language Modelling fine- tuning on half of our subset data – 27 743 news articles. We tried four diferent models to represent Lithuanian We trained with batch size of 4 for 5 epochs totalling news articles. Do Transformer models scored better 68 505 steps for uncased and 75 985 steps for cased ver- than PV-DBOW? As can be seen in Table 3, the best sions of pre-trained multilingual BERT model. Transformer model managed to outperform GloVe vec

The maximum number of input tokens to BERT is tors. However, PV-DBOW model is far ahead with the 512, including special tokens. The most articles are mean MCC score of 0.442. within 512 tokens limit but some are longer. We tried to estimate efect of this constraint by trying to feed even fewer tokens and analyzing how mean MCC score 6. Conclusions changes with the longer input sequence.

We carried out our experiments in Google Colab3 In this work we compared multilingual BERT, XLM-R, environment. It ofers 12 GB of RAM and GPU-accele- GloVe, and PV-DBOW text representations for Lithuarated machines which allows an order of magnitude nian news clustering. For BERT we found out that speed up of BERT model compared to CPU. the average of only the first 144 token vectors outperforms longer aggregations or the [CSL] token vector.

We observed that BERT fine-tuning with Lithuanian news articles also improves the results. The other pretrained Transformer type model XLM-R was too com3https://colab.research.google.com/ putationally expensive to optimize and out of the four its initial representations scored the worst. Regard- specially trained simpler PV-DBOW. ing GloVe vectors, we found that its best Softmax(tf- Multilingual BERT MCC score kept rising till last idf) embeddings (mean MCC score 0.264) are outper- fine-tuning steps and it is not clear how large the imformed by the BERT. Nevertheless, the best text repre- provement could be accomplished training longer. Our sentation method proved to be PV-DBOW with mean future plan is to clarify this by using more data. We MCC score 0.442. Our work on generating representa- also plan to train a new monolingual BERT model speciftions for Lithuanian news clustering showed that mul- ically for Lithuanian language. It would be interesting tilingual pre-trained Transformers can be better than to know if these resource-“hungry” approaches could independent GloVe vectors but under-performs against surpass the score of the relatively simple PV-DBOW

References

[19] T. Mikolov , K. Chen , G. Corrado, J. Dean , Efi- wordnet., Artificial intelligence and algorithms cient estimation of word representations in vec- in intelligent systems: proceedings of 7th comtor space , arXiv preprint arXiv:1301.3781 ( 2013 ). puter science on-line conference 2018 ( 2018 ) 394 -

[20] J. Pennington , R. Socher , C. D. Manning , Glove: 404 . Global vectors for word representation , in: [31] D. Chicco , G. Jurman, The advantages of the Proceedings of the 2014 conference on empir- matthews correlation coeficient (mcc) over f1 ical methods in natural language processing score and accuracy in binary classification eval(EMNLP ), 2014 , pp. 1532 - 1543 . uation, BMC genomics 21 ( 2020 ) 6 .

[21] M. E. Peters , M. Neumann , M. Iyyer , M. Gard- [32] F. Beritelli , G. Capizzi, G. Lo Sciuto , C. Napoli, ner, C. Clark, K. Lee , L. Zettlemoyer , Deep con- M. Woźniak , A novel training method to preserve textualized word representations, arXiv preprint generalization of rbpnn classifiers applied to ecg arXiv: 1802 . 05365 ( 2018 ). signals diagnosis, Neural Networks 108 ( 2018 )

[22] A. Radford , J. Wu , R. Child , D. Luan , D. Amodei , 331 - 338 . I. Sutskever, Language models are unsupervised [33] A. Bielinskienė , L. Boizou , I. Bumbulienė, J. Komultitask learners, OpenAI Blog 1 ( 2019 ) 9 . valevskaitė, T. Krilavičius, J. Mandravickaitė,

[23] C. Rafel , N. Shazeer , A. Roberts , K. Lee , E. Rimkutė, L. Vilkaite ̇-Lozdienė, Lithuanian S. Narang , M. Matena , Y. Zhou , W. Li , P. J. Liu , word embeddings, 2019 . URL: http://hdl.handle. Exploring the limits of transfer learning with a net/20 .500.11821/26, CLARIN-LT digital library unified text-to-text transformer, arXiv preprint in the Republic of Lithuania . arXiv: 1910 . 10683 ( 2019 ).

[24] M. Shoeybi , M. Patwary , R. Puri , P. LeGresley, J. Casper , B. Catanzaro , Megatron-lm: Training multi-billion parameter language models using gpu model parallelism , arXiv preprint arXiv: 1909 . 08053 ( 2019 ).

[25] G. Ciganaitė , A. Mackute ̇-Varoneckienė, T. Krilavičius, Text documents clustering ., Informacinės technologijos : 19-oji tarpuniversitetinė tarptautinė magistrantu˛ ir doktorantu˛ konferencija "Informacinė visuomenė ir universitetinės studijos" (IVUS 2014 ) : konferencijos pranešimu˛ medžiaga ( 2014 ) 90 - 93 .

[26] A. Mackute ̇-Varoneckienė, T. Krilavičius, Empirical study on unsupervised feature selection for document clustering ., Human language technologies - the Baltic perspective : proceedings of the 6th international conference , Baltic HLT 2014 ( 2014 ) 107 - 110 .

[27] V. Pranckaitis , M. Lukoševičius , Clustering of Lithuanian news articles , in: CEUR Workshop Proceedings , 2017 .

[28] J. Kapočiu ¯tė-Dzikienė, R. Damaševičius , M. Wozniak , Sentiment analysis of Lithuanian texts using traditional and deep learning approaches ., Computers 8 ( 2019 ) 1 - 16 .

[29] J. Kapočiu ¯tė-Dzikienė, R. Damaševičius , M. Woź- niak, Sentiment analysis of Lithuanian texts using deep learning methods ., Information and software technologies: 24th international conference, ICIST 2018 , Vilnius, Lithuania, October 4- 6 , 2018 : proceedings ( 2018 ) 521 - 532 .

[30] J. Kapočiu ¯tė-Dzikienė, R. Damaševičius , Intrinsic evaluation of Lithuanian word embeddings using