The task of organizing and clustering multilingual news articles for media monitoring is essential to follow news stories in real time. Most approaches to this task focus on high-resource languages (mostly English), with low-resource languages being disregarded. With that in mind, we present a much simpler online system that is able to cluster an incoming stream of documents without depending on language-specific features. We empirically demonstrate that the use of multilingual contextual embeddings as the document representation significantly improves clustering quality. We challenge previous crosslingual approaches by removing the precondition of building monolingual clusters. We model the clustering process as a set of linear classifiers to aggregate similar documents, and correct closely-related multilingual clusters through merging in an online fashion. Our system achieves state-of-the-art results on a multilingual news stream clustering dataset, and we introduce a new evaluation for zero-shot news clustering in multiple languages. We make our code available as open-source.
The last few decades have been characterized by an exponential growth in the amount of news
sources available, with the task of following news stories in real-time becoming very dificult to
perform manually. As such, a demand has risen for systems that are capable of monitoring and
organizing articles into news stories. Most approaches to this task focus mainly on the English
language [
The contributions described in the paper are: (i) We develop a system that is able to cluster documents without depending on language-specific features; (ii) We empirically demonstrate that the use of multilingual contextual embeddings as the document representation significantly improves clustering quality; (iii) We propose a method to train a classifier in order to merge similar clusters in an online setting, and demonstrate its impact in obtaining state-of-the-art results for multilingual clustering; (iv) We show that our system performs well on languages not seen during training and we describe a zero-shot experimental setting for Chinese, Russian, French, Italian, Slovenian and Croatian.
The Topic Detection and Tracking (TDT) task [
Regarding batch clustering approaches, Laban et al. introduce newsLens [
For online clustering, Miranda et al. [
Our system follows the described online clustering approaches to the TDT task: while Miranda et al. was bound to the usage of specific individual models for each language due to processing monolingual clusters, our approach provides a system that is able to leverage on dense multilingual document representations without the need to process the documents into monolingual clusters first. This is accomplished by using a single crosslingual representation for the documents and the consequent training of our ranking and classification models at a crosslingual level, which also allows for a fully dense clustering space and guarantees that our system is not limited to the English language, unlike Sarvanakumar et al.’s approach.
Our main focus for this task is to build an online multilingual news clustering system that
depends as little as possible on language-specific features in order to process news articles for
zero-shot languages (where we have no clustering training data) without having a considerable
loss in performance. Previous approaches mostly focus on a single language [
Our system is composed by four main steps (partially displayed in Figure 1): obtaining the document representations, computing the best-ranked cluster, deciding if the document accepts the best-ranked cluster and enters it, and merging clusters that pertain to the same story.
In contrast to previous work, we use a representation for each document that does not depend on
its language, thus eliminating the need of distinguishing representations between monolingual
and crosslingual. Documents are comprised of two components: a set of dense vectors
corresponding to a contextual representation of the document, and a temporal representation
. To obtain the dense vectors we use distiluse1, a model produced through knowledge
distillation [
For each document, contains three dense representations: 1 corresponds to its body+title, 2 to its f.p. (first paragraph), and 3 to its f.p.+title. For the first paragraph, the representation retrieved corresponds directly to the output of the model’s encoder, while in the case of the first paragraph + title, mean pooling is performed between the two output vectors corresponding to each component. Finally, for the body + title, the body is segmented into paragraphs and each paragraph is processed individually by the encoder, with the final output vector being obtained through mean pooling of each paragraph’s representation and the title. The title and the first paragraph of the documents are used as features with the intuition that sentences in the beginning of a news document usually have the greatest importance to the article [17, 18]. Additionally, a representation for the title alone as a feature was not used for the documents due to certain news articles containing only the text of the article and no title.
Regarding the temporal representation, we follow previous approaches [
Clusters are also divided between dense ( ) and temporal ( ) representations, with each
cluster keeping three centroids for each document representation (body+title 1, f.p. 2, f.p.+title
3) that correspond to the average of the respective representations of each accepted document.
When the document’s representations are received, each centroid is updated to take them into
account. A cluster also maintains timestamps for the newest document ( 1 ), the oldest ( 2 ), and
the mean timestamp between all documents in the cluster ( 3 ). To compare two timestamps,
we compute the Gaussian similarity between them as proposed in previous work [
. After a cluster is created, it is stored in the cluster pool, a structure that is responsible for maintaining the clusters and archiving old clusters as the system grows in size. the Gaussian similarity between the two timestamps (with and corresponding to hyperparameters) as represented in the function below: (1) (2) (3)
After computing its representations, a given document is compared against each cluster in the cluster pool in order to retrieve the most similar cluster to . To determine the similarity between and each cluster in the cluster pool, we compute ’s ranking score and the best-ranked cluster is then evaluated by the acceptance model. If the cluster is accepted by the model, then enters the cluster and its representations are updated; otherwise, a new cluster containing is created. Temporal features are computed through the aforementioned features are obtained through the computation of the cosine similarity ( given representation of the document and a representation of the cluster, as follows: ), with being a function, and the dense 3 =1 3
The ranking score for a cluster given a document and the ranking model’s learned SVM
weights and is represented as follows:
,
) ⋅ ) + ∑ (
( +1 , 1) ⋅ +3 )
The ranking model takes the form of a Rank-SVM model [13], which we train using a similar
scheme to Miranda et al. [
is greater than zero, then is accepted into , otherwise, a new cluster is created and initialized with . The acceptance model is an SVM trained on the training partition of the dataset: each document generates a positive sample for its corresponding gold cluster, and its second-best ranked cluster is given as a negative example. 2 =1 ) + (4) (5)
as well as sizes. The After a cluster receives a new document, we rank its similarity to each of the other clusters in the cluster pool using the ranking model (described in Section 3.2). Each candidate cluster is then evaluated by a third SVM model, which we call cluster merge model, and the documents from each cluster with a positive merge decision are inserted into the source cluster.
The intuition for this model is to find separate clusters that have grown to pertain to the same story, and subsequently merge them. This may happen throughout the clustering process, as few documents pertaining to a given story have entered the system, and the acceptance model may mistakenly assign separate clusters to those documents initially. As more relevant documents enter the system, those clusters may end up in similar points in the vector space, and thus should be merged. For this model, we use the eight features specified in Section 3.2
, and two additional features corresponding to the size of each cluster of the evaluated pair, with the intuition of associating the cluster merging to clusters of small , given a cluster with documents and a size limit vector of length , is represented by the following equation: into their gold clusters, and with the ranking and acceptance model trained accordingly.
We follow previous work on this task and evaluate our system on a news clustering dataset [14]. Besides the three main languages (English, Spanish and German), this dataset also provides a significant amount of documents in Chinese and Russian, as well as documents in Slovenian, Croatian, French and Italian. These samples allow us to roughly preview the performance of the system on other languages besides the ones it was trained in. The dataset is composed by 34,687 news documents, and it is divided into two sets: a training set comprised of 20,813 articles, and a test set that contains 13,874 articles. The articles in the training set are dated from 18-12-2013 to 02-11-2014, while the articles in the test set are dated between 02-11-2014 and 27-08-2015, thus guaranteeing that the articles in the test set are newer and their thematic have not been observed in the training set. Further statistics regarding the dataset are presented in Table 1.
Regarding evaluation metrics, we follow the same approach as [
For the BCubed F1 score, as previously described by Staykovski et al. [
BCubed = [
. = [ (, )]] BCubed = [ . = [ (, )]] BCubed 1 = 2 ∗ BCubed ∗ BCubed
BCubed + BCubed (6) (7)
For the monolingual evaluation, we evaluate the clustering performance of our model on
the three main languages of the dataset by performing clustering using only the documents
of the specified language, while the crosslingual evaluation uses the entirety of the test set
regardless of language. In order to evaluate the results, a gold set of cluster labels is provided for
each document that indicates the expected cluster of that document. The clusters are typically
multilingual, and in accordance to previous work [
As shown in Table 2, for the monolingual evaluation our system is on-par with Miranda et al.’s in English and German on both metrics, but is surpassed by Linger et al.’s in all languages and modularities except for German when evaluating both metrics.
For crosslingual clustering, as shown in Table 3, our system achieves state-of-the-art
performance on BCubed F1 [
BCubed
P
R ranking and acceptance model, which used only , 1) and the timestamp features.
Adding the other features (8-F Rank+Acc.) improved both standard (+1.42) and BCubed F1 (+1.22). Finally, the cluster merge model was added to our system (8-F Rank+Acc.+M.), resulting in gains for both standard (+3.35) and BCubed F1 (+0.86).
Given the nature of our system, we evaluated it on the remaining languages of the dataset as shown in Table 4. Our ranking, acceptance and cluster merge models were not trained on any data from these languages (with the exception of Chinese), making this a zero-shot clustering scenario. Chinese, French, Russian and Italian document clustering had high F1-scores, with results above 95%, and both Slovenian and Croatian had initial clustering scores above 70%.
We presented a clustering model that produces state-of-the-art results at a multilingual level without depending on language-specific features, and that maintains quality at a monolingual level on-par to previous work on news clustering. We demonstrated that it is possible to improve results by utilizing contextual embeddings to represent documents at a crosslingual level, and how a linear SVM can be trained in order to perform such a task. By reducing the complexity of the clustering space, we motivate future research on topics such as clustering while taking user feedback into account, and high-performance vector search to improve clustering speed and scalability. Our system also enables computational eficiency improvements by allowing most operations to be paralellized. We make our code available as open-source2.
2https://github.com/Priberam/projected-news-clustering
This work is supported by the EU H2020 SELMA project (grant agreement No 957017). [13] Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. (2002) [14] Rupnik, J., Muhic, A., Leban, G., Skraba, P., Fortuna, B., Grobelnik, M.: News across languages - cross-lingual document similarity and event tracking. In: Journal of Artificial Intelligence Research, Special Track on Cross-language Algorithms and Applications. (2015) [15] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using siamese BERTnetworks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. (2019) [16] Huang, K., Gardner, M., Papalexakis, E., Faloutsos, C., Sidiropoulos, N., Mitchell, T., Talukdar, P.P., Fu, X.: Translation invariant word embeddings. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. (2015) [17] Hong, K., Nenkova, A.: Improving the Estimation of Word Importance for News MultiDocument Summarization. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (2014) [18] Nenkova, A., Maskey, S., Liu, Y.: Automatic Summarization. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts. (2011)