Nowadays, digital news articles are widely available, published by various editors and often written in di↵erent languages. This large volume of diverse and unorganized information makes human reading very difficult or almost impossible. This leads to a need for algorithms able to arrange high amount of multilingual news into stories. To this purpose, we extend previous works on Topic Detection and Tracking, and propose a new system inspired from newsLens. We process articles per batch, looking for monolingual local topics which are then linked across time and languages. Here, we introduce a novel ”replaying” strategy to link monolingual local topics into stories. Besides, we propose new fine tuned multilingual embedding using SBERT to create crosslingual stories. Our system gives monolingual state-of-the-art results on dataset of Spanish and German news and crosslingual state-of-the-art results on English, Spanish and German news.
• We show that multilingual documents are best represented through fine tuned DistilBERT [SDCW19] multilingual model using SBERT [RG19] triplet network structure. 2
This paper directly follows a growing body of work on Topic Detection and Tracking (TDT). Most of these works aim to solve the TDT task processing articles according to well defined pipelines.
In a first approach, Laban and Hearst [LH17] propose newsLens, a two-steps streaming system which first extracts keywords to create topics using Louvain community detection algorithm [BGLL08], and then solidify these local topics clusters into stories by comparing their keywords distribution. Emphasis is put on scalability, as the volume of news articles processed in their proposed experimental setup is about 4 millions. However, the algorithm does not handle multilingual articles and uses simple TF-IDF based method to compare articles. Moreover, its performance was not formally evaluated at the time.
Later, Miranda et al. [MZCB18] introduce a novel method to cluster an incoming stream of multilingual documents into monolingual and crosslingual stories. Documents are embedded in two latent spaces, a ”monolingual space” and a ”multilingual space”, which are used to cluster articles into topics. Then, new incoming articles can contribute to stories if they are close enough to topics centroids. Alongside their proposed system, they introduce a multilingual dataset adapted from Rupnik et al. [RML+15] containing articles in English, Spanish and German which have been manually annotated with monolingual and crosslingual story cluster labels. To the best of our knowlege, no other multilingual benchmark dataset has been proposed for the TDT task.
Recently, Staykovski et al. [SBCMN19] use the English part of the corpus from Miranda et al. [MZCB18] to assess the importance of article representations for news clustering. Among other, they show that sparse vector representation with TF-IDF weighting yields better results than doc2vec-based dense representation [LM14]. 3
We extend newsLens with a per-batch procedure, where documents published within a close range of time are processed to form local monolingual topics. Monolingual stories are then created by linking topics across time (i.e. across batches, Figure 1a) using topic centroids method from Miranda et al. [MZCB18]. Finally, multilingual stories are created by aggregating monolingual stories from di↵erent languages whose representations in a multilingual latent space are close enough (Figure 1b). In order to create local topics, we process articles per batch of close range, computing similarities between each pair of articles and making use of a community detection algorithm. Then, we link the local topics along time thanks to a ”replaying” strategy based on topics centroids similarities. 3.1.1
Following Miranda et al. [MZCB18] and Staykovski et al. [SBCMN19] who demonstrated the ineciency of dense features to cluster documents of a same language, articles are represented using sparse TF-IDF features. Monolingual representations for each document consist of 9 TF-IDF weighted bag of words sub-vectors, corresponding to the entities, lemmas and tokens contained in the title, body and title+body of each document. Contrary to previous work, we do not use any time feature, time being implicitely taken into account by the per-batch procedure. Please note that in all proposed experiments, we use the same entities, lemmas and tokens as already extracted by Miranda et al. [MZCB18] to ensure fair comparison of our proposed system. 3.1.2
To group articles into local topics, we build a non-oriented graph, where nodes represent articles and edges are weighted by the similarity between articles. More precisely, weights associated to edges are a linear combination of the cosine similarities between each of the 9 TF-IDF based representative sub-vectors of articles. Formally, the similarity function between two articles i and j is computed as :
K X k=0 k ⇥ ✓ (dik, djk) (1) Where K is the number of sub-vectors used to represent an article (i.e. K = 9), k are learned weights associated to sub-vector k, ✓ is the cosine similarity function and dik is the sub-vector k of article i.
In order to learn the best weights to aggregate the cosine similarities of articles’ representations for each language, we fit a logistic regression using the training part of the dataset. More precisely, for articles of a same language, we compute the cosine similarities between sub-vectors of each pair of articles. We then assign positive labels for pairs which are indeed of the same story and a negative ones for pairs of di↵erent stories.
Besides, using (1), we are able to compute similarities between each pair of articles. The resulting adjacency matrix can be viewed as a graph displaying weak links for pairs of articles which are of di↵erent stories and strong ones for pairs of a same story. We apply the Louvain community detection algorithm [BGLL08] to this graph in order to extract well delimited communities of articles, which will be refered to as topics. 3.1.3
Once a batch of documents has been clustered into topics, topics are linked across time (i.e. across batches) to form monolingual stories. To this end, we introduce a ”replaying” strategy (Figure 2), based on the similarity between articles in the current batch and topic centroids from previous batches. More precisely, when topics are created within a batch of articles at time t 1, we compute their centers as the average of each articles representative sub-vectors (Figure 2a). Then, for a new batch of articles at t, we compute similarities between all new articles and all topics centers at t 1 using the formula introduced in (1). When a topic at t 1 has a similarity with a new article at t greater than a threshold T1 (in our implementation T1 = 0.43 for English, 0.61 for German and 0.52 for Spanish documents), we replay all the articles constituting the topic at t 1, i.e. we add those articles to the current batch so that they can be considered during the new round of topic detection at time t (Figure 2b). This approach allows for the emmergence of di↵erent topic behaviors across time (Figure 2c): indeed, a previously created topic can subsist (eventually aggregating new articles) or not; it can also be split into several new topics. Finally, two or more topics can be merged into one. 3.2
Whenever we create new monolingual stories, we try to link them with current and past stories in other languages. To do so, we compute a common representation for stories in di↵erent languages, and associate them solving an optimal assignment problem. 3.2.1
To represent each article in a multilingual space, we use the SBERT [RG19] triplet network structure. We use the training part of our dataset in order to create labeled sentence triplets: The anchor and the positive example are articles in di↵erent languages coming from the same story, while the negative example is an article in di↵erent language and story than the anchor. We fine tune the multilingual DistilBERT [SDCW19] model using the concatenation of title and body articles for 15 epochs on 6,000 semi-hard triplets selected at the beginning of each epoch. We use a batch size of 8, a gradient accumulation of 2 steps, Adam optimizer with learning rate 2e 5 and using the MEAN pooling strategy. Then, in order to get the representation of a monolingual story, we average these representations over all articles within the story.
si = 1
X ej |Si| j2 Si (2)
Where Si is story i, and ej is our fine tuned multilingual DistilBERT embedding corresponding to article j. 3.2.2
Based on the conclusions of Miranda et al. [MZCB18], we use English as a pivot language in order to link monolingual into crosslingual stories. More precisely, we compute cosine similarities between non English and English story embeddings. Assuming that, at maximum one story of a given language can contribute to a multilingual story, we have to solve an optimal assignment problem. Between two sets of stories in di↵erent languages, we have to find the stories assignment between the two languages such that the sum of the similarities (resp. distances) of the linked story pairs is the highest (resp. the lowest) possible. We use the Hungarian algorithm [Kuh55] to solve this problem in polynomial time. More precisely, we define the cost function as the distance matrix (1 - similarity matrix) between stories of two di↵erent languages. Since some stories may not be related between two languages, we allow assignments only if the distance between two stories is less than a threshold T2 ( T2 is set to 0.22 in our implementation). We make this connection between monolingual stories each time we receive a new batch of articles, taking into account all monolingual stories not already assigned to multilingual stories and not older than 4 batches. 4 4.1
We assess the e↵ectiveness of our proposed approach on the standard multilingual dataset introduced by Miranda et al. [MZCB18]. It is a collection of 33,807 news articles in three languages: English, Spanish and German. These articles are labeled by language and by story. Stories are multilingual, i.e. that they may contain articles from several languages. The training set contains 20,803 articles and the test set 13,004 articles. We further divide the training set in two: a train part to learn the weights of the linear combinations to aggregate similarities between articles (1) and a development part to set the hyper parameter T1 and the resolution parameter of the Louvain algorithm (Section 3.1) as well as threshold T2 (Section 3.2). In order to set these parameters, we perform a grid search maximizing the average between standard and BCubed F1 scores. Table 1 presents descriptive statistics of the dataset. In order to assess the performances of our system, we report the standard as well as the BCubed1 [AGAV08] precision, recall and F1 score. We evaluate our system for two tasks: monolingual and multilingual news clustering. 4.2.1
First, we can observe that for each language, our method produces a number of clusters closer to the reality. Then, on English documents, our method generates the best F1 and accuracy scores. However, when looking at BCubed metrics, we can see that our system ranks second after the method introduced in Staykovski et al. [SBCMN19]. Nevertheless, for the two other languages which are German and Spanish, our method surpasses the system introduced in Miranda et al. [MZCB18]. Indeed, it displays better F1 and BCubed F1 scores, with improvements of 1.51 points for the F1 and 1.08 points for the BCubed F1 scores on German articles. Sometimes, even if our system is less precise than the one of Miranda et al. [MZCB18], it displays a much higher recall, yielding better standard and BCubed F1 scores. Notice that only 2% of our articles are replayed using our ”replaying” strategy. This insure that we are not performing a clustering from scratch each time we receive new articles. 4.2.2
In the multilingual clustering setting, we compare our system to the one of Miranda et al. [MZCB18], which is the only system handling multilingual news articles. We can see that our system displays better F1 (+2.5 points), precision and recall scores. This result shows that our improved system is able to both better organize monolingual stories, and link these stories over languages making use of our fine tuned DistilBERT embedding model. 5
We described a new method to cluster multilingual news articles into stories. We processe articles per batch as in newsLens, and naturally link found topics along time by maintaining centroids for monolingual clusters. More precisely, we introduced a new ”replaying” strategy to link monolingual topics into stories, and then create crosslingual stories by embedding articles thanks to SBERT [RG19]. Our system gives both monolingual and crosslingual state-of-the-art results on the English, Spanish and German dataset introduced by Miranda et al. [MZCB18].
1Unlike the classic version, the BCubed version of precision, recall and F1 score favors solutions that (i) make errors in clusters with already many errors (ii) make errors in a large clusters rather than in small ones.
In future work, we plan to challenge the TF-IDF based representation of monolingual articles using fine tuned SBERT [RG19] embeddings. Moreover, it would be interesting to assess computational eciency of di↵erent systems by testing them on bigger news dataset.
The authors would also like to thank Mr. Cl´ement Rebu↵el, Mr. Pirashanth Ratnamogan, and Mr. Bruce Delattre from BNP Paribas for their valuable comments and suggestions. [AGAV08] [RML+15] Jan Rupnik, Andrej Muhic, Gregor Leban, Blaz Fortuna, and Marko Grobelnik. News across languages - cross-lingual document similarity and event tracking. In IJCAI, 2015.