In this paper we present a work on interactive event detection, visualization and labeling. We started from an unsupervised one-shot event detection system created by A. Guille and A.-C. Favre and modified it to optimize its interactive usage. We also added a new event labeling approach to enable a more meaningful event description for the human users, by selecting representative documents, in addition to the keywords extracted by the original method. The source code of our work is available online at https://github.com/CorentinForler/mabed-interactive-labeling.
events are described by a set of primary and secondary keywords and a time-based impact score histogram. This histogram describes, in a way, the life cycle of the event.
Our work focuses on two extensions of the method of Guille and Favre: its optimization for an interactive use, and an event labeling by full-text fragments, instead of keywords, for a better understanding. Indeed, one of the limitations of the original method is the need to precise the number of events to detect. Our system proposes an interactive interface to adjust in an assisted manner this number. We also extended the method to detect events from any timestamped text, not only tweets like the original method. In this paper, we will present our event detection interface, emphasizing the improvements we made on the original method to decrease its response time in an interactive event detection use case. We then present our label extraction approach having as objective selecting the most representative text(s) for a given event, described by keywords and an impact histogram.
We did a first implementation of the method described in [
The chosen dataset is a list of 238 872 news articles, collected using RSS feeds of mainly economyrelated English language newspapers, spanning from 2020-01-01 to 2020-09-30 (9 months), with a vocabulary amounting to 25 097 distinct words/tokens. This dataset was then used to extract 50 events, using the following parameters: • length of each time-slice tsl : 24 hours (1440 minutes) • number of events to extract ec : 50 • number of keywords to extract kwc : 10
Once events were computed, the event labeling step was then run. Example events produced by MABED are: • Event 1 – Begin/End date : 2020-03-16 – 2020-06-04 – Impact over time : – Main terms : global, pandemic, health, due, covid-19, amid, economic 1https://github.com/AdrienGuille/pyMABED 2The tsl parameter controls the temporal precision of the algorithm, because the input dataset is split into time-slices of equal duration of tsl hours (or minutes, seconds,... unit to choose).
– Related terms : package, trillion, ministry, postponed, impact, closed, concerns, confirmed, demand, deaths – Human-made description : On March 25, 2020, the U.S. Senate passed a $2 trillion coronavirus economic relief package. – Selected article : U.S. senators were set to vote on Wednesday on a $2 trillion bipartisan package of legislation to alleviate the devastating economic impact of the coronavirus pandemic. • Event 33 – Begin/End date : 2020-03-13 – 2020-07-29 – Impact over time : – Main terms : order, gov – Related terms : andrew, issued, cuomo, stay-at-home, executive, businesses, spread, york, court, health – Human-made description : On March 20, 2020, New York governor Andrew
Cuomo announced a stay-at-home executive order. It ended on May 15, 2020. – Selected article : New York Gov Andrew Cuomo is ordering all workers in nonessential businesses to stay home and banning gatherings statewide.
We created a new user interface for the method provided in [
A simple form is shown to the user. While the event detection algorithm can be adjusted with various parameters, most of them could be dificult to explain in the interface, and some are only used to fine tune the results. Therefore, we chose to emphasize three parameters: the duration of each time-slice, the number of events to detect, and the maximum number of words to describe an event.
The form is shown on Figure 1. It includes a button to start the computation once the desired parameters are entered. Below the form, and once a computation is done, the table of events, depicted in Figure 2, is presented to the user.
Its first column, titled “Event Terms”, contains main and secondary terms of the event given by the original algorithm (the two kinds of terms are styled diferently and secondary terms (in brown) are sorted by decreasing magnitude). The second column, titled “Proposed Articles”, contains a list of proposed descriptions for each event. Each description can be marked correct or incorrect by clicking on it, initially being in a neutral state (not shown on Figure 2). Moreover, each description can be moved up or down among the proposed descriptions of a given event, efectively allowing the users to rank the descriptions. These behaviors have been added for an easier data annotation during the experimental evaluation of the system, and won’t be shown to the final users. The third and last column, titled “Event Impact”, contains a graph showing the impact of the event over time.
The user can, at any time, change the values and click on the "Compute results" button to update the table of events and the graphs. This work on the user interface is supported by a set of improvements to the original MABED implementation.
Given that our goal is to provide an interactive event visualization service, we had to make the whole system faster, especially for consecutive runs where few parameters change. While we added many improvements to the original method, which we are going to present in the following sections, we mostly improved the response time of the service with a cache system.
In the initial MABED implementation, intermediary computations are not permanently stored, therefore all executions take the same time (when given the same parameters). To allow faster successive computations, we needed to store the results of these intermediary steps. We designed a cache system, allowing to keep these results instead of discarding them. For the same dataset and the same parameters, the computation time is zero, and only the parsing time of the cache ifles is required. If some parameters change, only the computations that depend on them need to be carried out again.
To measure the efect of this optimization, we performed a series of experiments whose results are presented in Table 1. First, we did three unoptimized runs that do not use the cache system, labeled “from scratch”. In the initial version of MABED, all runs would have lasted the same duration as these unoptimized computations.
Then, we ran a single “Baseline” computation, storing its results in an initially empty cache, using the following parameters: tsl of 1440 minutes (24 hours), ec of 10 events, kwc of 10 words. The run labeled “Baseline (cached)” shows that running the same computation again, using the cache, is very fast.
We then performed three computations, labeled “With cache”, each run with a cache containing only the results of the Baseline run, potentially reusing intermediary results, each one can be compared to its corresponding “From scratch” run duration. The runs labeled “—” show the speed gains when running with other parameter values. The values of parameters are shown in columns: tsl, ec and kwc.
With this cache system, computations could be performed of-line , allowing users to change parameters while benefiting form cached results.
Another optimization is the reduction of the number of file openings/closures during the discretization step. Time-slice files were opened and closed for each document, but, after implementing this optimization, the files are kept open as long as possible thanks to a circular bufer system (at most 512 files are kept open by the bufer to comply to the limitations imposed by the operating system, see ulimit open files). On a database composed of 20 000 documents, where 14 time-slices were needed, those files are opened/closed only once, at the beginning/end of the discretization step, saving 1.5 seconds on the total runtime.
• Multi-threading to speed up compatible computations • Automatic CSV format detection (separator, date/time format, column names) • Compatibility with date-only or date-time timestamps
We noticed that the keywords produced by the original MABED implementation to describe an event were often dificult to understand, as they covered several contexts. Therefore, we decided to label events with full-text excerpts (short descriptions of existing press articles) in order to enable an easier understanding by users. In order to do this we implemented a keyword based document search method to find the best matching texts according to the keywords generated to describe the event.
On the dataset were run 4 diferent methods to find descriptions for the detected events, used to query and rank potential descriptions (documents taken from the main dataset, for instance) and pick the best one(s) to describe a specific event.
With this method, the score of a document is simply the number of event terms that appear at least once in the document. Repetitions of a word do not increase the score of a document. Let T, the set of main and secondary terms of the event ( ∈ T) that are also present in the document ( ∈ W). The score of a document is: , = |T,| ∈ [0; 1]
|T|
With this method, the score of a document is the weighted sum of the number of event terms that appear at least once in the document. Repetitions of a word do increase the score of a document, but this efect is mitigated by the use of a logarithmic factor. Let , be the count of occurrences of the event term in the document . The weight of a term is 1 for the main terms, and their normalized magnitude for the secondary terms. The score of a document is: , = ∑︁ · (1 + log (,)) ∈T, 2.4.3. BM25: Okapi BM25 BM25 is a ranking function developed in the 1970s used to estimate the relevance of documents to a given search query. Despite its old age, variants of BM25 are still considered to be state-ofthe-art. For our experiment, we chose to use the ATIRE BM25 variant, which is often considered a baseline implementation. This variant is biased towards shorter documents, a drawback which becomes a desired property when we want concise event descriptions.
Using the spaCy Python Natural Language Processing framework, we produced an average vector embedding for each event based on documents that could describe it. With this average vector, we can rank each document by its distance to this average embedding (cosine similarity).
Let V the vector embedding of document . To construct document embeddings, spaCy computes the average of all the embeddings of the words of the document. Let V the mean vector embedding of the set of all documents ∈ that include at least one of the terms of the event , i.e. documents that could describe the event. The score formula is: , = cos (︀ V, V)︀ =
V · V ‖V‖ ‖V‖
The goal of our experiment is to evaluate the accuracy of each of the 4 labeling methods we described in the Section 2.4. Participants ( = 2) were provided with the interface depicted in Figure 2. They were given a list of 32 events, with 3 proposed descriptions for each. They were asked, for each full-text description, to indicate whether the proposed text is a correct description of the event, in their opinion, and rank the description according to its perceived correctness.
Based on the human annotators’ answers, we arrive at the results shown in Table 3. The first column is the name of the event description ranking method being evaluated. The following four columns present the percentage of correct proposals for which: the first ranked text is valid, the first ranked text is the best among the 3 presented texts, the top three proposed texts are valid, and finally the percentage of events for which all 3 proposed text are valid and their order is correct (the first description being the best, and the third being the least representative).
The most relevant metrics for our use case are the first two columns (1 st correct, 1st best). Indeed, they indicate the quality of the first description produced by each method, which would be the only description shown to end users. As we can see, the best results are given by the SKC Simple Keyword Counting method, which is also the fastest to run. The Term Embedding method might need thorougher work to be on par with the other methods. In this short paper, we presented an ongoing work on event detection optimization and event labeling 3. We optimized an existing method to be usable in an interactive event visualization context. We have also tested several event label search methods and were able to select a simple yet eficient one, that selects an event description taken from the input texts in order to complete the keywords provided by the baseline method.
We are currently working on the automatic estimation of the number of events present in the dataset, as well as on the dynamic estimation of the best value for the time-slice length. The creation of a larger event detection dataset and a larger scale user evaluation of user proposed labels are also in our future plans. 3The source code of our work is available online at https://github.com/CorentinForler/mabed-interactive-labeling