An increasing need to analyse event-centric cross-lingual information calls for innovative user interaction models that assist users in crossing the language barrier. However, datasets that re ect user interaction traces in cross-lingual settings required to train and evaluate the user interaction models are mostly missing. In this paper, we present the EventKG+Click dataset that aims to facilitate the creation and evaluation of such interaction models. EventKG+Click builds upon the eventcentric EventKG knowledge graph and language-speci c information on user interactions with events, entities, and their relations derived from the Wikipedia clickstream.
With a rapidly growing number of events with signi cant international impact, cross-lingual analytics gains increased importance for researchers and professionals in many disciplines, including digital humanities, media studies, and journalism. The most prominent recent examples of such events include the COVID-19 outbreak, the migration crisis in Europe, and Brexit. From the information science perspective, research on event-centric information spread across languages and communities, as well as cross-cultural and cross-lingual di erences in reporting, are of particular interest. However, very often, the language barrier hinders such research.
The development of novel methods for user interaction with event-centric cross-lingual information can help to overcome the language barrier in this context. Such methods can facilitate researchers with limited knowledge of target languages to narrow down the search space and to obtain an overview of the cross-lingual di erences e ectively and e ciently. However, currently, user interaction in multilingual settings is not su ciently studied. The benchmarks and datasets suitable for the evaluation of new methods for user interaction with cross-lingual information are mostly missing.
With the recent development of knowledge graphs that provide cross-lingual
information, such as Wikidata, DBpedia, and the event-centric EventKG
knowledge graph [
In this paper, we present a novel cross-lingual dataset that re ects the language-speci c relevance of events and their relations. This dataset aims to provide a reference source to train and evaluate novel models for event-centric crosslingual user interaction, with a particular focus on the models supported by knowledge graphs. Our dataset EventKG+Click is based on two data sources: 1) the Wikipedia clickstream2 that re ects real-world user interactions with events and their relations within language-speci c Wikipedia editions; and 2) the EventKG knowledge graph that contains semantic information regarding events and their relations that partially originates from Wikipedia. EventKG+Click is available online3 to enable further analyses and applications.
Without loss of generality, we adopt a language-speci c event ranking as an envisioned user interaction paradigm to illustrate our discussion. For example, Table 1 reveals the di erent language-speci c focus when ranking events. In each of the three languages contained in EventKG+Click, the list of most languagespeci c related events is clearly representing language-speci c views (e.g., \2016 Berlin truck attack" for German) that can be used for further exploration of events from language-speci c viewpoints. In the case of English, we see that the Southeast Asian Games are of high language-speci c relevance, which can be explained by the large percentage of Asian users of the English Wikipedia4.
In EventKG+Click, we enrich the information obtained from the Wikipedia clickstream with event and entity references from EventKG. Furthermore, we 2 https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream 3 https://github.com/saraabdollahi/EventKG-Click 4 https://stats.wikimedia.org/wikimedia/squids/SquidReportPageViewsPerCo untryBreakdown.htm create a cross-lingual view on the clickstream by combining information obtained from three Wikipedia language editions, namely English, German, and Russian. Moreover, we compute scores that re ect the language-speci c relevance of events and their relations, as indicated by the user interactions in the clickstream. Finally, to support further development of the event-centric user interaction methods in the cross-lingual settings, we analyse the correlations of the proposed scoring function and selected in uence factors.
We structure the rest of the paper as follows: First, we review related work regarding cross-lingual analytics, knowledge graphs, and the Wikipedia clickstream in Section 2. Then, we introduce our EventKG+Click dataset in Section 3. In Section 4, we propose scores to represent the language-speci c relevance of events and their relations. Given the EventKG+Click dataset and these scores, we analyse how selected factors in uence the language-speci c relevance in Section 5. Finally, we provide a conclusion in Section 6. 2
In this section, we brie y summarise related work in the areas of cross-lingual analytics, along with the aspects related to knowledge graphs and the Wikipedia clickstream.
Cross-lingual analytics and interaction. With the rise of the Web, there
came an uprise of user-generated content accessible over the whole world, leading
to knowledge diversity across languages [
In the context of cross-lingual analytics, events play a particularly important
role: When an event breaks out, this event is usually reported by a large number
of sources, whose coverage highly varies across language communities [
With EventKG+Click, our goal is to promote further cross-lingual analytics and interaction, facilitated by a combination of semantic information given in knowledge graphs and user interaction traces obtained from a clickstream.
Knowledge graphs. An essential resource to facilitate interaction with
cross-lingual information are knowledge graphs, in particular those containing
language-speci c labels and relations. Ka ee et al. [
The importance of multilingualism in knowledge graphs becomes even more
evident in the case of event-based applications. EventKG [
In our analysis, we observed that the closeness of event locations extracted from EventKG is an essential indicator to explain language-speci c relevance. Thus, we con rm the importance of event-centric and multilingual knowledge graphs in the context of cross-lingual analytics.
Wikipedia clickstream. The Wikipedia clickstream has been used as a
ground-truth to evaluate entity recommendation and relatedness in several
examples, as it reveals the navigation^al behaviour of users and their preferences
while exploring Wikipedia pages. Existing work, however, has not considered
language-speci c di erences and mainly focused on the English Wikipedia
clickstream: For example, Tran et al. used the English Wikipedia clickstream as
ground truth for constructing entity-context queries [
The Wikipedia clickstream holds the interaction of real users with the articles representing events and entities in the speci c Wikipedia language editions and their relations. In particular, the clickstream contains the counts of the (source, target) pairs extracted from Wikipedia's request logs. The clickstream contains all the requests to a Wikipedia page, including links from and to external web pages. As EventKG+Click and our analysis are based on Wikipedia click behaviour, we only consider those (source, target) click pairs in the clickstream where both the source and target are Wikipedia articles connected by a hyperlink.
In this work, we adopt the Wikipedia clickstream that covers the period from December 1, 2019, to December 31, 2019, and contains nearly 19; 521; 580 click pairs for the English, 2; 902; 878 click pairs for the German, and 2; 752; 340 click pairs for the Russian Wikipedia.
EventKG is an event-centric knowledge graph that contains more than 1:2 million events and more than 4 million temporal relations in nine languages in its release from February 2020. Knowledge graphs such as EventKG, DBpedia, and Wikidata include information extracted from the multilingual Wikipedia as the basis. This way, data regarding user interaction with Wikipedia articles and links, available from the Wikipedia clickstream dataset, can be directly mapped to the events, entities and their relations in these knowledge graphs.
When creating the proposed EventKG+Click dataset, we assume that: 1) the events of global importance are re ected in Wikipedia clickstreams of several languages, and 2) a clickstream in a speci c language re ects the importance of events and their relations as perceived by the users of the speci c Wikipedia language edition. Based on these assumptions, we employ the intersection of language-speci c clickstreams to build a dataset for training and evaluation of cross-lingual user interaction. In particular, we map the events and entities included in the Wikipedia clickstream to EventKG and extract relations for these events from all language-speci c clickstreams. Furthermore, we compute scores that represent the language-speci c relevance of events and their relations. These scores are presented in Section 4. To enable further cross-lingual analysis, we enrich EventKG+Click with several in uence factors extracted from EventKG and Wikipedia, which are presented in Section 5.
In EventKG+Click, we only consider entities that are clicked at least 10 times per language, so that we capture those entities that are of global importance and do not consider entities solely present in single Wikipedia language versions. We also only consider pairs which exist in the clickstreams of all considered languages and in which the target page is an event.
The resulting EventKG+Click dataset is available online5 and contains relevance scores for more than 4 thousand events, and nearly 10 thousand eventcentric click-through pairs. 4
To allow cross-lingual analytics with EventKG+Click, we need to capture the language-speci c relevance of events and their relations. Based on the Wikipedia clickstream, we propose two scores that rule out language-independent relevance.
To describe our scores, we rst de ne the concepts used for the computation: { L is the set of languages under consideration. The current release of Event
KG+Click comes in English, German, and Russian: L = fEN; DE; RU g. { E is the set of entities contained in EventKG+Click, that are all represented by speci c Wikipedia pages and EventKG resources. Formally, named events considered in this work are a speci c type of entity and thus included in E. { clicks(es; et; l) represents the number of clicks from the source entity es 2 E to the target event et 2 E in the clickstream of the given language l 2 L.
We distinguish between two scores de ned in the following: language-speci c event relevance and language-speci c relation relevance. 4.1
Wikipedia language versions di er a lot concerning the number of their active users, edits, and articles. For example, the English Wikipedia has 7:2 times as many active users as the German Wikipedia6. The clickstream also re ects this imbalance: There are 7 times more clicks in the English clickstream than in the German one. To observe language-speci c behaviour, we rst need to level the e ects that originate from the popularity of the speci c Wikipedia language versions. To do so, we normalise the number of clicks with respect to the total number of clicks in the respective language, which leads to normalised scores in the range [0; 1]. In order to create balanced click counts, we then multiply the normalised score by the total number of clicks in the clickstreams, as follows: balanced clicks(es; et; l) = clicks(es; et; l) P l02L Pe0s2E Pe0t2E clicks(e0s; e0t; l0)
Pe0s2E Pe0t2E clicks(e0s; e0t; l)
The popularity of an event can be inferred by the number of user interactions with its Wikipedia page. That way, we can identify the most popular events in a given language l 2 L by summing up all clicks from and to an event e 2 E: balanced clicks(e; l) = X balanced clicks(e; et; l)+ X balanced clicks(es; e; l) et2E es2E
As we focus on the language-speci c relevance in EventKG+Click, we need to rule out the events that are highly ranked across all languages under consideration. Therefore, we normalise the language-speci c click count by the overall number of clicks in all languages:
balanced clicks(e; l) event relevance(e; l) = Pl02L balanced clicks(e; l0) 2 [0; 1]
With this relevance score, events that are clicked often in a given language l 2 L, but rarely clicked in the other languages are assigned a relevance score close to 1. 4.2
To identify events relevant to a given source entity, we de ne the languagespeci c relation relevance score. This score assigns a relevance score to the relation between a source entity es and a target event et in a given language.
Similarly to the language-speci c event relevance, the language-speci c relation relevance is computed as the fraction of clicks in the given language compared to all languages:
balanced clicks(es; et; l) relation relevance(es; et; l) = Pl02L balanced clicks(es; et; l0) 2 [0; 1] Note that this score rules out the e ects resulting from the relevance of the source entity: Events that are highly related to an entity e can obtain relevance scores close to 1 independent of e's click count. 4.3
In Table 1 in Section 1, we have given an example of the language-speci c event
relevance, i.e., that table provides the top-ranked events per language,
according to our language-speci c event relevance score. As discussed before, we can
clearly observe events which are intuitively important for the respective language
community.
Given the EventKG+Click dataset with the relevance scores de ned in the
previous section, we now discuss several in uence factors that can potentially
impact the language-speci c relevance of events and analyse their correlations with
the proposed relevance scores. As in uence factors we consider language
community relevance, event location closeness and event recency, as de ned in the
following. In future work, we plan to investigate the role of further in uence
factors, as for example the event type that has been shown to in uence the
click-behaviour [
The language community relevance factor re ects the importance of an event for the community that speaks this language. We assume that events relevant for the language community should be mentioned and referred to more often in a language-speci c corpus.
Based on this assumption, we measure the language community relevance by counting the links to the event article and mentions of the event within the speci c Wikipedia language edition7. Dependent on the context (i.e., event or relation relevance), we make use of two in uence factors: { Links pointing to the event: The number of links in the whole Wikipedia language edition that link to the event article. { Co-mentions of a relation: The number of sentences in the whole Wikipedia language edition that jointly mentions the (source, target) pair participating in the relation. 5.2
The event location closeness factor expresses the intuition that users are likely to be interested in the exploration of local events, i.e., events located in spatial proximity of the user. To re ect this intuition, we introduce a binary in uence factor that indicates whether an event happened in a location where the respective language l 2 L is an o cial language. For example, the Battle of Stalingrad may be particularly important from the Russian perspective in the context of the Second World War. To compute this factor, we rst identify event location(s) using the sem:hasPlace8 property of EventKG and then derive the o cial languages of the location's country. 5.3
Wikipedia is heavily in uenced by recent events: Users tend to edit and read
articles about events that are happening right now [
Given EventKG+Click and the in uence factors, we now investigate the correlations between such in uence factors and the language-speci c relevance scores. To this end, we compute the Pearson correlation coe cients in several con gurations.
First, we compute the correlations of in uence factors with language-speci c event relevance scores of the events covered in the Wikipedia clickstream of all considered languages (i.e., event relevance, as de ned in Section 4). As in uence factors we select the event location closeness (Location), the number of links pointing to the respective event (Links), and the event recency (Recency ). Results are shown in Table 3.
The Location in uence factor for events indicates the largest positive correlation, which con rms the existence of di erent language viewpoints. This e ect can be most notably observed in the case of English, which has a correlation of 0:4 between the event relevance score and the Location closeness in uence factor. The other two in uence factors, namely Links and Recency, do not show any notable correlation. We assume that this is because the users are interested in both, recent and historical events, whereas recent events might not be well interlinked in Wikipedia yet.
Until now, we have considered the language-speci c event relevance scores, i.e., scores assigned to each event in isolation. Now, we investigate the user click behaviour from the perspective of the event relations (i.e., relation relevance, as de ned in Section 4). In particular, we focus on the properties of the target event, as the language-speci c relation relevance score is independent of the source entity's relevance.
The following in uence factors are used in this correlation analysis: i Location: The location closeness of the target event. ii Links: The number of links to the target event in Wikipedia. iii Recency : The recency of the target event. iv Co-Mentions : The number of co-mentions of the relation source and target in Wikipedia.
The correlation results are shown in Table 4. The correlation coe cient for the language-speci c relation relevance con rms our observations concerning the language-speci c event relevance. The closeness of the target event location has the largest in uence on language-speci c relevance. The links, recency and comentions do not correlate with the relevance scores in any of the three languages. That means, if the user reads a particular Wikipedia article, there is a higher chance that the next click leads to a spatially close event than to an event that is mentioned many times together with the source entity. 6
In this paper, we presented the EventKG+Click dataset and suggested scores for capturing language-speci c relevance scores for events and their relations. EventKG+Click builds upon the EventKG knowledge graph and language-speci c traces of user interaction with events derived from the Wikipedia clickstream. The resulting EventKG+Click dataset contains click counts and relevance scores for more than 4 thousand events and more than 10 thousand (source, target) pairs in English, German, and Russian. Furthermore, we analysed several in uence factors of language-speci c relevance. We believe that the EventKG+Click dataset is a valuable resource to evaluate event relevance in language-speci c contexts. In future work, we plan to develop novel user interaction models supporting cross-lingual event-centric analytics, where we will adopt the EventKG+Click dataset for training and evaluation.
Acknowledgements This work was partially funded by H2020-MSCA-ITN2018-812997 under \Cleopatra".