Multilingual Web Person Name Disambiguation is a new shared task proposed for the rst time at the IberEval 2017 evaluation campaign. For a set of web search results associated with a person name, the task deals with the grouping of the results based on the particular individual they refer to. Different from previous works dealing with monolingual search results, this task has further considered the challenge posed by search results written in different languages. This task allows to evaluate the performance of participating systems in a multilingual scenario. This overview summarizes a total of 18 runs received from four participating teams. We present the datasets utilized and the methodology de ned for the task and the evaluation, along with an analysis of the results and the submitted systems.
It is increasingly usual for people to turn to Internet search engines to look for information about people. According to Google Trends, three out of the top 10 Google Searches in 2016 were linked to person names5. However, person names tend to be ambiguous and hence a search for a particular name likely includes results for different individuals. In these cases, a list of individuals included in the results along with a breakdown of different individuals would come in handy for the user who is looking for a particular individual. This task was rst introduced in the WePS (Web People Search) campaigns6, and attracted substantial interest in the scienti c community, as manifested in a number of shared tasks that tackled it, particularly the WePS-1, WePS-2, WePS-3 campaigns [1{ 3]. These campaigns provided several annotated corpora becoming a referent for 5 https://trends.google.com/trends/topcharts#geo&date=2016 6 http://nlp.uned.es/weps/ this problem and allowing a comparative study of the performance of different systems. However, all those campaigns presented a monolingual scenario where the query results were written in only one language.
Despite the multilingual nature of the Web7, existing work on person name disambiguation has not considered yet search results written in multiple languages. The objective of M-WePNaD task is centered around a multilingual scenario where the results for a query, as well as each individual, can be written in different languages.
The remainder of this paper is organized as follows. Section 2 presents the task. Next, Section 3 describes the datasets we released for training and testing, and we brie y discuss the differences between the two sets. Section 4 brie y describes the evaluation measures. Section 5 summarizes the proposed approaches of the participants. Section 6 presents and discusses the results. Finally, conclusions are presented in Section 7. 2
The M-WePNaD task is a person name disambiguation task on the Web focused on distinguishing the different individuals that are contained within the search results for a person name query. The person name disambiguation task can be de ned as a clustering problem, where the input is a ranked list of n search results, and the output needs to provide both the number of different individuals identi ed within those results, as well as the set of pages associated with each of the individuals.
The heterogeneous nature of web results increases the difficulty of this task.
For instance, some web pages related to a certain individual could be professional
sites (e.g. corporation web pages), while others may contain personal information
(e.g. blogs and social pro les) and both kinds of web pages could have very little
vocabulary in common. Particularly, [
While previous evaluation campaigns had been limited to monolingual scenarios, the M-WePNaD task was assessed in a multilingual setting, considering the realistic scenario where a search engine returns results in different languages for a person name query. For instance, web pages with professional information for an individual who is not a native English speaker may be written in English, while other personal web pages may be written in their native language. Celebrities who are known internationally are also likely to have web pages in different languages.
We compiled an evaluation corpus called MC4WePS [
The MC4WePS corpus was collected in 2014, issuing numerous search queries and storing those that met the requirements of ambiguity and multilingualism.
Each query includes a rst name and last name, with no quotes, and searches were issued in both Google and Yahoo. The criteria to choose the queries took into account: { Ambiguity: non-ambiguous, ambiguous or highly ambiguous names. A person's name is considered highly ambiguous when it has results for more than 10 individuals. Cases with 2 to 9 individuals were considered ambiguous, while those with a single individual were deemed non-ambiguous. { Language: results can be monolingual, where all pages are written in the same language, or multilingual, where pages are written in more than one language. Additionally, for each cluster of pages belonging to the same individual, we considered whether the results were monolingual or multilingual. This was due to the fact that even though the results for a person name query are multilingual, the clusters for each different individual could be monolingual or multilingual.
The MC4WePS dataset contains search results of 100 person names with
a number of search results between 90 and 110 each. It is worth noting that
different person names in the corpus have different degrees of ambiguity; in
addition a web page can be multilingual, and not all the content in the corpus
are regular HTML web pages, but also other kinds of documents are included,
such as social media posts or pdf documents. A detailed description of the corpus
can be found in [
There can be overlaps between clusters as a search result could refer to two or more different individuals with the same name, for instance social pro le pages with lists of different individuals with the same name. When a search result does not belong to any individual or the individual cannot be inferred, then this is annotated as \Not Related" (NR). For each query, these search results are grouped as a single cluster of NR results in the gold standard annotations.
The MC4WePS corpus was randomly divided into two parts: training set (65%) and test set (35%). 3.1
We provided participants with a single set for training, which includes 65 different person names, randomly sampled from the entire dataset. The list of names and their characteristics can be seen in Table 1. The second part of the table contains in the last row the average values for the different data of the whole training set. 3.2
The test corpus consists of 35 different person names, whose characteristics can be seen in Table 2. The last row shows the average values for the different data of the whole test set. 3.3
As can be seen in Table 1 and Table 2, the training and test sets have a comparable average composition with regard to the percentages of NR search results and social web pages. The two sets are also similar in terms of the distribution of the most common language for a given person name. In the test set, the percentages are 28.57% ES, 68.57% EN, and 2.86% FR; whereas in the training set they are 30.76% ES, 67.69% EN, and 1.55% FR.
The main difference between both sets lies in the degree of ambiguity of the
person names. Based on the threshold de ned by Montalvo et al. [
The datasets are structured in directories. Each directory corresponds to a speci c search query that matches the pattern \name-lastname", and includes the search results associated with that person name. Each search result is in turn stored in a separate directory, named after the rank of that particular result in the entire list of search results. A directory with a search result contains the following les: { The web page linked by the search result. Note that not all search results point to HTML web pages, but there are also other document formats: pdf, doc, etc. { A metadata.xml le with the following information:
URL of search result.
ISO 639-1 codes for languages the web page is written in. It contains a comma-separated list of languages where several were found.
Download date.
Name of annotator. { A le with the plain text of the search results, which was extracted using Apache TiKa (https://tika.apache.org/).
Figure 1 shows an example of the metadata le for a search result for the person name query Julio Iglesias.
The access to training and test sets was restricted to registered participants.
The blind version of the test dataset did not include the ground truth les.8
8 The MC4WEPS corpus is available at http://nlp.uned.es/web-nlp/resources.
We use three metrics for the evaluation: Reliability (R), Sensitivity (S) and
their harmonic mean F0:5 [
Thirteen teams signed up for the M-WePNaD task, although only four of them managed to participate in the task on time, submitting a total of 18 runs.
In what follows, we analyze their approaches from two perspectives: search result representation (including whether or not translation resources were used), and the clustering algorithms.
{ The ATMC UNED team [
We produced two different rankings of the participants after evaluating all the submissions: { Evaluation results by not considering the Not Related results. This means that all the results of this kind and the corresponding cluster were not taken into account. { Evaluation results considering all web results. This means that all the results and the clusters were taken into account. The results obtained by all their runs are quite similar. However, Run 1 uses features from the original content of the web pages and gets worse results with respect to Run 2 and Run 3, which use a machine translation tool. Run 4 compares the web pages written in different language with their comparable features and gets similar results than Run 2 and Run 3 without the need of translation resources. The main advantage of this last approach is that it avoids additional preprocessing steps dedicated to translating the web pages, which is desirable in problems which have to be solved in real time. The ATMC UNED team has not proposed any method to group not related web pages, so their Sensitivity and the F-measure results are worse when considering them in the evaluation.
The results obtained by the LSI UNED team overcome the results obtained by the baselines but are lower than results obtained by the ATMC UNED team. Going into detail, using all the words in the documents (Run 5) is under the run that only considers named entities and uses the same threshold (Run 2). This implies that the addition of all the possible words in the documents introduces more noise than valuable information. On the other hand, in general, if using named entities and increasing the threshold value, the Reliability increases while the Sensibility decreases. Finally, considering all web pages can be seen as a more difficult task, but the number of unrelated web pages in the corpus is small and hence the results are quite similar between these two settings which use a threshold-based clustering approach.
The PanMorCresp Team Run 1 and Run 2 perform about equally well regardless of whether or not related pages are taken into account in the evaluation. Run 1 achieves good Reliability, which ts well with the fact that complete linkage was used. This comes at the cost of a low Sensitivity. For Run 2, the picture is reversed. Run 3 and Run 4 obtain a higher score than Run 1 and Run 2. Punctuation removal, stop word removal and the larger vocabulary may play a role in this. In addition, HAC single linkage was used in both of these runs. Run 4 is the best of the PanMorCresp Team runs; the only difference with regard to Run 3 is that it uses a xed number of clusters (9) as a stopping criterion. The score of Run 4 beats both baselines and is on par with scores obtained with the other approaches save the scores obtained by the ATMC UNED runs.
Out of the ve runs submitted by the Loz Team, only Run 1 managed to outperform the ALL-IN-ONE baseline. The rest of the runs only managed to outperform the ONE-IN-ONE baseline, performing worse than the ALL-IN-ONE baseline. One of the main reasons why these approaches did not perform as well may be due to the fact that the multilingualism and overlaps between clusters have not been considered, posing a signi cant limitation for this task. Their best performing approach (Run 1) uses a weighted representation of words, which shows that considering the frequency of words in documents leads to better performance than the sole use of a binary representation capturing the presence or not of words as well as TD-IDF. They also found that considering ve clusters as the stopping criterion, instead of 15, leads to an increased Sensitivity score, which is however at the expense of a little drop in Reliability. 7
The M-WePNaD shared task on Multilingual Web Person Name Disambiguation took place as part of the IberEval 2017 evaluation campaign. This shared task was the rst to consider multilingualism in the person name disambiguation problem, following a series of WePS shared tasks where the corpora were limited to documents in English. The M-WePNaD shared task provided the opportunity for researchers to test their systems on a benchmark dataset and shared task, enabling comparison with one another.
Despite a larger number of teams registering initially for the task, four of them managed to submit results on time, amounting to 18 different submissions. Only two of the four participants, namely the champions and the runners-up, made use of more sophisticated clustering algorithms, whereas the other two relied on the Hierarchical Agglomerative Clustering (HAC) algorithm. Only one of the teams presented an approach that does not require any prior knowledge to x thresholds, which came from the team that quali ed in the top position. We argue that this is a desirable characteristic for web page clustering, owing to the heterogeneous nature of the Web, which poses an additional challenge for learning generalizable patterns.
With respect to the approaches used for web page representation, most of the teams relied on traditional techniques based on bag-of-words and vector space models, with the exception of the runners-up, who used word embeddings.
While the novel aspect proposed in this shared task has been the multilingual nature of the dataset, only one team has proposed approaches that explicitly tackles multilingualism, ATMC UNED, particularly it has explored three approaches. The results obtained for these approaches slightly outperform the one that does not consider multilingualism. On the other hand, the dataset also included web pages from social media, unlike in previous shared tasks. However, only one of the teams, ATMC UNED, has taken this into account when developing their system. None of the systems has dealt with unrelated results and overlapping clusters.
This work has been part-funded by the Spanish Ministry of Science and Innovation (MAMTRA-MED Project, TIN2016-77820-C3-2-R and MED-RECORD Project, TIN2013-46616-C2-2-R).