We present the RGAI systems which participated in the third Web People Search Task challenge. The chief characteristics of our approach are that we focus on the raw textual parts of the Web pages instead of the structured parts, we group similar attribute classes together and we explicitly handle their interdependencies. The RGAI systems achieved top results on the attribute extraction subtask, and average results on the clustering subtask.
Personal names are among the most frequently searched items in Web search
engines. At the same time these types of search results ignore the fact that one name
may be related to more than one person. Sometimes person names are highly
ambiguous. The first WePS challenge organized in 2007 focused on this
disambiguation problem. As input, the participants’ systems received Web pages
retrieved from a Web search engine using a given person name as a query. The aim of
the task was to find all the different people among the results lists and assign a
corresponding document to each person. During the evaluation of WePS1, the
organizers realized that some attributes are very useful for the person disambiguation
problem. Hence the second WePS challenge organized in 2009 contained an
absolutely new challenge. The attribute extraction subtask was to identify 16 different
attributes from Web pages such as birth date, affiliation, and occupation. This subtask
proved very difficult and the best system only achieved an F-measure score of 12.2.
The third WePS shared task introduced a novel subtask which sought to mine
attributes for persons, i.e. the attribute extraction from the clusters of pages belonging
to each given person. We will now describe our system that participated in this third
WePS challenge.
The aim of Web Content Mining is to extract useful information from the natural
language-written parts of websites. The first attempts on Web Content Mining began
with the Internet began around 1998-’99 [
The person name disambiguation subtask of the second WePS challenge [
We shall focus on the raw text parts of the Web pages because we found that more pages express content in textual form than in structured form [16]. The first step of information extraction may be to construct a good section selection module. When handling the problem, we first extract the candidate attributes from the relevant sections of Web pages, then we cluster the pages by merging clusters having common person attributes and aggregate attributes to the persons identified.
The input of the participants’ system was a set of pages retrieved from a Web search engine using a given person’s name as a query. We assumed that useful information is available in the natural language-written part of websites and tables [16]. This is why we concentrated on the natural language-written part of websites and tables, and we discarded a lot of noisy and misleading elements from pages (e.g. menu elements). These elements can seriously hinder the proper functioning of Natural Language Processing (NLP) tools.
In order to identify textual paragraphs we applied the Stanford POS tagger for each
section of the DOM tree of the HTML files. We assumed that one piece of text was a
textual paragraph if it was longer than 60 characters and it contained more than one
verb. We extracted several attributes with our own Named Entity Recognition (NER)
[
Like other WePS2 systems [12],[
When handling this subtask of attribute extraction, it seems necessary to classify the attribute classes in several ways. First, we aggregated similar attributes into logical groups. For instance, the name group contains the other name, relatives and mentor attribute classes On the other, we can assume subordinate relations among the coherent attributes. For example, we only marked a candidate name as mentor if it was not relatives or other name. Next, we will elaborate on the extraction procedure for each of the attributes. Date of birth: if a paragraph contains born, birth or birthday phrases we find candidate dates with a date validator within a window of the word. This validator works with 9 different regular expressions rules, and can identify dates written in different formats in the span of text.
Birth place: when a paragraph contains born, birth, birthplace, hometown and native
phrases we use the location markups given by the NER tool [
Occupation: according to the WePS2 results, it was one of the most difficult,
ambiguous and frequent attribute classes, which is due to the abstract nature of this
attribute. Hence we avoided using lists. It is not available in any NER model or
training database. So we created a training database by matching all gold annotation
to paragraphs. We used simple string matching and we did not know where the actual
attribute occurred. However, the resulting dataset was very noisy. We trained our
NER tool [
Organizations (school, award, affiliation): we found that these types of attributes
were names of organizations so we grouped them together. We also used an NER tool
[
Names (relatives, other name, mentor): these types of attributes are person names so
we found that they occur together. To identify name attributes we used an NER tool
[
When extracting availability attribute classes we did not used just the textual paragraphs, but examined the whole text of Web pages as these types of attributes may occur in other parts as well.
Phone: when a text contains tel, telephone, ph:, phone, mobile, call, reached at, office, cell or contact words or a part of the original name, we applied the following regular expression:
(((?[0-9+(][.()0-9s/-]{4,}[
E-mail: we assumed that if somebody offers their e-mail address, it is also a link. Therefore, we examined links that contain the mailto tag. Moreover, we assumed that every mail address contains the original name or one part of the original name. Hence we defined an e-mail address validator that validates e–mail addresses. We generate all character trigrams from the original name and when an e-mail address contains at least one of them, the validator accepts it. We defined a stop list as well. This list contains words such as wiki, support, and webmaster. Should a candidate e-mail address contain one from the stop list, the validator does not accept it. Next we extracted the domain from all accepted e-mail addresses, which we used for the website attribute.
Web-site: we assumed that when somebody displays a Web address on a website, it is also a link too, so a Web address is a link at the same time. In this case we only extract a website attribute from links. We marked a potential candidate attribute as a website when it contained the original name or one part of the original or extracted domain name from the e-mail attribute.
Our chief hypothesis in the clustering subtask was that it can be effectively solved by using extracted person attributes. We defined a weighting of attribute classes. The most useful attribute classes were web address, e-mail, telephone, fax number and other name and they got a weight of 3. In addition, we weighted birth date as 2 while birth place, mentor, affiliation, occupation, nationality, relatives, school, and award each got a weight of 1. Then every document was represented by a vector with extracted person attribute values.
To define a document similarity measure, we needed to normalize the attribute values, i.e. spelling variants and synonyms have to be handled as equivalents. We developed individual normalization rules for each attribute class. For example, the birth place of United States of America could be referred to as USA, U.S.A. United States, Federal United States and so on. Here, we created a synonym dictionary based on the re-direct links of the English Wikipedia and we developed regular expressions or transformation rules for other attribute classes.
As a first approach for Web page clustering, a bottom-up heuristic clustering was performed. Here the starting clusters consist of the individual Web pages and then the clusters are merged iteratively until a stopping criterion is reached. For each step of this procedure the most similar clusters are merged (the union of their attributes formed the attribute set of the resulting cluster), where the similarity measure of the weighted size of the intersection of the cluster attribute sets was employed using normalization rules. The stopping criterion was defined to be a similarity value threshold of 2, i.e. if the similarity value of the closest clusters is less than 2 the procedure is terminated (RGAI5 submission).
Besides this heuristic bottom-up clustering, we employed the Xmeans algorithm in the WEKA Java package [15] as well. The advantage of this approach is that it is not necessary to define the number of clusters, but we can define the minimum number of clusters. We used the final number of clusters obtained by the heuristic clustering as the minimum number of clusters for Xmeans. (RGAI3)
In addition to person attribute-based Web page clustering, we also experimented with a text-based approach. With the results of RGA1, we only used the search engine snippet data. These types of representation compress the most important pieces of information. We represented the dataset with the tf-idf vector space model where every document is a vector. The RGAI1 and the RGAI2 results were almost identical. Lastly, RGAI4 is a hybrid method of the above two approaches, i.e. the feature sets of the person-based attribute and the snippet-based clustering were merged.
As a last step, we had to aggregate those attributes that occurred in Web pages and were found in a cluster of pages, i.e. belonged to a particular person. The official evaluation metric of the challenge required only one attribute from each class. As we extracted more than one potential attribute values for each class, we had to choose one (e.g. a person may mention several of his affiliations). In the end we chose the most frequent element per person from each attribute class. When some attribute frequency was equal, we just chose it at random.
Because the WePS3 attribute extraction subtask required clustering, the documents
we submitted ran the attribute extraction and clustering tasks as well. The test dataset
was composed of 300 person names and nearly 200 Web documents for each name.
The attributes had to be assigned to each person cluster rather than to individual
pages. The training dataset was the WePS2 train and test sets, which contains 5,122
websites with 187,032 textual paragraphs. We found 2,781 affiliation, 3,419
occupation and 2,092 biographical paragraphs. For the location, organization and
names, markups given by the NER tool [
During the evaluation of the clustering subtask the organizers used the extended versions of BCubed Precision and Recall, which was the official evaluation metric with alpha set to 0.5. They evaluated the clustering of documents for each query just focusing on two different people, except for 50 names, where only documents about one person were considered. The official results on the clustering task of the RGAI systems, the best performing participant and two baselines are shown in Table 1. Here our RGAI 1 system achieved the best scores. For the attribute extraction subtask1, the evaluation metrics were computed as follows, Precision: for a given person, it is the number of correct attribute/value pairs divided by the total number of attribute/value pairs extracted.
Recall: for a given person, it is the number of attributes having at least one correct value divided by the total number of attributes for which a correct value has been found by at least one of the systems.
F-measure: 1 / (alpha * 1/prec + (1-alpha) * 1/rec), where alpha was 0.5. The above defined “given person” is taken from the prediction of the clustering subtask. The gold standard annotation of clustering consists of two person (clusters) for every document set. During the evaluation process the most similar predicted clusters was taken into account where the F score or recall was used as similarity metric, where Precision: the number of documents in the cluster that refer to the person / number of documents in cluster. 1 Please note that at the time of preparing the workshop proceedings, official results for the attribute extraction subtask were not available due to unexpected difficulties of the task organizers with the manual assessments. Due the organizers, the results of the paper are achived on the 12.5 percent of the test dataset.
Recall: the number of documents in the cluster that refer to the person / number of documents that refer to the person.
Next, the organizers defined two different interpretations of the manual annotations, which were combined with the other two clustering evaluation options. Strict evaluation: we count as correct all attribute / value pairs judged as correct by a majority of annotators and as incorrect otherwise.
Lenient evaluation: we count as correct all attribute / value pairs judged as correct or inexact by a majority of annotators, and as incorrect otherwise.
Table 2 shows the results of the RGAI systems when the clustering resemblance was the recall approach and the manual annotation was lenient. Our best result was achieved by the RGAI 3 system, but the Intelius system was outstanding. However, when we used the lenient annotation interpretation and the clustering approach based on the F score, our RGAI 3 system achieved significantly better results (see Table 3). When we used the strict annotation and recall-based clustering, the results of Intelius system were dramatically better than those of other systems. It was able to cluster the documents better (see Table 4. ). Finally, Table 5 shows the results when the clustering approach was based on the F score and the annotation was strict. RGAI 3 could achieved the best result systems performed fairly well. The above tables show that our approach achieved an F score sligthly above 10 of F score based clustering. Compared to the WePS2 results – where the best system achieved about an of F score of twelve – these results are competitive as we solved a more complex problem here. On the other hand, the recall-based results show that our clustering approach has to be improved.
In this article we presented a person name disambiguation method with biographical attribute extraction from documents related to a person. We handled the name disambiguation problem from person Web search results. Our method is based on extracted biographical attributes and snippet information. The proposed clustering method was evaluated using the test dataset created for the name disambiguated subtask of the third Web People Search Task. Our clustering approach got an F score of 40 and was ranked fourth among the eight participants.
For the second subtask of the shared task, our method efficiently extracted the different types of attributes from Web pages and we achieved top results on the WePS3 challenge. We think that the reasons for the success of our attribute extractor are the followings. First, our approach groups attribute classes and introduces rules which efficiently handle the interdependencies among these classes. Second, we focused on the textual parts of the web pages using NLP tools which demonstrates that raw text parts of person Web pages should be analyzed besides the structured parts of the pages.
This work was supported in part by NKTH grant of the Jedlik Ányos R&D Programme (project codename TEXTREND) of Hungarian government. The authors would like to thank the shared task organizers for their devoted efforts.
15.Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, Ian H.
Witten (2009); The WEKA Data Mining Software: An Update; SIGKDD Explorations, Volume 11, Issue 1. 17.István Nagy, Richárd Farkas and Márk Jelasity. Researcher affiliation extraction from homepages. NLPIR4DL ACL Workshop 2009 pp 1-9