Most Information Retrieval models take documents as Bagof-Words and are thereby bound to the language of the documents. In this paper, we present an approach using Linked Open Data resources, i.e. URIs, as interlingual document representations. Documents and queries are summarized by the resources they contain. We show the applicability of our approach for multilingual retrieval with a case study on expert search.
When encountering a problem, there are often two ways to get to a solution. Either acquire the knowledge, in order to solve the problem by oneself, or ask for outside help, preferably somebody who has expertise and experience in the needed domain. The rst case is often not feasible or would require too much time. In the second case, the subsequent problem of nding the right expert arises. We address this problem and present an approach for expert search in this paper.
Identifying who an expert is for a certain domain can be done in many ways. One possible solution is to use documents and assume that the authors have expertise on the topics they wrote about. We apply this assumption and consider documents for the identi cation of experts.
Obviously, the more speci c the problem is the harder is it to nd an expert. Thus, extending the considered search space even across languages improves the situation. The scenario of considering documents in di erent languages is not an arti cial one, e.g. global companies have product documentations in many languages or online developer forums have discussion threads in di erent languages. Our approach addresses the problem of how to deal with di erent languages by applying an interlingual representation for documents based on Linked Open Data resources.
Expert Search Track at CriES Our approach participated at the expert search
track of the Cross-lingual Expert Search Workshop (CriES) at CLEF 2010 [
This paper is organized as follows. After the introduction in this section, we describe the usage of Linked Open Data as an interlingual representation in Section 2. In Section 3, we present our model for expert search, how we create pro les between resources and experts and how we estimate parameters. Section 4 presents the evaluation and Section 5 discusses related work. Finally, we conclude in Section 6. 2
Most common models in Information Retrieval see documents as Bag-of-Words,
i.e. they resolve the order of the words and take the collection of words as the
representation of a document. As a consequence, this representation is directly
bound to the language of the document. When using keyword queries in one
language, relevant documents in another languages are probably not retrieved.
We propose an approach using Linked Open Data resources as document
representation. Linked Open Data (LOD) refers to interlinked, publicly available,
and structured datasets on the web using semantic web standards, in particular
the Resource Description Framework (RDF) [
The rst principle of LOD states that things should be identi ed by Uniform Resource Identi ers (URIs), where things, i.e. resources, can be virtually everything. The notion is not limited to physical things, but comprises also abstract or intangible concepts, like happiness or re alarm. URIs are not necessarily human readable, since they are meant to be processed by machines. Therefore, human readable labels are often assigned to URIs. Since there can be multiple labels in di erent languages for one URI, the URI itself can be seen as an interlingual representation for the resource it identi es. Figure 1 illustrates an example about the resource representing Germany and its labels in several languages.
The resource in Figure 1 is taken from DBpedia2. DBpedia is a popular
LOD dataset extracted from Wikipedia, which exploits the interlanguage links
1 This dataset is provided by the Yahoo! Research Webscope program (see http://
research.yahoo.com/) under the following ID:L6. Yahoo! Answers Comprehensive
Questions and Answers (version 1.0)
2 http://dbpedia.org, Aug 4 2010
3 http://www.fao.org/agrovoc/, Aug 4 2010
4 http://www.illc.uva.nl/EuroWordNet/, Aug 4 2010
exploit additional information from other Linked Open Data sources, e.g. the
resource representing Germany from Figure 1 is linked through the typed link
owl : sameAs [
document 1: Bulgaria's best World Cup performance was in the 1994 World Cup in the United States, where they beat defending champions Germany to reach the semi-‐ finals. document 2: Deutschland ist ein föderalis'scher Staat in MiOeleuropa. Deutschland ist Gründungsmitglied der Europäischen Union und mit knapp 82 Millionen Einwohnern deren bevölkerungsreichstes Land. bag of resouces 1: db: Bulgaria db: FIFA_World_Cup db: United_States db: Germany bag of resouces 2: db: Germany db: Sovereign_state db: Central_Europe db: Ci'zen The Yahoo! Answers data corpus contains discussion threads consisting of an initial question and subsequent answers. The problem of expert search in this context is to nd users, who are likely able to answer a given question q, i.e. a topic, based on the threads in the data corpus.
We apply mixture language models. Potential experts are ranked according to the probability that the expert ex 2 E can answer the given question q 2 Q, i.e. P (exjq). A question q is modeled as a Bag-of-Resources: q = fr1; :::; rng.
P (exjq) / P (ex) P (qjex) = P (ex) (1) n Y P (rijex) i=1
We apply Bayes's theorem and assume P (q) and the prior P (ex) to be equal to 1. The probability P (rijex) is approximated as a weighted sum of several features f and smoothed by information over the entire corpus C.
P (rijex) = X( f Pf (rijex)) + C PC (ri) f s:t: X f f + C = 1 5 http://data.nytimes.com/55761554936313344161, Aug 4 2010 6 http://sws.geonames.org/2921044/about.rdf, Aug 4 2010
One answer per thread is marked by the questioner or by votes of other users as the best answer to the question. The user, who gave the best answer, is identi able by its ID. All other answers do not have a user ID. We exploit this setting by building two di erent models. These models are illustrated in Figure 4 and explained below.
aBest expert ID q a a* a a aAl
Best-Answer Model This model takes the question q and the best answer a together as abest and relates abest to the expert who gave the best answer, as illustrated in Figuren 4. The idea behind this model is that the user obviously understood the question, because he was able to give the best answer. Therefore he holds expertise about the covered resources. Formally, the model is de ned as follows, where f req(r; a) is the frequency of resource r in a.
Pbest(rjex) = X P (rjabest) abest
P (exjabest) P (abest)
P (ex) with
P (rjabest) = Pr2abest f req(r; abest)
f req(r; abest) P (abest) =
; P (ex) = P (exjabest) = 1; i ex author of abest, 0 otherwise 1 1 jQj jEj All-other-Answers Model This model relates all answers aall, except the best answer, to the expert, who gave the best answer. The assumption behind this model is that an expert, who gave the best answer, might also say that other answers are not correct. Therefore, we assume that the expert has expertise about the resources covered by these answers as well, at least to some extent. Formally, the model is de ned analogously to the previous one:
Pall(rjex) =
X P (rjaall) aall
P (exjaall) P (aall)
P (ex) 3.2
The mixture model presented in the previous section allows to balance the inuence of each model through the corresponding parameter f . In our case, we need to determine best, the weight for the Best-Answer-Model and all, the weight for the All-other-Answers-Model. The smoothing parameter C remains xed at C = 0:1. Hence, best + all =! 0:9 must hold.
In order to examine the e ect of di erent parameter con gurations on the performance of the retrieval, we used the 60 topics provided by workshop along with the given a priori relevance information. The a priori relevance information is directly taken from the data set, i.e. each topic has exactly one relevant expert, namely the one, who wrote the best answer for this topic. This setting is not optimal, since these questions are part of the data corpus itself and not distinct from it. Furthermore, it can be assumed that there are more than one relevant expert per question and that judging the performance by the occurrence of just one expert in the result set will deviate from the actual result. However, it allows at least to roughly estimate a parameter con guration. We used the Mean Average Precision (MAP) to measure the performance.
Since the 60 topics are part of the Best-Answer-Model a correlation between best and the M AP can be assumed in this setting. The M AP was measured in steps of 0:05 from best = 0, i.e. the performance without the Best-AnswerModel, to best = 0:9, the performance of the Best-Answer-Model alone. The observed MAP for the di erent parameters is shown in the left plot of Figure 5. As assumed, a correlation between best and the M AP can be observed. Remarkably, the MAP decreases for best = 0:9, despite the assumed correlation. This suggests that information is lost, if the All-Answers-Model is not involved and as a consequence, the optimal parameter con guration can not be the maximal observed MAP. We used least-square curve tting to approximate the observed values, i.e. the red line in Figure 5. The maximum of the tted curve is best = 0:66. Comparing the estimated values with the actual MAP computed ex post with the entire assessments shows that the maximum is even lower at about best = 0:53, see right plot of Figure 5. 4
We submitted three runs with di erent con gurations, see Table 2 for an overview of the results. Some of the 60 topics are very short and many are written in rather colloquial language and grammar or use abbreviations, e.g. "Why 0 . 00.0 0.2 0.4 λbest 0.6 0.8 do women get PMS?" or "hab es runtergeladen wie kann ich bei msn chatten?", which caused problems for the Wikipedia Miner to identify resources. For 11 topics the Wikipedia Miner did not identify any resources. In these cases, we extracted the resources manually, e.g. the resources db:Woman and db:Premenstrual syndrome for the rst question mention before and the resources db:MSN and db:Online chat for the latter. We did this for run1 and run3 and left the topics untouched for run2, in order to see how the approach performs without any manual intervention.
Strict Lenient Parameters Run Id P@10 MRR P@10 MRR best all C run3 0.49 (+157%) 0.76 (+90%) 0.87 (+123%) 0.93 (+48%) 0.7 0.2 0.1 run1 0.48 (+153%) 0.77 (+93%) 0.86 (+121%) 0.94 (+49%) 0.6 0.3 0.1 run2 0.35 (+84%) 0.65 (+63%) 0.61 (+56%) 0.74 (+17%) 0.6 0.3 0.1 BM25 + Z-Score 0.19 0.40 0.39 0.63
Table 2 shows the results for the top 10 retrieved experts. Precision at
cuto level 10 (P@10) and Mean Reciprocal Rank (MRR) are used as evaluation
measures. Precision/Recall curves for each run are presented in Figure 6 using
strict and lenient assessments [
Beside retrieving relevant experts for a topic, one main aim of our approach was to cross the language barrier and nd experts regardless of their language. Figure 7 visualizes the language distribution of the retrieved experts for each topic language for run1. In order to facilitate the comparison, the distribution of threads and experts in the data set is displayed on the right. One can see that indeed experts in all four languages were retrieved in most cases. Further, the domination of english speaking experts is due to the proportions in the data set and in addition due to the larger, underlying resource space, as illustrated in Figure 2. However, a bias towards the language of the topic is also observable, because not all resource have labels in all other languages as discussed in Section 2.
10 ● 8 ● 6 4 ● 2 ● 0 EN ES FR DE
English Topics 0 1 8 6 4 2 ● 0 EN ES FR DE
Spanish Topics 01 ● 8 6 4 ● 2 ● 0 E●N ES FR DE
French Topics 0 1 8 64 ● ●● 2 ● ● 0 EN ES FR DE
German Topics
Our approach uses Language Models, which have been studied by [
Using concepts instead of terms was studied by [
Not just indexing the terms of a document, but the idea of indexing what a
document is about, i.e. topic indexing, was introduced by [
A di erent approach to multilingual IR was introduced by [
We presented an approach for the Expert Search challenge of the CriES
Workshop at CLEF 2010 using Linked Open Data resources, i.e. URIs, as interlingual
document representations. We used Wikipedia as the corpus of resources, but the
approach is not limit to the usage of Wikipedia. Resources are extracted from the
documents using the Wikipedia Miner Toolkit [
Our approach yielded solid results by exceeding the standard BM25 + ZScore baseline from 17% to 157% regarding Mean Reciprocal Rank and Precision at 10. Another advantage of our approach is that not the entire documents need to be indexed, but just a summary consisting of several URIs, which decreases the index size.
In future, we plan to use more features of Linked Open Data for IR. In particular, exploiting the links between resources to leverage the interconnection. 7
We thank Philipp Sorg for the helpful discussions and his valuable feedback. We also thank David Milne for developing the Wikipedia Miner and making it available as open source. Research reported in this paper was supported by the German Federal Ministry of Education and Research (BMBF) under the iGreen project (grant 01IA08005K).