This paper describes the participation of Dublin City University in the CriES (Cross-Lingual Expert Search) pilot challenge. To realize expert search, we combine traditional information retrieval (IR) using the BM25 model with reranking of results using the HITS algorithm. The experiments were performed on two indexes, one containing all questions and one containing all answers. Two runs were submitted. The rst one contains the combination of results from IR on the questions with authority values from HITS; the second contains the reranked results from IR on answers with authority values. To investigate the impact of multilinguality, additional experiments were conducted on the English topic subset and on all topics translated into English with Google Translate. The overall performance is moderate and leaves much room for improvement. However, reranking results with authority values from HITS typically improved results and more than doubled the number of relevant and retrieved results and precision at 10 documents in many experiments.
The CriES pilot challenge [
For the CriES expert search experiments described in this paper, di erent approaches to nd experts likely to answer a question were investigated: 1. Finding experts by matching the current question with previously given answers.
This corresponds to a standard information retrieval approach on answer documents. 2. Finding experts by matching the current question with questions which have previously been answered. This approach is typically employed in FAQ (frequently asked questions) search and corresponds to IR on questions. 3. + 4. Reranking the results of the two former approaches by interpreting HITS authority values of question and answer documents as the level of expertise.
The rest of this paper is organized as follows: Section 2 introduces related work. Section 3 describes the theoretical background and the system setup for the CriES experiments. Section 4 presents the experimental setup and results, followed by an analysis and discussion of results in Section 5 and the paper concludes with an outlook on future work in Section 6. 2
Expert search on question answer (Q/A) pairs is a relatively new research area
which is related to search in FAQs, social network analysis, and question
answering (see, for example [
FAQ Search
Burke, et al. [
Wu, et al. [
Jijkoun and de Rijke [
Chiu, et al. [
Several retrieval experiments described in this paper are also based on nding
experts who answered similar questions by indexing all questions.
2.2
Balog, Azzopardi, and de Rijke [
MacDonald and Ounis [
Yang, et al. [
Similar to extracting experts from retrieved documents, some retrieval experiments described in this paper rely on retrieving answers to a given question and extracting their experts (authors). 2.3
Link Analysis
The HITS (Hyperlink-Induced Topic Search) algorithm is a link analysis
algorithm for rating web pages [
For our CriES experiments, we selected the HITS algorithm, as its values are computed at query time and on a smaller document base. Thus, the HITS algorithm does not require re-indexing the document collection to recompute scores after modi cations or extensions to the algorithm. HITS scores also highly correlate with in/outdegree of linked nodes, which intuitively correspond to the level of expertise: the more information a person produces on a given topic, the higher her/his level of expertise should be.
The experiments for the CriES pilot challenge can be based on two di erent types of data which were provided by the organizers: a collection of Q/A pairs and a linked graph model extracted from this collection. Furthermore, the CriES challenge is unique in that it aims at expert nding in a multilingual setting, i.e. topics are provided in di erent languages.
3.1
Topic and Document Processing Interpreting individual questions and answers as documents, standard IR techniques can be applied for expert search. In our work, the Lucene toolkit3 was utilized to preprocess the topics and documents, and to index and search the document collection. Standard Lucene modules were employed to tokenize the questions and answers and to fold upper case characters to lower case. Stopword lists from Jacques Savoy's web page on multilingual IR resources4 were used to identify stopwords. Stemming of topics and documents was performed using the Snowball stemmer for the corresponding language provided in Lucene. For all retrieval experiments, only the topic elds for `title' and `description' were used to create IR queries for Lucene (TD). The elds `narrative' and `questioner' were omitted for query formulation. The `answerer' eld was used to form documents IDs. Figure 1 shows a sample topic.
The CriES question-answer set was preprocessed by us to generate two types of documents from the original CriES documents: answer documents (A) and question documents (Q). The rst type of document contains the `answerer' ID as a document ID and the text of his answer concatenated with the category of the question. This retrieval approach realizes standard IR by nding answers based on the replies the users have already generated. The second type of document contains the `answerer' ID as a document ID and the question text concatenated with all category labels from the original document. Thus, retrieval on these documents aims at nding experts by matching the input question with previous questions the answerer has replied to. In detail, documents for indexing were created as follows: answer documents were extracted from answers given (i.e. `bestanswer' ); question documents consist of the question text (i.e. `subject', `content' ). Both types of documents were concatenated with the category elds (i.e. `cat', `maincat', `subcat' ). In addition, the link graph consisting of nodes representing experts and links between questioner and answerer (provided as part of the CriES challenge) was employed as input for the HITS algorithm. 3.2
The Information Retrieval System
Support for the BM25 retrieval model [
K + tf k3 + qtf (1)
<topic lang="en"> <identifier>3938625</identifier> <title>What is the origin of "foobar"?</title> <description>I want to know the meaning of the
word and how to explain to my friends.</description>
<narrative/>
<category>Programming & Design</category>
<questioner>u1061966</questioner>
<answerer>u25724</answerer>
</topic>
where Q is the query, containing terms t and w(1) is the RSJ (Robertson /
Sparck-Jones) weight of t in Q [
(d + 0:5)=(D d + 0:5)=(N n d + 0:5)
D + d + 0:5)
(2)
where k1, k3, and b are model parameters. The default parameters for the BM25
model used are b = 0:75, k1 = 1:2, and k3 = 7. N is the number of documents
in the collection and D is the number of documents known or presumed to be
relevant for the current topic. For the experiments described in this paper, D
was set to 0, i.e. no blind relevance feedback was employed, because the number
of experts and precision of our initial retrieval were presumed to be very low. n is
the document frequency for the term and d is the number of relevant documents
containing the term. tf is the frequency of the term within a document; qtf is
the frequency of the term in the topic. K = k1((1 b) + b doclen=avg doclen)
doclen and avg doclen are the document length and average document length,
respectively. The BM25 retrieval model has been employed for many years in
evaluation campaigns such as TREC [
Reranking with HITS
The HITS algorithm is a link analysis algorithm for rating web pages [
To apply HITS for expert search, the expert graph is viewed as a linked graph of experts (corresponding to web pages) with directed connections (links) from questioners to answerers if the answerer provided an answer to a question.
For the experiments described in this paper, the hub and authority values for an item are calculated with the following algorithmic steps, iterating steps (2)-(4) for k times (see also Figure 2): (1) Initialize: Set the hub and authority value for each item (node) to 1. (2) Update authority values: Update the authority value of each item to be equal to the sum of the hub values of each item that points to it. That is, items with a high authority value are linked to by items that are recognized as informational hubs. (3) Update hub values: Update the hub value of each item to be equal to the sum of the authority values of each item that it points to. That is, items with a high hub value link to items that can be considered to be authorities on the subject. (4) Normalize values: Normalize the authority and hub values by dividing each authority value by the sum of the squares of all authority values, and dividing each hub value by the sum of the squares of all hub values.
Applied to expert search, hubs can be interpreted as persons interested in a topic, and authorities can be seen as experts on a topic. 4
The dataset consists of Q/A pairs which are maintained and veri ed by experts. In contrast to IR, results represent experts which may be associated with di erent levels of expertise; in comparison with FAQ nding, expert search focuses on looking for people most capable of providing an answer. In the simplest case, people have already provided that answer to the same or to similar questions. A graph model was provided as part of the CriES data, which consists of a directed graph representation where nodes denote topics, incoming links are questions and outgoing links represent answers.
The answer documents (A) and question documents (Q) generated from the CriES data were indexed separately. The following experimental settings were varied: { index: retrieval on answer documents (A) or question documents (Q) { language: no topic translation; topic translation (using Google Translate)5;
English topics only. { retrieval method: using standard IR (BM25); combining BM25 with HITS authority values from top 50/100 results (HITS 50/100).
The BM25 retrieval model was used with default parameters (b = 0:75, k1 = 1:2, and k3 = 7), retrieving the top 100 results for each topic. The HITS algorithm was applied on the top 50 or top 100 results retrieved by standard retrieval with the BM25 model. The HITS algorithm was run for 50 iterations (k = 50). The experimental setting were chosen empirically after initial retrieval experiments on CriES test data. Table 1 shows results for o cial and additional expert search experiments on the CriES data. The submitted runs were obtained by retrieving 100 results via IR and reranking these results with the HITS authority value. The top ten results for each topic were extracted for submission. 5
As described in [
-
0.19 50 0.0106 0.0833 56 0.0123 0.0933 146 0.0584 0.2433 38 0.0102 0.0633 46 0.0113 0.0767 128 0.0441 0.2133 12 0.0107 0.0800 18 0.0139 0.1200 10 0.0070 0.0667 9 0.0046 0.0600 41 0.0067 0.0683 101 0.0409 0.1683 23 0.0039 0.0383 75 0.0290 0.1250 -
0.39
112 0.0143 0.1867
241 0.0426 0.4017
206 0.0393 0.3433
109 0.0134 0.1817
251 0.0425 0.4183
183 0.0328 0.3050
27 0.0133 0.1800
76 0.0569 0.5067
31 0.0164 0.2067
74 0.0509 0.4933
71 0.0063 0.1183
141 0.0266 0.2350
61 0.0067 0.1017
119 0.0213 0.1983
of relevant judgments were provided by the organizers, corresponding to strict
evaluation (experts likely able to answer are considered relevant) and lenient
evaluation (experts likely able to answer and experts which may be able to
answer are relevant). A comparison of our experimental results to the provided
baseline reveals that our best experimental results are slightly higher in terms of
P@10 (0.24 vs. 0.19 for strict, 0.42 vs. 0.39 for lenient relevance judgments). The
results do not consistently outperform the BM25 baseline, and show much lower
performance than the best results reported by the organizers [
The topic set contains 60 topics. The lenient relevance assessment contains 3602 relevant entries (60.03 relevant items on average per topic), the strict assessment 1736 entries (28.93 relevant items average). Table 2 shows the distribution of languages in topics, questions, and answers. As the topics are equally distributed among four languages (15 topics per language), a more detailed analysis of the language of questions and answers was performed. This analysis shows that languages of Q/A pairs are not equally distributed among these languages, i.e. there is a bias towards English (91.3%). The question index and answer index both contain 780,133 documents with the language distribution shown in Table 2.6 While the topics are equally distributed between the four languages German (DE), English (EN), Spanish (ES), and French (FR), the majority of question and answer documents are in English.
As an additional experiment, the experimental results were calculated for the English topics only (the rst 15 topics). However, there seems to be little bias towards English in relevant items compared to all items, because MAP slightly decreases from 0.1867 to 0.1800 and the number of relevant items is about a quarter of relevant items for all topics (27 vs. 112).
A comparison of retrieval on question documents and answer documents shows that results (i.e. rel ret, MAP, and P@10) are slightly higher for IR on answer documents. A possible explanation is that answers are typically longer than questions and provide more terms to match. Thus, a lexical mismatch may be less likely. The reranking with HITS authority typically shows considerable improvement in the number of relevant and retrieved results (rel ret), mean average precision (MAP), and P@10, more than doubling P@10 for the lenient evaluation.
There are several possible explanations of the moderate performance in
comparison to the best CriES runs submitted by other participants (see, for example,
the overview in [
Multilinguality: All 60 topics (corresponding to queries) are equally
distributed between German, English, Spanish, and French. Assuming that the
questions and answers in the CriES data was similarly distributed, no topic or
document translation was performed for our o cial experiments and all
documents in their original language were organized in a single index. However, most
answers in the CriES data are written in English. The baseline experiment in [
Expert model: Experts are represented by a set of individual questions or answers. An aggregated model, i.e. combining all questions and answers into a single representation (e.g. document or weighted term vector) has not been investigated. The main reason for this is that a single category (unrelated to the current topic) may dominate in all contributions of a single user.
External resources: No additional external resources have been used for
the experiments described in this paper. Standard approaches for FAQ search
typically utilize resources such as WordNet [
Link analysis: Traditional approaches for link analysis of the link graph provided in the CriES data have not been used. Instead, the HITS algorithm, which is typically applied for reranking web pages has been employed for reranking results. Interestingly, reranking initial retrieval results with HITS authority values improved performance considerably in most cases, increasing MAP and P@10 and yielding more than double the number of relevant results compared to the corresponding BM25 experiment, even for poor initial results. This result was unexpected, because the initial precision is very low for the result set retrieved with the BM25 model. Standard query expansion techniques such as blind relevance feedback also aim at improving at performance by reranking documents in a second retrieval phase, but build on the assumption that top-ranked documents in an initial retrieval phase are relevant (i.e. the initial precision is already high). If initial results have low precision, standard query expansion techniques will typically add noise instead of useful terms. Reranking results with HITS authority values improves performance despite low initial precision. 6
The experiments on the CriES data show that traditional IR methods alone (i.e. the BM25 retrieval model) may not be suitable for this kind of task and social network or link analysis may be more successful. Reranking results with HITS authority values seems to improve performance even when the initial precision is low. The multilingual aspect introduced in the CriES challenge seems arti cial because most contributions in the data are in English. However, experiments on the English topic subset did not show a bias.
Future work will include adding knowledge from external resources such as Wikipedia and expanding the use of categories and other metadata provided in the CriES data. Also, reranking with HITS authority scores for ad-hoc IR will be investigated.
This material is based upon works supported by the Science Foundation Ireland under Grant No. Grant 07/CE/I1142.