The goal of expert- nding is to retrieve a ranked list of people as a response to a user query. Some models that proved to be very successful used the idea of association discovery in a window of text rather than the whole document. So far, all these studies only considered xed window sizes. We propose an adaptive window-size approach for expert- nding. For this work we use some of the document attributes, such as document length, average sentence length, and number of candidates, to adjust the window size for the document. The experimental results indicate that taking document features into consideration when determining the window size, does have an e ect on the retrieval outcome. The results shows an improvement over a range of baseline approaches.
H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval
With the massive and ever-growing amount of electronic data, search engines have become crucial for any organisation that wants to help its employees with their day-to-day information needs. Traditionally, search engines, or information retrieval systems in general, function by returning a list of documents for the user's query, although the user's information need may not necessarily be in the form of documents. In fact, users more often search for speci c things Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
DIR 2013, April 26, 2013, Delft, The Netherlands.
(people, organisations, or products) [
One special type of entity search is expert- nding. In
expert- nding we are concerned with identifying experts who
possess the relevant skills and knowledge on a given topic
[
Given a search topic, state-of-the-art expert- nding
systems typically measure the knowledge of candidates from the
textual content of top ranking documents, which are used
to derive associations between candidates and search topics
based on co-occurrences [
In this paper, we consider the idea of an adaptive window size, where the size of the window is a function of various document features. We argue that each document has distinct features that di er from other documents in the collection. Using these features to set the window size could improve the overall ranking function. There are many document features that could be examined. We focus on three of them: document length, average sentence length, and candidate frequency (i.e. the number of candidates that appear in a document). To the best of our knowledge, no existing work has dealt with using the document features to determine the optimal window size for the proximity function.
It is important to note that the adaptive window size approach could be applied to any proximity search, in particular for an entity-oriented search, a generalisation of expert search. We carried out the study in the expert search domain due to the availability of an expert-search benchmark.
The main research question considered here is whether an adaptive window size leads to improvements over xed window size methods. 2.
As described above, the input for any expert- nding system is the user's submitted query. This query then could be normalised and di erent query expansion techniques could be applied to it. Next, the query is passed to an underlying search engine; in this work, we used Lucene2 as our search engine, with a BM25 ranking function. For each query, only the top 100 documents returned by the search engine were considered. We used these documents to rank the candidates based on two measures. First, based on their frequency occurrence, and second, based on the proximity between the candidate's evidence and the query occurrences in the document (Figure 1). c{a1 c{a2
TF.IDF ca1 score TF.IDF ca2 score kernels (Gaussian, Triangle, or Cosine) ca1 ca1 ca1 kernel1 kernel2 kernel3 kernels (Gaussian, Triangle, or Cosine) ca2 ca2 ca2 kernel1 kernel2 kernel3
Proximity approaches have been successfully used in
different applications, which enhance the quality of the
retrieval systems. In particular, the work of Petkova & Croft,
[
In this work, the window size for the proximity function will be determined for each document based on the following features.
(1) Document Length: According to Miao et. al. [
W indow Size =
1 (log( 3 DocLength + CanF req + AvgSentSize ) 2 1 3) (1) is a variable that allows to scale the window size. We explore a wide range of values for , (see below). The weighting factors, which determine each feature's contribution in the equation, have been set empirically, where Pi=1 i = 1. The TREC2005 data includes ten training topics3. We used these topics to train our variables, thus having a clear distinction between test and training data.
Although the proposed model used the three features, we will also report experiments for each feature individually.
After establishing the size of the window, it is applied to every full match for the query found in the document. Then, the candidate evidence neighbouring this term is extracted; each one within the window will be given a weight depending on its distance from the query.
The advantage of this window is that it provides a graded proximity boost. Candidates with an index close to the query terms will receive the highest boost. As the candidate indexes drift further and further away, the boost will gradually decrease until it reaches the end of the window. A document can contain multiple query terms. In this case, we place a window at each occurrence. If, for example, a document has two query terms, two windows are placed, but centred at di erent locations. If the two windows are close to each other, both windows could boost candidates that appear between them.
Three di erent kernel functions were used to calculate the
weight: Gaussian, Triangle, and Cosine [
To evaluate our approach, we used the document collection of the W3C corpus and the test sets of the 2005 TREC Enterprise track. The W3C corpus includes a prede ned 3http://trec.nist.gov/data/enterprise/05/ent05. expert.trainingtopics
baseline
baseline
baseline list of 1092 candidates, 331,037 documents, and 50 topics, each of which is provided with a relevance judgement. We selected this collection in order to test our method on a simple and most basic form of expert- nding4.
We removed stopwords and HTML markup, and treated all documents as plain text. For evaluation, we applied a range of standard IR measures, but in our discussion we focus on Mean Average Precision (MAP).
In this work, we use the two-stage model for the initial candidate ranking by calculating the probability of the candidate given the query, P (cajq), as follows:
P (cajq) = X P (djq) P (cajd) d (2) where P (djq) is the document relevance to the query, which is calculated by the underlying search engine, and P (cajd) is the candidate's probability given the document. In our baseline, P (cajd) is calculated using the full document without a proximity function. Whereas in all other experiments, we apply Equation 1 to nd the optimal window size for the current document. The proximity functions will only consider the occurrences within this window of text.
Our rst baseline is a frequency-based approach. In this baseline, a T F IDF weighting scheme is used in order to obtain the candidate's importance in a particular document, 4Other forms of expert nding include nding similar experts and nding all expertise for a given candidate.
CanFreq 250 0.2806 0.3452 0.4147 0.4199 0.3599 0.3100
AvgSentSize 600 0.2798 0.3269 0.4199 0.4189 0.3499 0.3100 xed window sizes Gaussian Triangle Cosine 100 200 300 400 500 600 700 800 900 1000 Window Sizes ) P A0.32 We used our adaptive window-size method (Equation 1), with the three proximity functions at di erent values ranging from 0 to 1000 with an increment of 50. We only report the results for values between 350 and 650. The results below 350 and above 650 drop gradually, so they were not reported. Furthermore, we calculate a baseline for each proximity function. In this baseline, we set the window size to be equal to the document length. Our results are summarised in Table 1.
The top MAP of 0.3454 is achieved using a Gaussian proximity function with an adaptive window size where = 500.5 We found that the di erence between our best run and the baseline is statistically signi cant (using paired t-tests on average precision values at p < 0:05). Moreover, we found that the di erences between the best run for each proximity function and its baseline were also statistically signi cant.
For comparison, we used a range of xed window sizes. We calculated MAP for xed windows in a range from 100 to 1000 in increments of 50. We repeated the experiments using the three proximity functions (Gaussian is shown to be signi cantly better than the other two functions, with a top result of MAP=0.27 at a window size of 200); see Figure 2. 5.
We introduced the idea of an adaptive window size for expert- nding. Thus, for the proximity function, the size of the window will be set based on current document features rather than a xed window for all documents in the collection. Adopting this method results in signi cant improvements over standard metrics. This is true for all proximity functions used in this study (i.e. Gaussian, Triangle, and Cosine). We found that the best results were achieved using a Gaussian function. As for future work, we plan to investigate the e ectiveness of using other document features such as the readability index for determining the optimal window size. We also plan to test the adaptive window size method on other expert- nding collections and also on other TREC benchmarks.