Many tools offer to search and filter the vast data sources available on the Web. In particular, there is a multitude of platforms directed to the scientific community. From the simple search engine for publications to the social network for researchers, all consume and produce valuable data for searching scientific content of interest. Expert finding is one the the most challenging problem that finds application in both academia and the industry. To tackle this challenge, recent advances in document network embedding (DNE) has the potential to inspire new unsupervised models that can deal with the heterogeneous network of documents of the scientific literature. However, the design of such efficient algorithms heavily depends on the development of strong evaluation frameworks.
In this paper, we propose a methodology and provide 4 datasets that extend the limited scope of expertise retrieval evaluation frameworks. Furthermore, we provide experiment results computed with unsupervised methods and we extend document network embedding algorithms to this specific task.
Our contributions are the following:
we provide 4 datasets for expert finding extracted from a scientific publication network and three question & answer (Q&A) websites and make them publicly available 1 ; we describe an evaluation methodology based on the ranking of expert candidates given a set of labeled document queries; we report experiment results that give some insights on this expert finding task; we explore and analyze the use of state-of-the-art document network embedding algorithms for expert finding and we show that further research is needed to bridge the gap between DNE methods and expert finding.
The rest of the paper is organized as follows. In Section 2, we survey related works. We detail in Section 3 our evaluation methodology, the datasets we extracted, the evaluation measures and the algorithms we use. In Section 4 we show and analyze the results of our experiments. Finally, in Section 5, we discuss our findings and provide future directions.
In this section, we first present a formal definition for expert finding. Then we present algorithms of the literature that address expert finding. Finally, we describe recent methods for document network embedding that have the potential to deal with this particular task.
The concept of expert finding can cover a large range of tasks. The main principle behind expertise retrieval is the search for candidates given a query. To match these two, an algorithm will be provided with some data to link the output space, a ranking of candidates, with the input space, which is often a textual content. However, many different types of data can be considered to address this challenge. To fairly compare algorithms, we choose a fixed structure for the data which reflects common use cases. Furthermore, if supervised methods benefit from labeled fields of expertise associated with the candidates, they are beyond the scope of this paper which focuses on unsupervised methods only. Our goal is to compare methods that do not require sometimes costly annotations.
Early works in expert search [8] usually consider a small set of topical queries. The direct namings of these topics are used to retrieve a list of candidates by leveraging a collection of documents they published (e.g., emails, scientific papers). This type of evaluation is used across several public datasets [15,16,26].
More recently, the concept of expert finding has been merged into the wider concept of entity retrieval [1]. As more and more complex data are produced on the Web, expert finding becomes a particular application of entity search. At the same time, Q&A websites such as Stack Overflow 2 generate and make publicly accessible a big amount of questions with expert answers, collaboratively curated by their users. Several works address the search for experts in such websites [20,28]. Often, the task consists in either finding the exact list of users who answered a specific question or ranking the answers according to the user votes. In the first case, the task involves considering the evolution of the users across time and, in the second case, the task involves understanding the intrinsic quality of a written answer. Nevertheless, [25] reviews several models for expert finding in Q&A websites. Their experiments show that matrix factorization-based methods perform better than tree based and ranking based methods.
In this paper, we adopt the document-query methodology recently proposed in [3]. The expert search is performed given a set of queries that are particular textual instances of some expert topics (or fields of expertise). Given a query, an algorithm should rank first the candidates that are associated to the same fields of expertise. We provide 4 datasets for which we annotated experts and document queries. Each dataset consists in candidates and documents linked by authorship relations (candidate-document e.g. authorship) and by response relations (document-document e.g citation or answer). A query is therefore one of the documents (e.g. a scientific paper or a question) for which we aim to retrieve some experts of the topics depicted in it. This configuration reflects many real case scenarios such as (1) the automatic search for scientific reviewers, (2) the recommendation of expert users in Q&A websites or even (3) the retrieval of interesting profiles for job offers.
Numerous works have addressed automatic expertise retrieval. We describe here the main approaches and some interesting recent methods. P@noptic Expert [7] creates meta-documents for a candidate by concatenating the contents of all documents she produced. In this manner, ranking the candidates given a query becomes a similarity search between the query representation and the meta-documents representations. A voting model [14] computes the similarities between the query and the documents. The algorithm then aggregates these scores at the candidate level by using a fusion technique such as the reciprocal rank [27]. A propagation model [21] takes advantage of the links between candidates and documents to propagate the similarities between the query and the documents. Using random walks with restart [17], the iterative propagation of the scores converges in a few steps to a stationary distribution over the candidates. WISER [6] models each candidate as a small, weighted, sub-graph of the Wikipedia Knowledge Graph. Information derived from these graphs and traditional document retrieval techniques are combined to identify experts w.r.t a query. Note that methods leveraging external data are out of the scope of our benchmark. LT Expertfinder is an evaluation framework for expert finding [11] based on an interactive tool. It integrates various existing algorithms (such as [1]) in a user-friendly way. The underlying corpus used by this tool is the ACL Anthology Network. However, it does not include a wellestablished ground truth to assess who are the experts. Indeed, the evaluation is purely done in an online manner since the user has to evaluate the degree of expertise based on several features, such as author's citations, h-index, keywords, etc. Recent works [9,23] propose ad hoc embedding techniques, whereas, in this work, we're interested in measuring the performance of conventional network embedding techniques.
Network embedding [12,19] provides an efficient approach to represent nodes in a low dimensional vector space, suitable for solving various machine learning tasks. Recent techniques extend NE for document networks. Text-Associated DeepWalk (TADW) [24] extends DeepWalk to deal with textual attributes. Yang et al. prove, following the work in [13], that Skip-Gram with hierarchical softmax can be equivalently formulated as a matrix factorization problem. TADW then consists in constraining the factorization problem with a pre-computed representation of the documents by using Latent Semantic Analysis (LSA) [10]. Graph2Gauss (G2G) [2] is an approach that embeds each node as a Gaussian distribution instead of a vector. The algorithm is trained by passing node attributes through a non-linear transformation via a deep neural network (encoder). GVNR-t [4] is a matrix factorization approach for document network embedding, inspired by GloVe [18], that simultaneously learns word, node and document representations by optimizing a least-square objective over a co-occurrence matrix of the nodes constructed by truncated random walks. IDNE [5] introduces a Topic-Word Attention mechanism, trained from the connections of a document network, to represent documents as mixtures of topics. DNE algorithms do not directly apply to expert finding data since they are not designed to handle multiple types of nodes, in particular candidate nodes. In this paper, we show (1) two methods to extend their applicability to the task of expert finding and (2) the impact of their representations when they are used as document representations for traditional expert finding algorithms.
We present in this section the evaluation methodology that we follow to access the performances of several algorithms for expert finding. We first describe the task we seek to solve, then we describe the datasets that we extracted and explain how we annotated them in order to access the quality of the algorithms' outputs. Finally, we detail the models used in our experiments.
Expert finding is a complex task that can be formalized in multiple ways. Early works define this task as a ranking problem given several topic-queries where the naming of these topics are directly used as queries to retrieve the expert candidates. However, in many real world applications, a user is asked to provide a specific and detailed query. In a Q&A website for instance, a user usually exposes the problem she faces in full detail and does not necessarily know the exact naming of the fields of expertise needed to solve her problem. Furthermore, querying an algorithm with a small set of topicqueries can lead to poor evaluation measures due to the usually small number of fields of expertise associated with the dataset. For this reasons, we follow the document-query evaluation methodology proposed in [3] by processing 4 datasets for which a set of document-queries is manually annotated.
The expert finding task in this paper is a ranking problem. Given a document labeled with a ground truth set of fields of expertise, an algorithm is queried to rank a set of candidates, among which a subset of experts are associated with the same set of labels. The data provided to the algorithms consists in a corpus of n d documents D, n c candidates C, a network of authorship with adjacency matrix A dc ∈ N n d ×nc and a network of documents with adjacency matrix A dd ∈ N n d ×n d . Figure 1 shows an hypothetical dataset used in this paper. The ranking is performed in an unsupervised setting, that is, no ground truth labels of expertise are given to the algorithms. The set of labeled documents (the queries) can be smaller than n d and the set of labeled candidates (experts) can be smaller than n c (i.e not all documents and candidates are labeled).
Fig. 1: Hypothetical example of an expert finding dataset we use in this paper. 5 candidates are authors of 6 documents. The 6 documents are connected to each other by citation in a scientific corpus, or by answer in a same post in a Q&A website. Among the candidates, 3 are known to be experts in stars and/or in circles. 4 documents are associated to these 2 fields of expertise as well. In our evaluation methodology, we query an algorithm with these 4 documents and expect a ranking of candidates that will match each document's fields of expertise. As an example, a perfect algorithm might generate the rankings
To evaluate the candidate scores provided by the algorithms, we compare the resulting rankings with the ground truth fields of expertise. If a document is associated with three different labels, we expect the algorithm to rank first all experts associated to at least one of these labels. We report the area under the ROC curve (AUC), the precision at 10 (P@10) and the average precision (AP) and we compute their standard deviation along the queries. That is, we evaluate the robustness of the algorithms against the variety of document-queries.
We consider 4 datasets. The first one is an extract of DBLP [22] in which a list 199 experts in 7 fields are annotated [26] by human judgments 3 . Our dataset only considers the annotated experts and the other candidates that are close in the co-authorship network which explains the relatively small size of our network compared to the original one. In addition to the expert annotations, our evaluation framework requires document annotations since we adopt the document-query methodology for expertise retrieval. We asked two PhD students in computer science to associate independently 20 randomly drawn documents per field of expertise (140 in total). Then, only the labels on which the two annotators agreed were kept, leaving 114 annotated papers. The mean Cohen's kappa coefficient across the labels is 0.718. An advantage of our methodology is that we can evaluate the algorithms on more queries (114 documents) than the traditional method (7 labels). This allows us to assess the robustness of the algorithms by computing the standard deviations of the ranking metrics along all queries. However, one might suggest that these 7 labels do not reflect a representative set of expertise as there are too broad. For this reason, we seek for a wider granularity of expertise by the use of well-know question & answer website.
If scientific publication networks are easy to find on the Web, scientific expertise annotations are rarely available for both authors and publications. We use data downloaded in June 2019 from Stack Exchange4 to create datasets for expert finding collected from three communities closely related to research. Academia5 is dedicated to academics and higher education. Mathoverflow6 gathers professional mathematicians and is widely used by researchers. Stats7 (also known as Cross Validated) addresses statistics, machine learning and data mining issues. For each dataset, we first keep questions with at least 10 user votes that have at least one answer with 10 user votes or more. We build the networks by linking questions with their answers and by linking answers with the users who published them. The field of expertise are the tags associated with the questions. Only the tags that occur at least 50 times are kept. We annotate an expert with the tags of a question if her answer to that question received at least 10 votes. Note that the tags are first provided by the users who ask the questions but they are thereafter verified by experimented users.
The general properties of our 4 datasets are presented in Table 1. The annotations and the preprocessed datasets are made publicly available.
We run the experiments with 4 baseline algorithms and 4 document network embedding algorithms. The laters are adapted with two aggregation schemes in order to deal with the candidates since they are primarily designed for document network only. These aggregations are arbitrary and are voluntarily the most straightforward way to run DNE algorithms on bipartite networks of authors-documents. We further discuss these choices in section 4.
Baselines We run the experiments with the same models as in [3], using the tf-idf representations and the cosine similarity measure. Also, we add a random model to have reference metrics:
-Random model: we randomly draw scores between 0 and 1 for each candidate; -P@noptic model [7]: we concatenate the textual content of each document associated to the candidates, use their tf-idf representations and compute the cosine similarity to produce the scores; -Voting model [14]: we use the reciprocal rank to aggregate the scores at the candidate level; -Propagation model [21]: we concatenate the two adjacency matrices A dc and A dd to construct a transition matrix between candidates and documents such that
. The initial scores are the cosine similarities between the tf-idf representations of the query and the documents. The scores are propagated iteratively until convergence with a restart probability of 0.5.
We also run the voting and propagation models using document representations produced by IDNE in place of the tf-idf vectors. The document network provided to IDNE has adjacency matrix
Extending DNE algorithms for expert finding DNE methods usually operate in networks of documents, with no candidate nodes. To apply them in the context of expert finding, we propose two straightforward approaches:
-pre-aggregation: as in the P@noptic model, meta-documents are generated by aggregating the documents produced by each candidates. Furthermore, an adjacency matrix of a meta-network between candidates and documents is constructed. We compute a candidate network as A c = A dc A dc and a document network as
. The candidate and document representations are then generated by treating this meta-network and the concatenation of the documents and meta-documents as an ordinary instance of document network. From this meta-network, we generate representations with the DNE algorithms. The scores of the candidates are generated by cosine similarity between the representation of the document-query and the representations of the candidates; -post-aggregation: in this setting, we first train the DNE algorithm on the network of documents defined by
Once the representations are generated for all documents, a representation for a candidate is computed by averaging the vectors of all documents associated to her. The scores are then computed by cosine similarity.
We run the experiments with 4 document network embedding algorithms, using the authors' implementations. For all methods, the dimension of the representations is set to 256:
-TADW [24]: we follow the original paper by using 20 iterations and a penalty term λ = 0.2; -GVNR-t [4]: we use γ = 10 random walks of length t = 40, a sliding window of size l = 5 and a threshold x min = 2 with 4 iterations; -Graph2gauss (G2G) [2]: we make sure the loss function converges before the maximum number of iterations; -IDNE [5]: we run all experiments with n t = 32 topic vectors with 5000 balanced mini-batches of 16 positive samples and 16 negative samples.
Tables 2 to 5 show the experiments results. In the following, we analyze the performances of the aggregation scheme against the baseline algorithms, we highlight the interesting results obtained when using the baselines with pre-computed document representations with a DNE algorithm and finally we make some observations on the differences between the datasets. Note that the implementation of TADW, provided by the authors, could not scale to Mathoverflow.
For all datasets, the propagation model performs generally better than the other algorithms, particularly in terms of precision. Both aggregation schemes yield to poor results and none of these two methods appear to be better than the other. GVNR-t is the best algorithm among the document network embedding models. We believe that, if DNE algorithms are well suited for document network representation learning, the gap between simple tasks such as node classification and link prediction and the task of expert finding is too big for our naive aggregation schemes to perform well. Especially, the network structure changes significantly between an homogeneous network and an heterogeneous network. Moreover, expert finding algorithms often benefit from information about the centrality of the candidates and documents. DNE algorithms do not particularly preserve this information neither do our aggregation schemes.
Since the baseline algorithms perform well, we study the possibility to apply them using a DNE algorithm for the representations of the documents. We only report the results with the representations computed with IDNE but we observe the same behaviors with other DNE models. First, these representations constantly improve the voting model, which achieves best results in terms of AUC on Stats and Mathoverflow. Then, the most surprising effect is the significant decrease of performance of the propagation model. If the precision for the first ranked candidates is not affected, the AUC score significantly drops for the three Q&A datasets. We believe that document network embeddings captures too long-range dependencies between the documents in the network, which are then subsequently exaggerated by the propagation procedure. Figure 2 shows the effect of the representations used with the propagation model on the ROC curve.
The results achieved by the algorithms on all three Stack Exchange datasets are consistent. However, they do not behave the same with DBLP. First, DNE methods get closer scores to the baselines on DBLP. In the Q&A datasets, the interactions are more isolated i.e. there are more users having fewer interactions. This difference of network properties might disadvantage DNE methods who are usually trained on scale-free networks whose degree distribution follows a power law. Moreover, the propagation method does not suffer with DBLP from the decrease of performance induced by the IDNE representations. We hypothesize that the low number of expertise fields associated with this dataset largely reduces the effect described in the previous section.
In this paper, we provide experiment materials for expert finding with the help of four annotated datasets and further report results based on several baseline algorithms. Moreover, we study the ability of document network embedding methods to tackle the expert finding challenge. We show that DNE algorithms can not be trivially adapted to achieve state-of-the-art scores. However, we reveal that document network embeddings can improve the voting model but diminish the propagation model.
In future work, we would like to find an efficient way to bridge the gap between DNE algorithms and expert finding. To do so, taking the heterogeneity into account should help better capturing the real similarity between a document and a candidate. Furthermore, a deeper analysis of the interplay between the candidates and the text content of the documents appears to be a necessary way to better understand the task of expert finding.