New Datasets and a Benchmark of Document Network Embedding Methods for Scientific Expert Finding

Abstract

The scientific literature is growing faster than ever. Finding an expert in a particular scientific domain has never been as hard as today because of the increasing amount of publications and because of the ever growing diversity of expertise fields. To tackle this challenge, automatic expert finding algorithms rely on the vast scientific heterogeneous network to match textual queries with potential expert candidates. In this direction, document network embedding methods seem to be an ideal choice for building representations of the scientific literature. Citation and authorship links contain major complementary information to the textual content of the publications. In this paper, we propose a benchmark for expert finding in document networks by leveraging data extracted from a scientific citation network and three scientific question & answer websites. We compare the performances of several algorithms on these different sources of data and further study the applicability of embedding methods on an expert finding task.

Introduction

Many tools offer to search and filter the vast data sources available on the Web. In particular, there is a multitude of platforms directed to the scientific community. From the simple search engine for publications to the social network for researchers, all consume and produce valuable data for searching scientific content of interest. Expert finding is one the the most challenging problem that finds application in both academia and the industry. To tackle this challenge, recent advances in document network embedding (DNE) has the potential to inspire new unsupervised models that can deal with the heterogeneous network of documents of the scientific literature. However, the design of such efficient algorithms heavily depends on the development of strong evaluation frameworks.

In this paper, we propose a methodology and provide 4 datasets that extend the limited scope of expertise retrieval evaluation frameworks. Furthermore, we provide experiment results computed with unsupervised methods and we extend document network embedding algorithms to this specific task.

Our contributions are the following: 2 – we provide 4 datasets for expert finding extracted from a scientific publication network and three question & answer (Q&A) websites and make them publicly available 1; – we describe an evaluation methodology based on the ranking of expert candidates given a set of labeled document queries; – we report experiment results that give some insights on this expert finding task; – we explore and analyze the use of state-of-the-art document network embedding algorithms for expert finding and we show that further research is needed to bridge the gap between DNE methods and expert finding.

The rest of the paper is organized as follows. In Section 2, we survey related works. We detail in Section 3 our evaluation methodology, the datasets we extracted, the evaluation measures and the algorithms we use. In Section 4 we show and analyze the results of our experiments. Finally, in Section 5, we discuss our findings and provide future directions. 2

Related Works

In this section, we first present a formal definition for expert finding. Then we present algorithms of the literature that address expert finding. Finally, we describe recent methods for document network embedding that have the potential to deal with this particular task. 2.1

Formal definition of expert finding

The concept of expert finding can cover a large range of tasks. The main principle behind expertise retrieval is the search for candidates given a query. To match these two, an algorithm will be provided with some data to link the output space, a ranking of candidates, with the input space, which is often a textual content. However, many different types of data can be considered to address this challenge. To fairly compare algorithms, we choose a fixed structure for the data which reflects common use cases. Furthermore, if supervised methods benefit from labeled fields of expertise associated with the candidates, they are beyond the scope of this paper which focuses on unsupervised methods only. Our goal is to compare methods that do not require sometimes costly annotations.

Early works in expert search [ 8 ] usually consider a small set of topical queries. The direct namings of these topics are used to retrieve a list of candidates by leveraging a collection of documents they published (e.g., emails, scientific papers). This type of evaluation is used across several public datasets [ 15, 16, 26 ].

More recently, the concept of expert finding has been merged into the wider concept of entity retrieval [ 1 ]. As more and more complex data are produced on the Web, expert finding becomes a particular application of entity search. At the same time, Q&A websites such as Stack Overflow 2 generate and make publicly accessible a big amount of questions with expert answers, collaboratively curated by their users. Several works 1 https://github.com/brochier/expert_finding 2 https://stackoverflow.com/ address the search for experts in such websites [ 20,28 ]. Often, the task consists in either finding the exact list of users who answered a specific question or ranking the answers according to the user votes. In the first case, the task involves considering the evolution of the users across time and, in the second case, the task involves understanding the intrinsic quality of a written answer. Nevertheless, [ 25 ] reviews several models for expert finding in Q&A websites. Their experiments show that matrix factorization-based methods perform better than tree based and ranking based methods.

In this paper, we adopt the document-query methodology recently proposed in [ 3 ]. The expert search is performed given a set of queries that are particular textual instances of some expert topics (or fields of expertise). Given a query, an algorithm should rank first the candidates that are associated to the same fields of expertise. We provide 4 datasets for which we annotated experts and document queries. Each dataset consists in candidates and documents linked by authorship relations (candidate-document e.g. authorship) and by response relations (document-document e.g citation or answer). A query is therefore one of the documents (e.g. a scientific paper or a question) for which we aim to retrieve some experts of the topics depicted in it. This configuration reflects many real case scenarios such as (1) the automatic search for scientific reviewers, (2) the recommendation of expert users in Q&A websites or even (3) the retrieval of interesting profiles for job offers. 2.2

Algorithms for expert finding

Numerous works have addressed automatic expertise retrieval. We describe here the main approaches and some interesting recent methods. P@noptic Expert [ 7 ] creates meta-documents for a candidate by concatenating the contents of all documents she produced. In this manner, ranking the candidates given a query becomes a similarity search between the query representation and the meta-documents representations. A voting model [ 14 ] computes the similarities between the query and the documents. The algorithm then aggregates these scores at the candidate level by using a fusion technique such as the reciprocal rank [ 27 ]. A propagation model [ 21 ] takes advantage of the links between candidates and documents to propagate the similarities between the query and the documents. Using random walks with restart [ 17 ], the iterative propagation of the scores converges in a few steps to a stationary distribution over the candidates. WISER [ 6 ] models each candidate as a small, weighted, sub-graph of the Wikipedia Knowledge Graph. Information derived from these graphs and traditional document retrieval techniques are combined to identify experts w.r.t a query. Note that methods leveraging external data are out of the scope of our benchmark. LT Expertfinder is an evaluation framework for expert finding [ 11 ] based on an interactive tool. It integrates various existing algorithms (such as [ 1 ]) in a user-friendly way. The underlying corpus used by this tool is the ACL Anthology Network. However, it does not include a wellestablished ground truth to assess who are the experts. Indeed, the evaluation is purely done in an online manner since the user has to evaluate the degree of expertise based on several features, such as author’s citations, h-index, keywords, etc. Recent works [ 9,23 ] propose ad hoc embedding techniques, whereas, in this work, we’re interested in measuring the performance of conventional network embedding techniques. 4 2.3 Network embedding [ 12, 19 ] provides an efficient approach to represent nodes in a low dimensional vector space, suitable for solving various machine learning tasks. Recent techniques extend NE for document networks. Text-Associated DeepWalk (TADW) [ 24 ] extends DeepWalk to deal with textual attributes. Yang et al. prove, following the work in [ 13 ], that Skip-Gram with hierarchical softmax can be equivalently formulated as a matrix factorization problem. TADW then consists in constraining the factorization problem with a pre-computed representation of the documents by using Latent Semantic Analysis (LSA) [ 10 ]. Graph2Gauss (G2G) [ 2 ] is an approach that embeds each node as a Gaussian distribution instead of a vector. The algorithm is trained by passing node attributes through a non-linear transformation via a deep neural network (encoder). GVNR-t [ 4 ] is a matrix factorization approach for document network embedding, inspired by GloVe [ 18 ], that simultaneously learns word, node and document representations by optimizing a least-square objective over a co-occurrence matrix of the nodes constructed by truncated random walks. IDNE [ 5 ] introduces a Topic-Word Attention mechanism, trained from the connections of a document network, to represent documents as mixtures of topics.

DNE algorithms do not directly apply to expert finding data since they are not designed to handle multiple types of nodes, in particular candidate nodes. In this paper, we show (1) two methods to extend their applicability to the task of expert finding and (2) the impact of their representations when they are used as document representations for traditional expert finding algorithms. 3

Evaluation Methodology

We present in this section the evaluation methodology that we follow to access the performances of several algorithms for expert finding. We first describe the task we seek to solve, then we describe the datasets that we extracted and explain how we annotated them in order to access the quality of the algorithms’ outputs. Finally, we detail the models used in our experiments. 3.1

Ranking expert candidates from document queries

Expert finding is a complex task that can be formalized in multiple ways. Early works define this task as a ranking problem given several topic-queries where the naming of these topics are directly used as queries to retrieve the expert candidates. However, in many real world applications, a user is asked to provide a specific and detailed query. In a Q&A website for instance, a user usually exposes the problem she faces in full detail and does not necessarily know the exact naming of the fields of expertise needed to solve her problem. Furthermore, querying an algorithm with a small set of topicqueries can lead to poor evaluation measures due to the usually small number of fields of expertise associated with the dataset. For this reasons, we follow the document-query evaluation methodology proposed in [ 3 ] by processing 4 datasets for which a set of document-queries is manually annotated.

The expert finding task in this paper is a ranking problem. Given a document labeled with a ground truth set of fields of expertise, an algorithm is queried to rank a set of candidates, among which a subset of experts are associated with the same set of labels. The data provided to the algorithms consists in a corpus of nd documents D, nc candidates C, a network of authorship with adjacency matrix Adc 2 Nnd nc and a network of documents with adjacency matrix Add 2 Nnd nd . Figure 1 shows an hypothetical dataset used in this paper. The ranking is performed in an unsupervised setting, that is, no ground truth labels of expertise are given to the algorithms. The set of labeled documents (the queries) can be smaller than nd and the set of labeled candidates (experts) can be smaller than nc (i.e not all documents and candidates are labeled).

Expertise labels

C1

C2

C3

C4

C5 D1

D2

D3

D4

D5

D6 Adc Add the rankings D1 7! C3C4C5C1C2 and D6 7! C4C5C3C2C1.

To evaluate the candidate scores provided by the algorithms, we compare the resulting rankings with the ground truth fields of expertise. If a document is associated with three different labels, we expect the algorithm to rank first all experts associated to at least one of these labels. We report the area under the ROC curve (AUC), the precision 6 at 10 (P@10) and the average precision (AP) and we compute their standard deviation along the queries. That is, we evaluate the robustness of the algorithms against the variety of document-queries. We consider 4 datasets. The first one is an extract of DBLP [ 22 ] in which a list 199 experts in 7 fields are annotated [ 26 ] by human judgments 3. Our dataset only considers the annotated experts and the other candidates that are close in the co-authorship network which explains the relatively small size of our network compared to the original one. In addition to the expert annotations, our evaluation framework requires document annotations since we adopt the document-query methodology for expertise retrieval. We asked two PhD students in computer science to associate independently 20 randomly drawn documents per field of expertise (140 in total). Then, only the labels on which the two annotators agreed were kept, leaving 114 annotated papers. The mean Cohen’s kappa coefficient across the labels is 0:718. An advantage of our methodology is that we can evaluate the algorithms on more queries (114 documents) than the traditional method (7 labels). This allows us to assess the robustness of the algorithms by computing the standard deviations of the ranking metrics along all queries. However, one might suggest that these 7 labels do not reflect a representative set of expertise as there are too broad. For this reason, we seek for a wider granularity of expertise by the use of well-know question & answer website.

If scientific publication networks are easy to find on the Web, scientific expertise annotations are rarely available for both authors and publications. We use data downloaded in June 2019 from Stack Exchange 4 to create datasets for expert finding collected from three communities closely related to research. Academia 5 is dedicated to academics and higher education. Mathoverflow 6 gathers professional mathematicians and is widely used by researchers. Stats 7 (also known as Cross Validated) addresses statistics, machine learning and data mining issues. For each dataset, we first keep questions with at least 10 user votes that have at least one answer with 10 user votes or more. We build the networks by linking questions with their answers and by linking answers with the users who published them. The field of expertise are the tags associated with the questions. Only the tags that occur at least 50 times are kept. We annotate an expert with the tags of a question if her answer to that question received at least 10 votes. Note that the tags are first provided by the users who ask the questions but they are thereafter verified by experimented users.

The general properties of our 4 datasets are presented in Table 1. The annotations and the preprocessed datasets are made publicly available. 3 https://lfs.aminer.cn/lab-datasets/expertfinding/#expert-list 4 https://archive.org/details/stackexchange 5 https://academia.stackexchange.com/ 6 https://mathoverflow.net/ 7 https://stats.stackexchange.com/

Datasets and Benchmark of DNE Methods for Expert Finding 7

DBLP Stats Academia Mathoverflow

We run the experiments with 4 baseline algorithms and 4 document network embedding algorithms. The laters are adapted with two aggregation schemes in order to deal with the candidates since they are primarily designed for document network only. These aggregations are arbitrary and are voluntarily the most straightforward way to run DNE algorithms on bipartite networks of authors-documents. We further discuss these choices in section 4.

Baselines We run the experiments with the same models as in [ 3 ], using the tf-idf representations and the cosine similarity measure. Also, we add a random model to have reference metrics: – Random model: we randomly draw scores between 0 and 1 for each candidate; – P@noptic model [ 7 ]: we concatenate the textual content of each document associated to the candidates, use their tf-idf representations and compute the cosine similarity to produce the scores; – Voting model [ 14 ]: we use the reciprocal rank to aggregate the scores at the candidate level; – Propagation model [ 21 ]: we concatenate the two adjacency matrices Adc and Add to construct a transition matrix between candidates and documents such that A = AAdd|dc A0dc . The initial scores are the cosine similarities between the tf-idf representations of the query and the documents. The scores are propagated iteratively until convergence with a restart probability of 0:5.

We also run the voting and propagation models using document representations produced by IDNE in place of the tf-idf vectors. The document network provided to IDNE has adjacency matrix Ad = AdcAd|c + Add.

Extending DNE algorithms for expert finding DNE methods usually operate in net

works of documents, with no candidate nodes. To apply them in the context of expert finding, we propose two straightforward approaches: – pre-aggregation: as in the P@noptic model, meta-documents are generated by aggregating the documents produced by each candidates. Furthermore, an adjacency matrix of a meta-network between candidates and documents is constructed. We compute a candidate network as Ac = Ad|cAdc and a document network as 8 For all datasets, the propagation model performs generally better than the other algorithms, particularly in terms of precision. Both aggregation schemes yield to poor results and none of these two methods appear to be better than the other. GVNR-t is the best algorithm among the document network embedding models. We believe that, if DNE algorithms are well suited for document network representation learning, the gap between simple tasks such as node classification and link prediction and the task of expert finding is too big for our naive aggregation schemes to perform well. Especially, the network structure changes significantly between an homogeneous network and an Datasets and Benchmark of DNE Methods for Expert Finding 9 (a) Propagation model with tf-idf representations: the curve has a nice shape which means the ranking of candidates are good even for the last ranked experts. (b) Propagation model with IDNE representations: the first ranked candidates are good but the algorithm tends to wrongly rank last many true experts. heterogeneous network. Moreover, expert finding algorithms often benefit from information about the centrality of the candidates and documents. DNE algorithms do not particularly preserve this information neither do our aggregation schemes. 4.2

Using DNE as document representations for the baselines

Since the baseline algorithms perform well, we study the possibility to apply them using a DNE algorithm for the representations of the documents. We only report the results with the representations computed with IDNE but we observe the same behaviors with other DNE models. First, these representations constantly improve the voting model, which achieves best results in terms of AUC on Stats and Mathoverflow. Then, the most surprising effect is the significant decrease of performance of the propagation model. If the precision for the first ranked candidates is not affected, the AUC score significantly drops for the three Q&A datasets. We believe that document network embeddings captures too long-range dependencies between the documents in the network, which are then subsequently exaggerated by the propagation procedure. Figure 2 shows the effect of the representations used with the propagation model on the ROC curve. 4.3

Differences between the datasets

The results achieved by the algorithms on all three Stack Exchange datasets are consistent. However, they do not behave the same with DBLP. First, DNE methods get closer scores to the baselines on DBLP. In the Q&A datasets, the interactions are more isolated i.e. there are more users having fewer interactions. This difference of network properties might disadvantage DNE methods who are usually trained on scale-free networks whose degree distribution follows a power law. Moreover, the propagation method does not suffer with DBLP from the decrease of performance induced by the IDNE representations. We hypothesize that the low number of expertise fields associated with this dataset largely reduces the effect described in the previous section. 5

Discussion and Future Work

In this paper, we provide experiment materials for expert finding with the help of four annotated datasets and further report results based on several baseline algorithms. Moreover, we study the ability of document network embedding methods to tackle the expert finding challenge. We show that DNE algorithms can not be trivially adapted to achieve state-of-the-art scores. However, we reveal that document network embeddings can improve the voting model but diminish the propagation model.

In future work, we would like to find an efficient way to bridge the gap between DNE algorithms and expert finding. To do so, taking the heterogeneity into account should help better capturing the real similarity between a document and a candidate. Furthermore, a deeper analysis of the interplay between the candidates and the text content of the documents appears to be a necessary way to better understand the task of expert finding.

Datasets and Benchmark of DNE Methods for Expert Finding 11

R. Brochier et al.

Datasets and Benchmark of DNE Methods for Expert Finding 13

R. Brochier et al.

References

1. Balog , K. , Serdyukov , P. , De Vries , A.P. : Overview of the trec 2010 entity track . Tech. rep. , NORWEGIAN UNIV OF SCIENCE AND TECHNOLOGY TRONDHEIM ( 2010 )

2. Bojchevski , A. , Gu¨nnemann, S.: Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking . In: International Conference on Learning Representations . pp. 1 - 13 ( 2018 )

3. Brochier , R. , Guille , A. , Rothan , B. , Velcin , J.: Impact of the query set on the evaluation of expert finding systems . Proceedings of the 3rd Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2018 ) co-located with the 41st International ACM SIGIR Conference ( 2018 )

4. Brochier , R. , Guille , A. , Velcin , J.: Global vectors for node representations . In: The World Wide Web Conference . pp. 2587 - 2593 . ACM ( 2019 )

5. Brochier , R. , Guille , A. , Velcin , J.: Inductive document network embedding with topic-word attention . In: Proceedings of the 42nd European Conference on Information Retrieval Research . Springer ( 2020 )

6. Cifariello , P. , Ferragina , P. , Ponza , M. : Wiser: A semantic approach for expert finding in academia based on entity linking . Inf. Syst . 82 , 1 - 16 ( 2019 )

7. Craswell , N. , Hawking , D. , Vercoustre , A.M. , Wilkins , P. : P@ noptic expert: Searching for experts not just for documents . In: Ausweb Poster Proceedings, Queensland, Australia . vol. 15 , p. 17 ( 2001 )

8. Craswell , N., de Vries , A.P. , Soboroff , I. : Overview of the trec 2005 enterprise track . In: Trec . vol. 5 , pp. 1 - 7 ( 2005 )

9. Dargahi Nobari , A. , Sotudeh Gharebagh , S. , Neshati , M. : Skill translation models in expert finding . In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval . pp. 1057 - 1060 . ACM ( 2017 )

10. Deerwester , S. , Dumais , S.T. , Furnas , G.W. , Landauer , T.K. , Harshman , R.: Indexing by latent semantic analysis . Journal of the American Society for Information Science 41 ( 6 ), 391 - 407 ( 1990 )

11. Fischer , T. , Remus , S. , Biemann , C. : Lt expertfinder: An evaluation framework for expert finding methods . In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) . pp. 98 - 104 ( 2019 )

12. Grover , A. , Leskovec , J.: node2vec: Scalable feature learning for networks . In: Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining . pp. 855 - 864 . ACM ( 2016 )

13. Levy , O. , Goldberg , Y. : Neural word embedding as implicit matrix factorization . In: Advances in neural information processing systems . pp. 2177 - 2185 ( 2014 )

14. Macdonald , C. , Ounis , I. : Voting for candidates: adapting data fusion techniques for an expert search task . In: Proceedings of the 15th ACM international conference on Information and knowledge management . pp. 387 - 396 . ACM ( 2006 )

15. Macdonald , C. , Ounis , I. , Soboroff , I. : Overview of the trec 2007 blog track . In: TREC . vol. 7 , pp. 31 - 43 ( 2007 )

16. Mislevy , R.J. , Riconscente , M.M. : Evidence-centered assessment design . In: Handbook of test development , pp. 75 - 104 . Routledge ( 2011 )

17. Page , L. , Brin , S. , Motwani , R. , Winograd , T. : The pagerank citation ranking: Bringing order to the web . Tech. rep. , Stanford InfoLab ( 1999 )

18. Pennington , J. , Socher , R. , Manning , C. : Glove: Global vectors for word representation . In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) . pp. 1532 - 1543 ( 2014 )

19. Perozzi , B. , Al-Rfou , R. , Skiena , S. : Deepwalk: Online learning of social representations . In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining . pp. 701 - 710 . ACM ( 2014 )

20. Riahi , F. , Zolaktaf , Z. , Shafiei , M. , Milios , E.: Finding expert users in community question answering . In: Proceedings of the 21st International Conference on World Wide Web . pp. 791 - 798 . ACM ( 2012 )

21. Serdyukov , P. , Rode , H. , Hiemstra , D. : Modeling multi-step relevance propagation for expert finding . In: Proceedings of the 17th ACM conference on Information and knowledge management . pp. 1133 - 1142 . ACM ( 2008 )

22. Tang , J. , Zhang, J., Yao , L. , Li , J. , Zhang, L. , Su , Z. : Arnetminer: extraction and mining of academic social networks . In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining . pp. 990 - 998 . ACM ( 2008 )

23. Van Gysel , C. , de Rijke , M. , Worring , M. : Unsupervised, efficient and semantic expertise retrieval . In: WWW . vol. 2016 , pp. 1069 - 1079 . The International World Wide Web Conferences Steering Committee ( 2016 )

24. Yang , C. , Liu , Z. , Zhao , D. , Sun , M. , Chang , E. : Network representation learning with rich text information . In: Twenty-Fourth International Joint Conference on Artificial Intelligence ( 2015 )

25. Yuan , S. , Zhang, Y. , Tang , J. , Hall , W. , Cabota`, J.B. : Expert finding in community question answering: a review . Artificial Intelligence Review 53 ( 2 ), 843 - 874 ( 2020 )

26. Zhang , J., Tang , J. , Li , J. : Expert finding in a social network . In: International Conference on Database Systems for Advanced Applications . pp. 1066 - 1069 . Springer ( 2007 )

27. Zhang , M. , Song , R. , Lin , C. , Ma , S. , Jiang , Z. , Jin , Y. , Liu , Y. , Zhao , L. , Ma , S.: Expansionbased technologies in finding relevant and new information: Thu trec 2002: Novelty track experiments . NIST SPECIAL PUBLICATION SP 251 , 586 - 590 ( 2003 )

28. Zhao , Z. , Zhang , L. , He , X. , Ng , W. : Expert finding for question answering via graph regularized matrix completion . IEEE Transactions on Knowledge and Data Engineering 27 ( 4 ), 993 - 1004 ( 2014 )