Explainable IR for personalizing professional search

Abstract

In this position paper we establish the need for transparency in personalized professional search. We provide a brief overview of prior work, identify the gaps, and list four research directions that need to be explored to close these gaps. The central idea of our proposal is the professional knowledge graph. Graphs are a natural and transparent means of representing knowledge. A graph-based search paradigm enables and stimulates the exploratory search behaviour for complex information needs that are inevitable in professional work environments.

Introduction

Professional searchers, such as lawyers, policy o cers, architects, and scientists, need to process increasing amounts of documents to nd relevant, complete, high quality, work-related information [ 4, 35 ]. Not being able to nd the needed information is a costly problem in our information-driven society in which the amount of available information from diverse sources is amplifying (internet, digital libraries, internal collections).

A problem of the general search paradigm when applied work-related search is that result ranking relies on popularity of web pages: the more often a result is clicked for a given query, the higher it is ranked in future searches [19]. However, information search by professionals is essentially di erent from generic web search in three important aspects:

The search tasks of professionals are complex, i.e. highly-speci c and typically recall-oriented: the searchers want to be sure that they have found all the relevant information [27, 21]; The searching is not limited to sending one query and clicking one result, but is often exploratory by nature [15], and includes browsing, analysing [ 26 ] and re- nding previously used information [ 36 ]; Each user has their own individual needs: not only interests, expertise and information needs di er per user, but also the perceived relevance of retrieved documents [ 40 ]. The search evolves on the searcher's own knowledge.

Because the information needs are highly speci c and individual in professional search, the click data available from other users is limited and irrelevant [17]. Hence, result ranking cannot depend on popularity.

Thus, for e ective professional search, a di erent approach is necessary. We argue that the search results should not depend on a single query matched to the collection of documents, but should be centred around the knowledge of the individual user, allowing to serve their highly speci c information needs. This idea is based on the classic model for information seeking by Dervin in which a search is motivated by the gap between what the user already knows and what he wants to know [ 11, 25 ].

Copyright c by the paper's authors. Copying permitted for private and academic purposes.

For a search engine to be centred around the knowledge of the user, a user pro le must be created and utilized for personalized ranking. User pro ling and personalization have been addressed extensively in IR research, but barely in the context of professional search. The reason is that transparency is essential in work-related search: professional users do not want to have the feeling they lose control over the search process because the ranking of the search results is not stable or not predictable.

Our position

The lack of methods for transparent personalized professional search is a gap that should be addressed in IR research. We argue that it is time to change the classic query-based paradigm of information retrieval and move towards environments that allow users to explore their own knowledge, identify the knowledge gap, explore the surrounding content and nding the hooks where the new information should be attached.

For that purpose we propose the concept of the professional knowledge graph, an automatically deduced knowledge graph of terms and documents that are relevant to the individual user. Graphs are a natural and transparent means of representing knowledge [ 6 ]. A graph-based search paradigm enables and stimulates the exploratory search behaviour for complex information needs that are inevitable in professional work environments [15].

In the remainder of this paper we rst de ne the aims and objectives for transparent personalization in professional search (Section 2). In Section 3 we give an outline of prior and related work. In Section 4 we outline the research topics that need to be addressed in order to meet the aims and objectives. We conclude our paper with recommendations in Section 5. 2

Future aims and objectives

Successful personalized professional search relies on transparent and explainable IR: opening up the black box of the search algorithm and making the user's knowledge the central component of the search experience. To that end, three research challenges need to be addressed: 1. to create a human and machine understandable representation of the knowledge of the user. Methods should be developed to deduce the user's professional knowledge graph from his searching and reading history. 2. to utilize the user's knowledge graph for more e ective information retrieval. Methods should be developed to utilize the information in the graph by an existing retrieval system to better rank the relevant documents for the user.

3. to do this in a transparent way.

3

Prior and related work

User pro ling in domain-speci c search. Approaches to user pro ling and personalization typically learn user preferences by collecting queries and clicked documents [ 28 ]. A rich user pro le can be learned by extracting prominent terms from the clicked documents and storing them in a term pro le [ 43, 41 ]. The term pro le can then be used for re-ranking search results [ 29 ], for query disambiguation [ 42 ], query expansion [ 48 ], or query suggestion [ 22, 46 ]. Often, the extracted information is linked to a reference ontology [ 39, 10 ].

Although all these works report an improvement of personalization over the non-personalized baseline, the actual implementation of personalization strategies in search environments is limited: on average, only 11.7% of Google Web Search results show di erences due to personalization [ 13 ]. This is because users are wary when it comes to personalization. Privacy-preserving personalization is an important societal topic [ 32, 20 ]. As such, a crucial step in the development of privacy-secure systems is to make the system transparent and explainable [16]. Recently, explainable methods have found their way to the eld of recommendation and search [ 2, 7 ]. It is important for users to have insight in the data that is stored by the search engine [ 47 ] and to understand the in uence of their personal data on the search results. Transparent/Explainable IR was also addressed as a discussion topic during the Third Strategic Workshop on Information Retrieval [ 8 ], indicating its importance for the research community.

Knowledge graphs in IR. The use of knowledge graphs for text processing and information retrieval has gained attention from the research community in the past years [ 1, 3, 12 ]. Knowledge graphs have been shown to be especially helpful in exploratory search [ 37, 38 ], and to model the semantic relations between documents [ 34 ]. When knowledge graphs are combined with search logs they give insight in the user's facts and beliefs of the search topic [14]. Most previous works in graph-based search use an external knowledge graph covering all domain knowledge. A graph representing the knowledge and interests of one user, is much smaller than a graph representing the complete index of a search engine [ 5 ] and can be stored locally (client-side), if privacy regulations require it.

Data for user pro ling in domain-speci c search. An important gap for learning and evaluating

user pro les for professional search is that there are no data sets available that contain explicit descriptions of information needs and background knowledge, together with search activity data and relevance judgments.

An important example was set by the iSearch data set [ 24 ]: this collection contains 65 personal information needs (topics) described by 23 physics scientists. The iSearch dataset is unique in size and richness of the topics; it provides a valuable test bed for domain speci c search. However, for experiments on user pro ling the iSearch data lacks an important component: user interaction data corresponding to the topic, i.e. issued queries and clicked documents in a search engine. 4

Research proposal

In this section we propose a line of research that addresses the need for transparent personalized search in professional contexts. The research line consists of four steps, which will each be addressed in the following subsections:

1. Data collection;

2. Methods for constructing professional knowledge graphs;

3. Methods for transparent personalization;

4. Evaluation protocol for transparent personalized search.

The rst necessary step will be to collect personal information needs (topics) of professionals and link them to individual user interaction data. Like with the iSearch data, the topics can be collected by means of semi-structured interviews and questionnaires with users of a professional search engine who agree to provide information needs and give consent to storage of their interactions with the search engine. These users would be asked to describe an information need, their background knowledge on the topic and the context of the work task (cf. [ 24 ]).

The search logs of the participants should then be collected from the back-end of the search engine. The search logs consist of all issued queries and accessed documents, together with timestamps. The data could be analysed semi-automatically, to split the search behaviour in search stages [15] and to map the information needs to the actual search behaviour. 4.2

Methods for constructing the professional knowledge graphs

For graph construction, the main challenge is selecting those terms that constitute a good (informative) user pro le. Terms could be selected in three ways: keyword extraction from clicked documents, named entity extraction from clicked documents and term extraction from queries. Documents, terms and user interactions (clicks, reads) can then be stored as nodes in a heterogeneous weighted graph. The edges between nodes might represent the similarity between two documents [ 34 ], the similarity between terms [ 31, 30 ], and the representativeness of a term for a document (tf-idf weight). Figure 2 illustrates a professional knowledge graph with an excerpt of a graph with two terms and two documents as nodes, and weights on the relations between the nodes.

One additional challenge in storing the user pro le is that there will be change in information needs over the time (gradually or suddenly, because of diverging professional interests). The risk is that users will end up searching their own lter bubble. Therefore, it is important to balance the exploitation of the user pro le and the exploration of new directions. 4.3

Methods for transparent personalization

E ectively utilizing knowledge graphs from sparse user data for e ective information nding is probably the most challenging research direction of the four.

In pilot experiments we explored how the professional knowledge graph can be used for better ranking the retrieved documents given a user query using in a two-stage retrieval method [ 45 ] (thus, implementing the professional knowledge graph in the current classic query-based IR model). Given a user query, the rst step was retrieval of the 1000 most relevant documents according to the default ranking algorithm in the search engine. The professional knowledge graph was then utilized to re-rank (2nd stage) the 1000 documents, resulting in a personalized ranking. The goal in the re-ranking (personalization) step was to estimate the personal relevance of the retrieved documents, based on the knowledge in the graph. We did this by temporarily adding each candidate document to the user's graph and computing their centrality. This is challenging in a heterogeneous weighted graph with multiple types of nodes, edges and weights. We tackled this challenge by building on methods for combining multiple node characteristics in one metric [ 33 ] and implement them in a learning-torank framework [ 23, 9 ]. We obtained a small but signi cant improvement over the non-personalized baseline.

Future research with professional knowledge graphs should diverge from the classic IR model. Completely new methods need to be developed for (1) browsing the professional knowledge graph, (2) assisting the user in identifying their knowledge gap, (3) assessing the relevance of documents in the heterogeneous graph. 4.4

Evaluation protocol for transparent personalized search

The e cacy of the professional knowledge graph for personalized ranking could be evaluated in two ways: 1) with a simulation using log data, and 2) with users.

For the data-centric evaluation, historical user queries and relevance assessments can be deduced from click data [19] to set up interaction simulations [ 46 ] in order to measure the e ect of personalized ranking compared to the original, non-personalized ranking of documents.

For the user-centric evaluation, a demo interface needs to be developed in which the user can view his professional knowledge graph and see the e ect of the graph content on the document ranking. In a withinsubject setting, the classic view of the search engine (control setting) can then be compared with the personalized search engine (experimental setting). Outcome measures should be: (1) how long do the users take to ful l the information need [18], to be measured using server-side logging measure; and (2) user satisfaction, do be measured using a post-task questionnaire [ 44 ]. In the questionnaire, it should be evaluated (a) how satis ed the users are with the answer; (b) how satis ed the users are with the usability of the interactive viewer for the task and (c) how satis ed the users are with the transparency of the tool. 5

Conclusions and recommendations

In this position paper we have established the need for transparency in personalized professional search. We have provided a brief overview of prior work, identi ed the gaps, and listed four research directions that need to be explored to close the gaps.

In summary, we argue that: 1. Data collection is instrumental for research in professional search. Data sets with user-generated input are sparse in the eld, because user-centric research is time-expensive and target group users are not always available to provide input. Work is needed to collect a truly user-centric dataset that includes both information needs and search engine logs. The data should be made available to other researchers in the eld. 2. Knowledge graphs provide a great potential to transparency and personalization in information search. Research is needed to develop methods for constructing individual professional knowledge graphs and evaluating those with expert users. 3. There is a large body of academic work on personalization, but personalization in professional search engines is still limited, because transparency is essential for professional users. Research on professional search should include transparency by design. The IR community should bring together research on professional search, knowledge graphs, and explainable IR. 4. For the e ective exploitation of the professional knowledge graphs, new methods need to be developed for retrieval environments that are centred around the knowledge graphs.

Towards graphical models for text processing. Knowledge and information [2] Ai, Q., Zhang, Y., Bi, K., Chen, X., Croft, W.B.: Learning a hierarchical embedding model for personalized product search. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM (2017) 645{654

[3] Alonso, O., Hearst, M.A., Kamps, J.: Report on the First SIGIR Workshop on Graph Search and Beyond (GSB'15). ACM SIGIR Forum 49(2) (2016) 89{97 [14] He, J., Bron, M.: Measuring Demonstrated Potential Domain Knowledge with Knowledge Graphs. In: Proceedings of The First Workshop on Knowledge Graphs and Semantics for Text Retrieval and Analysis (KG4IR). (2017) [15] He, J., Bron, M., de Vries, A.P.: Characterizing stages of a multi-session complex search task through direct and indirect query modi cations. In: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, ACM (2013) 897{900 [16] Holzinger, A., Biemann, C., Pattichis, C.S., Kell, D.B.: What do we need to build explainable AI systems for the medical domain? arXiv preprint arXiv:1712.09923 (2017) [17] Huang, Z., Cautis, B., Cheng, R., Zheng, Y.: KB-Enabled Query Recommendation for Long-Tail Queries.

In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, ACM (2016) 2107{2112 [18] Ingwersen, P., Jarvelin, K.: The Turn: Integration of Information Seeking and Retrieval in Context (The

Information Retrieval Series). Volume 18. Springer (2005) [19] Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA, ACM (2005) 154{161 [20] Karwatzki, S., Dytynko, O., Trenz, M., Veit, D.: Beyond the PersonalizationPrivacy Paradox: Privacy Valuation, Transparency Features, and Service Personalization. Journal of Management Information Systems 34(2) (apr 2017) 369{400 [21] Kim, Y., Seo, J., Croft, W.B.: Automatic boolean query suggestion for professional search. In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval.

SIGIR '11, New York, NY, USA, ACM (2011) 825{834 [23] Liu, T.Y.: Learning to rank for information retrieval. Foundations and Trends R in Information Retrieval 3(3) (2009) 225{331 [27] Mason, D.: Legal Information Retrieval StudyLexis Professional and Westlaw UK. Legal Information

Management (2006)

In: arXiv

References

[1] Aggarwal , C.C. , Zhao , P. : systems 36(1) ( 2013 ) 1 { 21

[4] Bawden , D. , Robinson , L. : The dark side of information: Overload, anxiety and other paradoxes and pathologies . Journal of Information Science 35 ( 2 ) (apr 2009 ) 180 { 191

[5] Blanco , R. , Lioma , C. : Graph-based term weighting for information retrieval . Information retrieval 15(1) ( 2012 ) 54 { 92

[6] Chein , M. , Mugnier , M.L. : Graph-based knowledge representation: computational foundations of conceptual graphs . Springer Science & Business Media ( 2008 )

[7] Chen , X. , Zhang , Y. , Ai , Q. , Xu , H. , Yan , J. , Qin , Z. : Personalized key frame recommendation . In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval , ACM ( 2017 ) 315 { 324

[8] Culpepper , J.S. , Diaz , F. , Smucker , M.D.e.: Research frontiers in information retrieval report from the third strategic workshop on information retrieval in lorne (swirl 2018 ). Technical report ( 2018 )

[9] Dalton , J. , Dietz , L. , Allan , J.: Entity query feature expansion using knowledge base links . In: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval , ACM ( 2014 ) 365 { 374

[10] Daoud , M. , Lechani , L.T. , Boughanem , M. : Towards a graph-based user pro le modeling for a session-based personalized search . Knowledge and Information Systems 21 ( 3 ) ( 2009 ) 365 { 398

[11] Dervin , B. : From the mind's eye of the user: The sense-making qualitative-quantitative methodology . Qualitative research in information management 9 ( 1992 ) 61 { 84

[12] Dietz , L. , Meij , E. , Bloomberg , L.P. : Overview of The First Workshop on Knowledge Graphs and Semantics for Text Retrieval and Analysis . CEUR Workshop Proceedings 51 ( 3 ) ( 2017 ) 139 { 144

[13] Hannak , A. , Sapiezynski , P. , Molavi Kakhki , A. , Krishnamurthy , B. , Lazer , D. , Mislove , A. , Wilson, C. : Measuring personalization of web search . In: Proceedings of the 22Nd International Conference on World Wide Web. WWW '13 , New York, NY, USA, ACM ( 2013 ) 527 { 538 [22] Leung , K.W.T. , Ng , W. , Lee , D.L. : Personalized concept-based clustering of search engine queries . IEEE transactions on knowledge and data engineering 20 ( 11 ) ( 2008 ) 1505 { 1518

[24] Lykke , M. , Larsen , B. , Lund , H. , Ingwersen , P. : Developing a Test Collection for the Evaluation of Integrated Search . In Gurrin, C. , He , Y. , Kazai , G. , Kruschwitz , U. , Little , S. , Roelleke , T. , Ruger, S., van Rijsbergen, K., eds. : Advances in Information Retrieval. Volume 5993 of Lecture Notes in Computer Science . Springer Berlin / Heidelberg ( 2010 ) 627 { 630

[25] Mai , J.E. : Looking for Information: A Survey of Research on Information Seeking, Needs, and Behavior . Emerald Group Publishing ( 2016 )

[26] Makri , S. , Blandford , A. , Cox , A.L. : Investigating the information-seeking behaviour of academic lawyers : From Ellis ' s model to design . Information Processing & Management 44 ( 2 ) ( 2008 ) 613 { 634

[28] Micarelli , A. , Gasparetti , F. , Sciarrone , F. , Gauch , S. : Personalized search on the world wide web . The Adaptive Web 4321 ( 2007 ) 195 { 230

[29] Micarelli , A. , Sciarrone , F. : Anatomy and empirical evaluation of an adaptive web-based information ltering system . User Modeling and User-Adapted Interaction 14 ( 2 ) ( 2004 ) 159 { 200

[30] Mihalcea , R. , Radev , D. : Graph-based natural language processing and information retrieval . Cambridge University Press ( 2011 )

[31] Mihalcea , R. , Tarau , P.: TextRank: Bringing Order into Text . In: EMNLP . Volume 4 . ( 2004 ) 404 { 411

[32] Mittelstadt , B. : Auditing for Transparency in Content Personalization Systems . International Journal of Communication 10 (June) ( 2016 ) 4991 { 5002

[33] Opsahl , T. , Agneessens , F. , Skvoretz , J.: Node centrality in weighted networks: Generalizing degree and shortest paths . Social networks 32(3) ( 2010 ) 245 { 251

[34] Paul , C. , Rettinger , A. , Mogadala , A. , Knoblock , C.A. , Szekely , P.: E cient graph-based document similarity . In: International Semantic Web Conference , Springer ( 2016 ) 334 { 349

[35] Sappelli , M. : Knowledge Work in Context. User Centered Knowledge Worker Support . PhD thesis , Radboud University Nijmegen ( 2016 )

[36] Sappelli , M. , Verberne , S. , Kraaij , W. : Evaluation of context-aware recommendation systems for information re- nding . Journal of the Association for Information Science and Technology 68 ( 4 ) ( 2017 ) 895 { 910

[37] Sarrafzadeh , B. , Vechtomova , O. , Jokic , V. : Exploring knowledge graphs for exploratory search . In: Proceedings of the 5th Information Interaction in Context Symposium on - IIiX ' 14 . ( 2014 ) 135 { 144

[38] Sarrafzadeh , B. , Vtyurina , A. , Lank , E. , Vechtomova , O. : Knowledge Graphs versus Hierarchies . In: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval - CHIIR '16 . ( 2016 ) 91 { 100

[39] Speretta , M. , Gauch , S. : Personalized search based on user search histories . In: Web Intelligence , 2005 . Proceedings. The 2005 IEEE/WIC/ACM International Conference on, Ieee ( 2005 ) 622 { 628

[40] Sun , Y. , Li , H. , Councill , I.G. , Huang , J. , Lee , W.C. , Giles , C.L. : Personalized ranking for digital libraries based on log analysis . In: Proceedings of the 10th ACM workshop on Web information and data management , ACM ( 2008 ) 133 { 140

[41] Tang , J. , Yao , L. , Zhang , D. , Zhang, J.: A combination approach to web user pro ling. ACM Transactions on Knowledge Discovery from Data (TKDD) 5(1) ( 2010 ) 2

[42] Tanudjaja , F. , Mui , L. : Persona: A contextualized and personalized web search . In: System Sciences , 2002 . HICSS. Proceedings of the 35th Annual Hawaii International Conference on, IEEE ( 2002 ) 1232 { 1240

[43] Teevan , J. , Dumais , S.T. , Horvitz , E.: Personalizing search via automated analysis of interests and activities . In: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval , ACM ( 2005 ) 449 { 456

[44] Verberne , S. , Boves , L., van den Bosch, A.: Information access in the art history domain: Evaluating a federated search engine for Rembrandt research . Digital Humanities Quarterly 010 ( 4 ) ( 2016 )

[45] Verberne , S. , Kraaij , W., de Vries, A.P. : preprint: 1804 . 11131 . ( 2018 ) Author-topic pro les for academic search .

[46] Verberne , S. , Sappelli , M. , Kalervo , J. , Jarvelin, K. , Kraaij , W.: User simulations for interactive search: Evaluating personalized query suggestion . In: Advances in Information Retrieval . Volume 9022 ., Springer International Publishing ( 2015 ) 678 { 690

[47] Xu , Y. , Wang , K. , Zhang , B. , Chen , Z. : Privacy-enhancing personalized web search . In: Proceedings of the 16th international conference on World Wide Web, ACM ( 2007 ) 591 { 600

[48] Zhou , D. , Lawless , S. , Wade , V. : Improving search via personalized query expansion using social media . Information retrieval 15 ( 3-4 ) ( 2012 ) 218 { 242