In this paper, we investigated how academic search can profit from personalization by incorporating query history and background knowledge in the ranking of the results. We implemented both techniques in a language modelling framework, using the Indri search engine. For our experiments, we used the iSearch data collection, a large corpus of documents from the physics domain together with 65 search topics from scientists and students. We found that it is possible to improve academic search by taking into account query history. However, we have not been able to prove that terms extracted from the user's background data can improve academic search.
Professional search is di↵erent from ad hoc search in several aspects: it takes place in a specific domain, information is to be found in di↵erent types of sources (articles, books and web pages), and the search tasks often have a high complexity. In this paper we focus on academic search, which is the professional search carried out by scientists in their daily work. Researchers need to find scientific literature to keep up to date with their field and as a source for writing their own papers.
From the point of view of the search engine, academic information seeking behaviour is often reflected by a sequence of queries, related to one or more specialized topics, intertwined with clicks on scientific papers, books and domain-specific web pages. The user performing the search is a professional in his or her field, which means that he or she has background knowledge on the topic. Given these characteristics of academic information seeking, we argue that the following information retrieval (IR) technologies should at least be integrated in a search engine for academic search: (1) Aggregated search (merging results from di↵erent types of sources); (2) IR over query sessions (exploiting query history to improve ranking); (3) Personalization based on background knowledge.
In this paper, we investigate how academic search can profit from incorporating (2) query history and (3) background knowledge in the ranking of the results. We implement both techniques in a language modelling framework, using the Indri search engine.1 For our experiments, we use the iSearch data collection [LLLI10], a large corpus of documents from the physics domain together with 65 search topics from scientists and students. The collection contains three types of documents (articles, books and metadata), which makes results aggregation necessary. However, the focus of our work is on personalization, not on aggregation. 2
In this section, we first summarize the work on personalized search and user profiling, which includes work on exploiting query history. Then we will introduce the iSearch collection (Section 2.2), which plays a central role in our work. There is a small body of work using the (relatively new) iSearch collection, which we discuss in Section 2.3. 2.1
Many approaches to content-based personalization represent user profiles as a list of terms from a reference ontology such as the Open Directory Project (ODP)2 or Yahoo! directory. For example, in the work by [PG99, GCP03, SG05], the user profile has the form of an ontology in which each term (e.g. the ODP term ‘support vector machines’) has a weight indicating the user’s interest in the concept represented by that term. User data such as previous queries or snippets of visited web pages are automatically mapped to the terminology of the ontology. The resulting profiles are in most studies used to re-rank the search results. Alternatively, in the work by [MPS07], the ontology-based approach is exploited to improve a categorization-based retrieval system for knowledge workers. In [PG99], an improvement of 8% in terms of 11-point precision is reported as an e↵ect of adding user profiles to result ranking. [SG05] report that the average rank of the results that were selected as relevant by the user improves with 33% by adding profiles that were extracted from query history.
A somewhat similar approach to user profiling is to classify user data in topical categories. In [LYM04], a category (e.g. the ODP category ‘Articifial Intelligence’) is defined as a set of terms with weights, which reflect the significance of the user’s interest in that category. In [LYM04], user profiles are learned from the user’s search history (queries and click data) and 12%–13% improvement over non-personalized results is obtained. In [QC06], a user’s preference is represented as a vector of topics (e.g. ‘Computers’) with weights. These user preferences are learned from click-history data. The authors implement a personalized search system that extends the PageRank function based on the topics of web pages and the user’s preferences for these topics. They find that their personalized method performs 25% to 33% better than Topic-Sensitive PageRank without personalization.
An alternative option for user profile learning is term extraction: extracting prominent terms from user data. These terms can then be used for re-ranking search results based on the similarity between the user profile and the retrieved documents (e.g. [MS04]), or for query modification. For example, in [CS98], the query is expanded with the terms from the user profile that are the most correlated to the terms in the query. In [SZ03], a language modelling approach to query modification is followed: a query model is created in which the terms from previous queries are weighted with the terms from the current query. Up to 52% improvement in terms of average precision is achieved when the user’s previous three queries are added to the model of the current query. In [STZ05], the current query is expanded with terms from previous queries and from the documents retrieved for those queries. The authors are able to improve over the Google ranking baseline and stress again that their method can improve existing web search performance without any additional e↵ort from the user.
In this paper, we follow a language modelling approach to personalization, incorporating the user’s query history and his background knowledge in a merged query model. 2.2
For our experiments on academic information seeking behaviour, we use the iSearch data [LLLI10]. The collection consists of 65 natural search tasks (topics) from 23 researchers and students from three di↵erent university departments of physics. The search task description form had five fields that the searchers filled in before they started to search for answers: • a) What are you looking for? (information need) • b) Why are you looking for this? (work task context) • c) What is your background knowledge of this topic? (knowledge state) • d) What should an ideal answer contain to solve your problem or task? (ideal information) • e) Which central search terms would you use to express your situation and information need? (search terms) A collection of 18K book records, 144K full text articles and 291K metadata records from the physics field is distributed together with the topics. For each topic, the developers used Indri to collect a pool of up to 200 documents from the collection and the topic owner (also called ‘user’ or ‘searcher’ in the remainder of this paper) assessed these documents on their relevance for the topic. Relevance assessments were made on a 4-point scale: highly relevant, fairly relevant, marginally relevant and non-relevant. The average number of topics per searcher is 2.8. 2.3
All previous work with the iSearch collection uses Indri as index and retrieval engine. As evaluation measure, normalized Discounted Cumulative Gain (nDCG) [JK02] is generally used, because of the graded relevance assessments in the data. An overview of the obtained results in the previous literature is in Table 1. [LKS11] compare two types of queries: ‘short queries’ (fields a and e together) and ‘long queries’ (all fields a, b, c, d and e together). The results are very similar, short queries giving slightly higher nDCG scores. [LLFS12] calculate the density of technological terms in the documents in the iSearch collection and boost the weight of technical terms in the query. They obtain a significant improvement over the simple retrieval baseline, but a smaller improvement than they achieve with pseudo-relevance feedback. Combining their boosting method with pseudo-relevance feedback gives a small improvement over using pseudo-relevance feedback only. [NdVA12] study the potential of contextualization by re-scoring the Indri result list using inlinks and outlinks of documents. They find that the context from in- and outlinks can help improve retrieval results, but not by a large margin. [SBL12] investigate what gives better results for a collection with di↵erent document types (abstracts, papers, books): combining all documents in one index or creating three separate indexes and apply fusion techniques on the result lists. They found no significant di↵erence between the two approaches. The best results so far were obtained by [LLI12] using fields a (information need) and b (work task context) from the topic. They reached an nDCG score of 0.3572. 3 3.1
In previous work with the iSearch collection, the terms from the search terms field (field e) are treated as if they are one single query, sometimes even extended with terms from other fields [LKS11]. Technically, this is not a problem because Indri is “truly best match”: not all search terms have to be present in a relevant document. However, we think that concatenating all search terms leads to queries that are unrealistically long for real information seeking behaviour: the average length of the search terms field is 9.4 words. We noticed that all searchers used punctuation (semicolon, comma or full stop) to structure the search terms field. The sequences of terms can be interpreted in di↵erent ways, and it may be the case that they should be interpreted di↵erently for di↵erent users: some searchers seem to have listed multiple facets of the same topic (e.g. “Nano spheres, beads, magnetic, sorting”); others seem to have listed multiple queries that they intended to issue subsequently (e.g. “Electrostatic Force Microscopy (EFM), protein-protein interaction, Avidin-Biotin”). In our inital (baseline) experiments, we consider the search terms field to be a sequence of queries that were entered within one session. Splitting the search terms field on the symbols [; , .] leads to an average of 3.5 queries per topic, with an average query length of 2.9 words. This way, we constructed 65 search sessions for 23 searchers with 3.5 queries per session. In Section 4.1 we investigate how we can use the sequence of query terms to improve result ranking.
We indexed the iSearch collection using Indri. We created three separate indexes: one for the 18K book records (BK), one for the 144K full text articles (PF) and one for the 291K metadata records (PN). We did not apply stemming and did not remove stop words. 3.2
We used Indri’s Language Modelling retrieval function for our retrieval experiments. We found that for our data, Dirichlet smoothing gives better results than Jelinek-Mercer smoothing. Therefore we applied Dirichlet smoothing in our experiments:
Here, D represents the document, t is a query term, C represents the collection and |D| is the number of words in the document. tft,D is the term frequency of t in D. The smoothing factor µ is equal to 2500.
We also compare our results to the pseudo-relevance feedback method that has been implemented in Indri. We tune the parameters for this method using the same grid as [LLFS12]: the number of feedback documents f bDocs = [1, 2, 5, 10, 20] and the number of feedback terms f bT erms = [3, 5, 10, 20, 40]. We optimized these parameters for each index separately.
We retrieved a maximum of 1000 results per query, after finding that recall keeps improving when increasing the maximum number of results from 100 up to 1000. We evaluate our results using several di↵erent measures, of which the most important are recall and normalized Discounted Cumulative Gain (nDCG) [JK02]. 3.3
We follow the collection fusion approach described by [SBL12] for merging the results from the three indexes BK (books), PF (articles) and PN (metadata). For each query, we retrieved 1000 results from each of the indexes. We merged the result lists by first normalizing each Indri retrieval score relative to the minimum and maximum scores for that index: scorenorm = scoreoriginal scoremax
scoremin scoremin
We then ranked all documents that are retrieved from the three indexes for a given query by their normalized scores, yielding a combined result list. 4
The iSearch data give us two di↵erent oppurtunities for personalizing search results. The first method is by using the user’s query history (Section 4.1) and the second is by using the user’s background knowledge as described in their own words (Section 4.2). 4.1
We implemented three di↵erent methods for exploiting previous queries within a session: (A) Simple concatenation of all queries in the session into one query, thereby simulating a form of query refinement where the initial query is extended with more terms subsequently3; (B) Combining results from multiple queries by summing the retrieval scores for each document over all queries for which the document is retrieved, and then rank the documents by their combination scores; (C) Combining query models following the language modelling approach proposed by [SZ03]. In the latter approach, a maximum likelihood estimation is applied to create a unigram language model for each session, assuming a bag of words. This way, a query language model is estimated in 3This method makes it possible to compare our results to results in previous work using the iSearch data, in which all search terms were concatenated into one query. (1) (2) which the terms from all queries from the current session are combined and weighted by the length of the query they occur in:
Here, c(t, qi) is the number of occurrences of term t in query qi and |qi| is the length of query qi. k is the number of queries in the session. We used the weight operator in the Indri query language to retrieve results for the merged query models: for each session, a query is constructed of the form #weight(w1 t1 w2 t2 ... wn tn) and submitted to Indri. We evaluate the results per topic (session). 4.2
The average number of topics per searcher in the iSearch data is 2.8. For each topic, the searcher formulated a short text describing his or her background knowledge on the topic. We hypothesize that this background knowledge can be a valuable source for personalization. We exploited the background knowledge descriptions to create profiles of the searchers. In a real information seeking situation, the searcher would not have provided the system with descriptions of his/her background knowledge. Instead, a search system that aims at personalization of retrieval results would use previously accessed documents (online or on the searcher’s harddisk) to build a user profile. We therefore consider the background knowledge descriptions as approximations of bigger collections of user data. For modelling the user’s knowledge, we concatenated the background knowledge fields from all topics belonging to one searcher. This results in a set of 24 text documents with an average of 328 words per searcher.4
We extracted a user profile from the background knowledge documents in the form of a list of prominent n-grams (n = 1, 2, 3). ‘Prominence’ is determined using the language modelling approach described by [TH03].5 This method needs a background corpus for determining the prominence of terms in the document. We chose the Corpus of Contemporary American English as background corpus because the developers provide a word frequency list and n-gram frequency lists that are free to download.6 Note that the same approach could be applied to bigger user collections. The main di↵erence is that the background knowledge descriptions in the iSearch data are relatively sparse because of their limited size. Table 2 shows excerpts from the term lists extracted for the three first searchers in the iSearch data (author ids 085, 086 and 087).
According to [MGSG07], the user profile can a↵ect the search process in three phases: (1) as part of retrieval process (web pages have already been scored according to the user profile before the search takes place); (2) in a re-ranking step (the user profile is used to re-score the documents that have been retrieved using a nonpersonalized retrieval step); (3) in a query modification step (where the user profile is used to adapt the user’s query, after which retrieval and ranking is performed as usual). We chose the third of these possibilities because 4We are aware of the fact that the topic-specific background knowledge field might give better results but using this is somewhat less realistic because in reality the searcher does not provide a separate background model for each search topic.
5We only used the ‘informativeness’ criterion from this paper, excluding the ‘phraseness’ factor from the model because it heavily penalizes unigram terms, while we are interested in unigrams as well as multiword terms.
6http://www.wordfrequency.info it is more ecient than the first option (which requires that a similarity score between the user profile and each document in the collection is calculated before the search starts); we plan to experiment with the second method (re-ranking) in future work.
We incorporated the user profiles in the query results by creating a merged query model, incorporating the top-k (k = 10 for the current experiments) terms from the user profile. For the purpose of weighting the terms in the user profile, we used a variant of equation 3, but instead of using the relative counts of each term in all k queries ( P c(t,qi) ), we used the prominence scores from [TH03].
i=1 |qi| 1 k p(t|uk) = prom(ti, u) (4)
Here, k (which was the number of queries in Equation 3) is the number of terms from the user profile that is incorporated in the query. We again used the weight operator in the Indri query language to retrieve results for the merged query model mixed with the user profile model: for each session, a query is constructed of the form “#weight( w1 t1 w2 t2 ... wn tn )”, where a term ti can be a term from the query model or from the user model. 5
The results that we obtained in the retrieval experiments with use of query history are in Table 3. Note that the results for pseudo-relevance feedback were obtained with optimal parameter settings on these data; we did not held out a separate tuning set. The results for personalization with user profiles are in Table 4. In this paper, we investigated how academic search can profit from personalization. We experiment with language modelling methods to incorporate the user’s query history and background knowledge in the ranking of the results.
We aggregated the result lists from three di↵erent indexes (books, articles and metadata) in one result list. Using the Indri ranking model, we first obtained a baseline nDCG score of 0.231 and using pseudo-relevance feedback an nDCG score of 0.249. Second, we implemented three methods for incorporating query history in the results. With the simplest method (all query terms concatenated) we replicated the results reported in previous research on the same data. The second method, merging the result lists for the subsequent queries after retrieval, yielded the highest recall (63.4%), while the third method (merging query models) resulted in the highest nDCG score (0.364).
Third, we created user profiles by extracting prominent terms from the background knowledge texts formulated by the searchers. We incorporated these profiles in the query model. Unfortunately, we were not able to improve over the results that we obtained without personalization; the highest nDCG that we achieved on the combined indexes was 0.366. We suspect that the sparseness of the user profiles plays an important role. Therefore, in the future, we would like to re-run the experiments with user profiles that are based on more data. In addition, we will experiment with di↵erent methods for incorporating the user profile in the result list.
In conclusion, we found that it is possible to improve academic search by taking into account query history. However, we have not been able to prove that terms extracted from the user’s background data can improve academic search. More work is needed, and planned, in this direction. 7
This publication was supported by the Dutch national program COMMIT (project P7 SWELL). [CS98] BK PN Combined method Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Peudo-relevance feedback Baseline (Dirichlet smoothing) Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Pseudo-relevance feedback Baseline (Dirichlet smoothing) Baseline (Dirichlet smoothing) Pseudo-relevance feedback session aggregation none none A. queries concatenated A. queries concatenated B. merged query results C. merged query models C. merged query models none none A. queries concatenated A. queries concatenated B. merged query results C. merged query models C. merged query models none none A. queries concatenated A. queries concatenated B. merged query results C. merged query models C. merged query models none none A. queries concatenated A. queries concatenated B. merged query results C. merged query models C. merged query models
C. Lioma, A. Kothari, and H. Schuetze. Sense discrimination for physics retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 1101–1102. ACM, 2011. [LLFS12] B. Larsen, C. Lioma, I. Frommholz, and H. Schu¨tze. Preliminary study of technical terminology for the retrieval of scientific book metadata records. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 1131–1132. ACM, 2012. [LLI12] [LLLI10]
Christina Lioma, Birger Larsen, and Peter Ingwersen. Preliminary experiments using subjective logic for the polyrepresentation of information needs. In Proceedings of the 4th Information Interaction in Context Symposium, pages 174–183. ACM, 2012.
Marianne Lykke, Birger Larsen, Haakon Lund, and Peter Ingwersen. Developing a test collection for the evaluation of integrated search. In Cathal Gurrin, Yulan He, Gabriella Kazai, Udo Kruschwitz, Suzanne Little, Thomas Roelleke, Stefan Rger, and Keith van Rijsbergen, editors, Advances in Information Retrieval, volume 5993 of Lecture Notes in Computer Science, pages 627–630. Springer Berlin / Heidelberg, 2010. [LYM04] [MS04] [PG99] [QC06] [SBL12] [SG05] [STZ05] [SZ03] [TH03]
F. Liu, C. Yu, and W. Meng. Personalized web search for improving retrieval e↵ectiveness. Knowledge and Data Engineering, IEEE Transactions on, 16(1):28–40, 2004. BK PN personalization method none (merged query model) merged query model with user background model none (merged query model) merged query model with user background model none (merged query model) merged query model with user background model none (merged query model) merged query model with user background model