Our research aim is twofold: the rst one is to get leverage from the relevance feedback that the user provides over the course of every search session. In this way, we explore PIR task as a text categorization problem in order to distinguish between relevant and not relevant documents for the given user. A number of supervised machine learning algorithms has been applied in order to accomplish this task. The second one is related to implicit feedback. More concisely, time spent reading documents and text complexity. From the point of view of text complexity, we propose the hypothesis that there are signi cant di erences of perplexity between judged/non-judged documents by the user. We nd some weak statistical evidence that points out that high perplexity and judged documents are correlated.
Information Retrieval (IR) is a discipline that involves the retrieval of certain
information within a document collection based on speci c information needs[
The subject of this paper concerns task 2 of the PIR-CLEF laboratory. The challenge in this work is to use the les provided by the organization (csv1 csv5) to create user pro les that can be used to improve the quality of the results provided by an IR System. Both explicit and implicit data can be found in these les, although certain characteristics can be inferred from the former.
This entire process involves the development of a Personalized Information Retrieval system (PIR) based on a number of explicit and implicit feedbacks such as user document relevance assessments, search logs including timestamps of every action, and ranking of documents as a result of every search performed by the user. 2
Two di erent types of input data sources are usually identi ed in the
scienti c literature: implicit and explicit feedback [
When implicit feedback is used for the development of this type of system, there may be uncertainty when interpreting the results. To this end, an attempt will be made to nd some kind of correlation between the implicitly inferred information and the explicit evaluation given by the user. 3
In this section, we present the di erent actions that we have accomplished in our participation in CLEF PIR 2018 Task 2 Evaluation of Personalised Information Retrieval. 3.1
Firstly, data acquisition has been accomplished to retrieve the Clueweb documents that each user evaluated as a result of the execution of each of them [11]. For this rst step, the attribute \query text" belonging to the le csv2.csv provided by the organization is used. A query result is made of the rst one hundred documents obtained using the API provided by the organisation and developed by the University of Dublin1, each of which contains an attribute called "id", which is used to download the corresponding web page using page rendering of Clueweb12 data-set online services2.
Secondly, once the corresponding web pages are downloaded we move on to the pre-processing phase: - Only the text contained in the labels \title" and \p" (paragraph) has been considered. - The resulting text has been processed by eliminating stop words and by executing the Porter stemming algorithm. 1 http://clueweb.adaptcentre.ie/WebSearcher/search?query=\query
String"&selection=[selection numbers separated by comma] (last visited:
30/05/2018)
2 https://www.lemurproject.org/clueweb12/services.php (last visited: 30/05/2018)
- Every document is represented as a set of tokens made by unigrams, bigrams
and trigrams.
- TF.IDF has been calculated for each token belonging to each of the di erent
user categories [
Relevance Feedback is a well-known technique in the eld of information
retrieval. The idea behind relevance feedback is to take the results that are initially
returned from a given query, to gather user feedback, and to use information
about whether or not those results are relevant to perform a new query [12].
On the other hand, methods based on supervised machine learning are the most
frequent approach to accomplish the task of text Categorization [
As a consequence, a number of supervised machine learning algorithms have
been trained using the user relevance judgments, characterizing each of the
documents based on sequences of n grams (Unigrams, Bigrams and Trigrams)
extracted from the texts and using the bag of words method, which allows us to
represent documents ignoring the order of the words that make it up.
3.2.1 Results. In order to evaluate our approach, we have followed a k-fold
cross-validation approach with k = 10 [
The results obtained in Table 1 represent the success rates of the classi ers in the test phase. These data must be analyzed from the perspective of the problems presented by the classi cation algorithms used. In this case, not all users present su cient or su ciently good data to con rm the results presented.
For experiments with 4 categories (1-2-3 and 4), 10-fold cross-validation can only be accomplished for user 02, user 11, user 15 and user 07 in the Books topic only. This is because many of the samples provided by the organization are not representative enough to perform cross-validation up to 10 folds. In general, it can be said that, when it is possible to de ne 10-fold, explicit feedback gets systematically results whose performance improvement (47% on average) is statistically signi cant when considering random values as case base. For the rest of the users, the results presented have been obtained by applying a smaller number of folds because some of the samples are unbalanced.
In the case of coarse-grained experiments, with 2 categories, it has not been possible to balance the evaluation values in any of the cases, the reference values taken to check the result of the classi ers remain the same as in the case of 4 categories. It can be seen that all cases the success rates of the classi ers increase by around 15% with respect to ne-grained results.
The best result is obtained by user 17 and Books as topic. The reason is that this case is extremely unbalanced: 128 of a total of 133 documents are judged as o -topic or not relevant.
Regarding the best classi er, it is not possible to select the best one since this depends on the user and topic. This suggests that a voting strategy could be a good option as future work. 3.3
Consider our rst approximation to Personal Information Retrieval, we have
focused on two di erent issues regarding implicit feedback [
Suddenly, there is no way to be sure about the time spent on a page just by using the search logs provided by the PIR-CLEF organization since there is no timestamps about close document events. Even though this drawback we have tried to accomplish a study to correlate timing and user assessments. Consequently, we interpreted the time spent on a page as the di erence between two consecutive open document events, start session and open document event or open document and end session event. The next step we tested is whether this sequence of time periods follows a normal sequence. Thus, we apply a ShapiroWilk test for each user search log. In addition, we eliminated time periods that are out the two central percentiles 25, with the aim of removing anomalous time periods. We nd that the most of time distributions are not normal even though ltering anomalous data. Finally, we applied a two tailed Student T-Test for those cases where normality is found. For the rest of cases, we applied MannWhitney-Wilcoxon.
In the end of this process we have to conclude that, in general and following
the procedure described above, it is not possible to reject the null hypothesis:
user document judgments and time spent on a page are correlated.
3.3.2 Investigating the relationship between language model
perplexity and user relevance measures. The canonical measure of goodness
of a statistical language model is normally reported in terms of perplexity: the
exponential of the negative normalized predictive likelihood under the model,
and gives an indication of the expected word error rate as in speech recognizers
[
We used trigram language models with interpolated Kneser-Kney discounting trained using the SRI language modeling toolkit [14]. We generated di erent models by varying the training corpus.
{ Simple-wiki [
The perplexity of three sets of documents per each user query was measured: the set of relevant documents (user relevance judgment is 3 or 4), non-relevant documents (user relevance judgment is 1 or 2) and unjudged documents (there is no user relevance judgment in spite of they are part of the ranked list of documents retrieved)
Finally, an one-tailed Mann-Whitney-Wilcoxon (MWW) test was performed4. When language models based on Simple-wiki and ClueWeb12 search datasets are applied we have no found any signi cative di erence between the perplexity of the the three set of documents considered (relevant, non-relevant or unjudged) When Sphinkx-70k is used to train the language model, we nd some evidence that the complexity of judged documents (relevant or not relevant) are greater than those that are unjudged (U-value=59, critical U-value at p<0,05=51). This is quite surprising since it could be interpreted in the way that the user tends to evaluate the most complex texts. Once we revise some of the non-judged document we nd that it is quite frequent that these documents do not have textual content at all, only lists of sections, menus and stylesheets but no or very little meaningful text. 4
In this work, an overview has been given of the use of explicit and implicit feedback for the generation of user pro les. In the case of explicit feedback, a classi cation system is trained based on the user's judgements of relevance provided for a given set of documents. In the case of implicit feedback, the intention is to seek a correlation between the information that can be inferred from the data and the relevance judgments provided by users, so that user pro les and system accuracy can be improved.
In future work it would be interesting to have information relevant to certain aspects. The le csv6.csv provided by the organization, shows the TF.IDF 3 https://sourceforge.net/projects/cmusphinx/ les/Acoustic and Language Models/US 4 When the dataset is small, the P-Value from t-Student is likely to be the most usual test but it requires a normal distribution of the dataset. For this reason, we applied Shapiro-Wilk test that is suited for small datasets and we found that it is not always possible to assert that the considered datasets follow a normal distribution. As a consequence we applied a non-parametric test, the Mann-Whitney-Wilcoxon U test values of the tokenized terms, it would be interesting to be able to relate each of these terms with the document to which it belongs, in this way, they could be used for the realization of the training phase characterization of web pages.
On the other hand, in order to search for a correlation between the interest that an user has in a web page and the time he spends visiting it, it would be interesting to have the \CLOSE DOCUMENT" attribute of all the records. Currently there are only 5 records with this attribute in the csv2.csv le.
In order to further deepen this task, it would be helpful to have the traditional expert judgements of relevance. In this way, relationships could be sought in those cases in which the user did not agree with the expert and try to discern the most probable cause. 5
Work supported by a grant from the Ministry of Education, Culture and Sport (MECD-Scholarship BES-2016-076609) and the REDES project (TIN2015-65136C2-1-R) of the Spanish Government. 11. Pasi, G., Jones, G.J., Marrara, S., Sanvitto, C., Ganguly, D., Sen, P.: Evaluation of personalised information retrieval at clef 2017 (pir-clef): towards a reproducible evaluation framework for pir 12. Salton, G., Buckley, C.: Improving retrieval performance by relevance feedback.
Journal of the American society for information science 41(4), 288{297 (1990) 13. Sebastiani, F.: Machine learning in automated text categorization. ACM computing surveys (CSUR) 34(1), 1{47 (2002) 14. Stolcke, A.: Srilm-an extensible language modeling toolkit. In: Seventh international conference on spoken language processing (2002)