Exploring Relevance in Microblog Search (Extended Abstract)

Abstract

Microblog search has some important diferences with respect to Web search, both in objectives and intents, and in the format and content of microblogs, e.g tweets. Properties such as credibility, informativeness and interestingness of microblogs could be considered as relevance dimensions to be assessed during query processing. In this paper we propose an approach aimed to investigate the association of diferent relevance dimensions with distinct search tasks in Microblog search. We hypothesize that the importance of a relevance dimension can be diferent, depending on the considered task. The preliminary results shows that the consideration of diferent relevance dimensions has a diferent impact on diferent search task.

CCS CONCEPTS

• Information systems → Evaluation of retrieval results; Relevance assessment; Web and social media search.

INTRODUCTION

Nowadays, more and more people rely on social media to freely express their opinions and to gather information relevant to their needs. Based on the report from the Pew Research Center, more than half of US adults use social media to fulfill their information needs 1; for example, users who consume news using Twitter grew up to 15% between 2016 to 2017. In [ 8 ], the authors argue that search in Microblogs such as Twitter is diferent from Web search, specifically regarding the users’ search behaviour and intents. In information retrieval, not all topically-related information are relevant to the users’ information need expressed in a query. The concept of multidimensional relevance is widely used in the context of Web search, which implies that several relevance dimensions should be considered to assess the utility of an information item to a user query. In [ 1 ], the authors conclude that interestingness is an important relevance dimension in Microblog search; they also outline that users consider less interesting the personal tweets generated from social interactions between users. Other research contributions employed diferent quality indicators of User-Generated Content, 2 Microblog search is mainly based on topicality to estimate relevance; however, several studies introduce more relevance dimensions that indicate the quality of microblogs to provide users with more useful information. They include informativeness, interestingness, credibility, and opinionatedness. The relation between relevance and informativeness has been studied in [ 2 ], where the authors employ human annotations to assess whether a microblog is relevant or not, and if it is informative or not. The reported results show that informativeness and relevance are correlated. Interestingness can also be a signal of the quality of microblogs, as outlined in [ 1 ]. In this study, an interestingness score is computed by exploiting several Microblog features including the probability of a tweet of being retweeted. Another quality indicator of UGC that has been considered in previous research is credibility [ 7 ]. The authors adopt a state of the art approach to assess the credibility of a blog post. Another interesting relevance criterion is opiniontedness, which constitutes a relevance dimension related to the opinion retrieval task [ 3–5 ]. In [ 4 ], the authors have defined lexicon-based approaches to estimate the opinionatedness of microblogs. While in [ 3 ] the authors considered the topic and the stylistic variation of the tweet in the opinion retrieval task. Most of these works do not consider the relation between search task and relevance dimensions in their retrieval systems. They compare the obtained search results with topicality assessment as a baseline (e.g BM25), and it is not clear which combination of relevance dimensions can achieve better retrieval results. The aim of our research is to study the correlation between relevance dimensions and search tasks. 3

PROPOSED METHOD

The hypothesis at the basis of the research shortly reported in this paper is that distinct search tasks can be related to the assessment of distinct relevance dimensions. To tackle this issue, an incremental approach is adopted. Topicality is assumed to be the basic relevance criterion for all considered tasks; the topical relevance of each document is assessed by using a standard retrieval model, namely the query likelihood model, and it is used as the baseline. Relevance Dimensions We consider four additional relevance dimensions namely credibility, informativeness, opinionatedness, interestingness. In this preliminary study, for each of the considered tasks, we combine each of the additional relevance dimensions with topicality only, to get the final Retrieval Status Value (RSV) of a document; in the future we aim to consider and combine more relevance dimensions. To combine the relevance scores associated with the two considered dimensions (topicality plus one of the four above mentioned dimensions), we use a linear combination RSV = αRSVt + (1 − α )RSVi , where RSVt is the topicality score, RSVi is the score assessed for the additional relevance dimension, and α denotes a weight between 0 and 1. As it has been done in the literature, we compare the overall RSV with the topicality score only as a baseline, to see the impact of the additional dimension to the retrieval result. 4

EXPERIMENTS AND RESULTS

For the experiments we have used diferent data collections for diferent search tasks. We utilize the data collection from the First International Workshop on Exploitation of Social Media for Emergency Relief and Preparedness (SMERP) 2017 3 for the disasterrelated retrieval task, and the data proposed by [ 4 ] for the opinion retrieval task. The disaster-related dataset contains five topics and the opinion retrieval dataset consists of 50 topics. We adopt the approach of [ 6 ] to calculate the informativeness score, based on logistic regression; we use the predicted probability as the informativeness value of a tweet. The opinionatedness score is obtained by using the approach proposed in [ 3 ]. The opinionatedness score is calculated not only based on term-based opinion score but also based on the topic and the stylistic variations of tweets. We build the proposed ranking system based on Apache Lucene4. In this experiment, we use MAP to compute the performance of the retrieval system. We report here the results related to two search tasks, namely the Disaster-related retrieval task and the Opinion retrieval task. Fig.1 shows the results of the experiment for each of the two considered tasks by comparing MAP values of the system combining two relevance dimensions with the system based on topicality (the baseline). The y-axis shows the MAP score and the x-axis represents the weight of the topicality score in the linear combination (RSVt ). Fig. 1 illustrates how the retrieval results change according to the weight of the topicality dimension. The upper part of Figure 1 shows the behavior of the system using informativeness as an additional relevance dimension in the disaster-related task. The bottom part of Figure 1 shows the behavior of the system in the opinion retrieval task; it can be seen that opinionatedness can improve the results of the retrieval system with increasing weights of topicality in the linear combination.

CONCLUSIONS

In this work, we study the correlation between multidimensional relevance criteria and search tasks in Microblog search. Besides topicality we aim to use four additional relevance dimensions and we consider three search tasks. In future, we plan to implement all relevance dimensions and to evaluate the system using more data collections related to diferent search task. We also intend to study and model the interactions between relevance dimensions.

References

[1] Arifah Che Alhadi , Thomas Gottron, Jérôme Kunegis, and Nasir Naveed . 2011 . LiveTweet: Microblog Retrieval Based on Interestingness and an Adaptation of the Vector Space Model. . In TREC.

[2] Jaeho Choi , W Bruce Croft , and Jin Young Kim. 2012 . Quality models for microblog retrieval . In Proceedings of the 21st ACM international conference on Information and knowledge management. ACM , 1834 - 1838 .

[3] Anastasia Giachanou , Morgan Harvey, and Fabio Crestani . 2016 . Topic-specific stylistic variations for opinion retrieval on twitter . In European Conference on Information Retrieval . Springer, 466 - 478 .

[4] Zhunchen Luo , Miles Osborne, and Ting Wang . 2015 . An efective approach to tweets opinion retrieval . World Wide Web 18 , 3 ( 2015 ), 545 - 566 .

[5] Zhunchen Luo , Jintao Tang , and Ting Wang . 2013 . Propagated opinion retrieval in twitter . In International Conference on Web Information Systems Engineering . Springer, 16 - 28 .

[6] Debanjan Mahata , John R Talburt, and Vivek Kumar Singh. 2015 . From chirps to whistles: discovering event-specific informative content from Twitter . In Proceedings of the ACM web science conference. ACM , 17 .

[7] Kamran Massoudi , Manos Tsagkias, Maarten De Rijke, and Wouter Weerkamp . 2011 . Incorporating query expansion and quality indicators in searching microblog posts . In European Conference on Information Retrieval . Springer, 362 - 367 .

[8] Jaime Teevan , Daniel Ramage, and Merredith Ringel Morris. 2011 . # TwitterSearch: a comparison of microblog search and web search . In Proceedings of the fourth ACM international conference on Web search and data mining. ACM , 35 - 44 .