Active Tweet Recommendation Based on User Interest Profiles ∗ Raheleh Makki Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada niri@cs.dal.ca ABSTRACT With the constant growth of blogs, forums, and online The rapid growth of Twitter has made it one of the most social networks, the need for information retrieval systems, popular information sources of current affairs. Twitter users recommenders and analytical tools increases. On the other gather information about their topics of interest through hand, existing standard techniques may be inadequate to their followees’ posts or by searching for relevant posts. How- overcome new challenges introduced by particular charac- ever, users are often overwhelmed by the large number of teristics of the data generated from these sources. Online tweets which makes it difficult for them to find relevant and chats, tweets, blogs and any other data sources with infor- non-redundant information about their interests. Informa- mal settings contain short texts riddled with spelling errors, tion filtering and recommender systems can help users by incorrect grammar, acronyms and non-standard words. In suggesting informative tweets based on their interests. Con- addition, methods for processing and analyzing these type of sidering the wide variety of topics in users’ interest profiles messages should be ideally real-time since they are usually and the sheer volume of tweets being published daily, it is only interesting during short time after being published. difficult to have adequate and proper labeled data to train Twitter has become one of the most popular social media these systems on. We aim to tackle this problem by inte- platforms with vast and various number of discussed topics grating active learning techniques into tweet recommenda- and shared information. Many users seek and collect infor- tion, more specifically for finding relevant tweets and rank- mation about their areas of interest from Twitter. How- ing them. Using active learning methods in the context of ever, considering the sheer volume of tweets published daily Twitter recommenders has not been well explored before. (around 500 million tweets), users can be overwhelmed by Our objective is to exploit these methods for improving the large amounts of tweets from their followees [10]. Informa- accuracy of tweet recommenders the most, while keeping the tion filtering and recommender systems aim to find useful cost of labeling to a minimum. and novel information based on the user’s preferred topics and suggest them in a ranked order. Since users can have di- verse interests, each changing over time, and also as current CCS Concepts events are rapidly added to the vast set of topics of discus- •Information systems → Recommender systems; sion on Twitter [2], it is not appropriate to use old data for training recommenders on. In addition, considering the cost of labeling and the real-time nature of Twitter, it would be Keywords infeasible to label large quantities of data for every user in Social recommender, active learning, Twitter, information different time spans. In this Ph.D., we aim to introduce and filtering apply active learning strategies, which select a number of instances to be labeled by an information source, in order 1. INTRODUCTION to improve the accuracy of the recommender systems while minimizing the labeling cost. ∗Research presented here has been jointly done with Dr. Axel J. Soto (National Centre for Text Mining, School of Computer Science, University of Manchester, UK. 2. PROPOSED METHOD Email:axel.soto@manchester.ac.uk) and my Ph.D. thesis su- We propose to integrate active learning strategies with pervisors Prof. Stephen Brooks and Prof. Evangelos E. Mil- ios (Faculty of Computer Science, Dalhousie University, Hal- learning-to-rank and information filtering methods. Our ifax, NS, Canada. Email:sbrooks@cs.dal.ca, eem@cs.dal.ca). framework is presented in Figure 1, which shows the use of active learning in both tweet relevance model and rank- ing model of our recommender system. We briefly discuss each component in this section. We assume that users specify the gist of their interest in a few keywords. Therefore, we formulate a query from these keywords and summarize our recommendation task as “At time T , give a ranked list of most relevant and novel tweets about query Q”. We consider the language model-based re- trieval using Dirichlet smoothing [13] as our baseline rele- vance model. This model retrieves a list of tweets ranked by user interest in a particular day, the recommender system should remain silent (achieving nDCG = 1) for that day, otherwise any tweet recommendation penalizes the system (nDCG = 0). The focus of the evaluation metric on the precision of the recommended tweets requires including only tweets the sys- tem is confident about. Considering a threshold and dis- carding tweets with relevance score less than that thresh- old is one possible way. However, not having labeled data makes it difficult to tune this parameter for different users with various interests. Therefore, we consider the most basic strategy for selecting tweets for labeling (i.e. to be labeled by the information source), which is to select only the top recommended tweet for each user profile per day. If the top selected tweet is not relevant to the user interest, we discard all selected tweets for that day. Figure 1: Framework of the active tweet recommen- The results of applying this simple strategy to the rele- dation system. vance model is reported in Table 1. It also shows the re- sults of top 3 teams that participated in TRECMicro, which indicates the potential of active learning techniques in im- their relevance to the search query. Verifying the relevance proving the performance of the tweet recommender systems. of retrieved tweets by active learning strategies can improve TRECMicro categorized the systems based on the amount the accuracy of tweet relevance model (see Section 3 for of human involvement into three different categories: auto- some preliminary results). To take into account the novelty matic with no human input, manual preparation with hu- of the recommended tweets, we cluster the retrieved tweets man input only before the evaluation starts and manual in- and only include the tweet that was published earlier within tervention with human input all the time. Our proposed each cluster in the final recommendation list [8]. We con- approach using active learning falls under the category of sider these sequence of steps as our baseline recommender manual intervention. system. In the baseline model, we rank retrieved tweets based on their relevance score by the language model. It has been Table 1: nDCG@10 of the proposed method and top shown that social attributes such as the number of retweets, 3 TRECMicro participants. URLs or hashtags in the text of tweets, or followers of the Run(Group) nDCG@10 Type tweets’ authors contain information about the credibility of tweets [6] and should also be considered in ranking of tweets. Proposed Method 0.4371 manual intervention Having a set of features, learning-to-rank methods have been SNACS LB(NUDTSNA) 0.3670 manual used for constructing ranking models for tweet recommen- SNACS(NUDTSNA) 0.3345 manual dation [3, 4, 14, 9]. However, these methods need labeled CLIP-B-0.6(CLIP) 0.2491 automatic data for training the ranking model. Assuming the avail- Baseline Recommender 0.2271 automatic ability of human labels or limited labeled data, active learn- ing and semi-supervised techniques focus on selecting the best instances to be considered in the training data. Us- ing these techniques in recommender systems in general has 4. FUTURE PLAN been studied before [7, 11, 5, 1]. However, adapting these To overcome some of the challenges related to real-time strategies for Twitter message recommendation has not been tweet recommender systems, we plan to introduce and ap- explored much to the best of our knowledge. ply active learning strategies. These strategies select the instances to be labeled by an information source in a way that the quality of the recommended tweets improves the 3. DATASET AND INITIAL EVALUATION most. Initial analysis, reviewing state-of-the-art methods, In order to study how user involvement can improve the and preliminary experiments has already been done. Our performance of our tweet recommender system, we performed future plan for the next year is to integrate active learning some preliminary experiments using the TREC 2015 Mi- into different retrieval and ranking models and investigate croblog track (TRECMicro) dataset [8]. We collected tweets their effectiveness in improving the results of our baseline during the evaluation period of the TRECMicro using the recommender system. In addition, we would like to analyze Twitter’s streaming API. We gathered 40,264,332 tweets, the effect of different groups of features including content- out of which only English tweets (16,302,498) have been used based features and social attributes on the accuracy of tweet in our experiments. We consider the judgment scores pro- relevance and ranking models. vided by the TRECMicro and the normalized Discounted Cumulative Gain (nDCG) [12] as our gold standard and evaluation metric respectively. nDCG is calculated over k 5. ACKNOWLEDGMENTS (k = 10 here) top suggested tweets for each user profile per This research was funded by the Natural Sciences and En- day. The final value is the average over the evaluation period gineering Research Council of Canada (NSERC) and The and all 51 user interest profiles (also provided by TRECMi- Boeing Company. cro). In cases where there are no relevant tweets to the 6. REFERENCES [1] L. Blédaité and F. Ricci. Pairwise preferences elicitation and exploitation for conversational collaborative filtering. In Proceedings of the 26th ACM Conference on Hypertext & Social Media, pages 231–236, 2015. [2] A. Bruns and J. Burgess. Researching news discussion on twitter: New methodologies. Journalism Studies, 13(5-6):801–814, 2012. [3] E. Diaz-Aviles, L. Drumond, L. Schmidt-Thieme, and W. Nejdl. Real-time top-n recommendation in social streams. In Proceedings of the sixth ACM conference on Recommender systems, pages 59–66, 2012. [4] Y. Duan, L. Jiang, T. Qin, M. Zhou, and H.-Y. Shum. An empirical study on learning to rank of tweets. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 295–303, 2010. [5] M. Elahi, F. Ricci, and N. Rubens. Active learning in collaborative filtering recommender systems. In E-Commerce and Web Technologies, pages 113–124. 2014. [6] A. Gupta and P. Kumaraguru. Credibility ranking of tweets during high impact events. In Proceedings of the 1st Workshop on Privacy and Security in Online Social Media, page 2, 2012. [7] D. Li, B. He, T. Luo, and X. Zhang. Selecting training data for learning-based twitter search. In Advances in Information Retrieval, pages 501–506. 2015. [8] J. Lin, M. Efron, Y. Wang, G. Sherman, and E. Voorhees. Overview of the TREC-2015 microblog track. In Proceedings of the TREC, 2015. [9] F. Martins, J. Magalhães, and J. Callan. Barbara made the news: mining the behavior of crowds for time-aware learning to rank. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, pages 667–676, 2016. [10] M. G. Rodriguez, K. Gummadi, and B. Schoelkopf. Quantifying information overload in social media and its impact on social contagions. arXiv preprint arXiv:1403.6838, 2014. [11] N. Rubens, D. Kaplan, and M. Sugiyama. Active learning in recommender systems. In Recommender systems handbook, pages 735–767. 2011. [12] G. Shani and A. Gunawardana. Evaluating recommendation systems. In Recommender systems handbook, pages 257–297. 2011. [13] C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to information retrieval. ACM Transactions on Information Systems (TOIS), 22(2):179–214, 2004. [14] X. Zhang, B. He, T. Luo, and B. Li. Query-biased learning to rank for real-time twitter search. In Proceedings of the 21st ACM international conference on Information and knowledge management, pages 1915–1919, 2012.