Active Tweet Recommendation Based on User Interest Profiles

Abstract

The rapid growth of Twitter has made it one of the most popular information sources of current affairs. Twitter users gather information about their topics of interest through their followees' posts or by searching for relevant posts. However, users are often overwhelmed by the large number of tweets which makes it difficult for them to find relevant and non-redundant information about their interests. Information filtering and recommender systems can help users by suggesting informative tweets based on their interests. Considering the wide variety of topics in users' interest profiles and the sheer volume of tweets being published daily, it is difficult to have adequate and proper labeled data to train these systems on. We aim to tackle this problem by integrating active learning techniques into tweet recommendation, more specifically for finding relevant tweets and ranking them. Using active learning methods in the context of Twitter recommenders has not been well explored before. Our objective is to exploit these methods for improving the accuracy of tweet recommenders the most, while keeping the cost of labeling to a minimum.

Keywords

• •Information systems → Recommender systems

1. INTRODUCTION

With the constant growth of blogs, forums, and online social networks, the need for information retrieval systems, recommenders and analytical tools increases. On the other hand, existing standard techniques may be inadequate to overcome new challenges introduced by particular characteristics of the data generated from these sources. Online chats, tweets, blogs and any other data sources with informal settings contain short texts riddled with spelling errors, incorrect grammar, acronyms and non-standard words. In addition, methods for processing and analyzing these type of messages should be ideally real-time since they are usually only interesting during short time after being published.

Twitter has become one of the most popular social media platforms with vast and various number of discussed topics and shared information. Many users seek and collect information about their areas of interest from Twitter. However, considering the sheer volume of tweets published daily (around 500 million tweets), users can be overwhelmed by large amounts of tweets from their followees [ 10 ]. Information filtering and recommender systems aim to find useful and novel information based on the user’s preferred topics and suggest them in a ranked order. Since users can have diverse interests, each changing over time, and also as current events are rapidly added to the vast set of topics of discussion on Twitter [ 2 ], it is not appropriate to use old data for training recommenders on. In addition, considering the cost of labeling and the real-time nature of Twitter, it would be infeasible to label large quantities of data for every user in different time spans. In this Ph.D., we aim to introduce and apply active learning strategies, which select a number of instances to be labeled by an information source, in order to improve the accuracy of the recommender systems while minimizing the labeling cost. 2.

PROPOSED METHOD

We propose to integrate active learning strategies with learning-to-rank and information filtering methods. Our framework is presented in Figure 1, which shows the use of active learning in both tweet relevance model and ranking model of our recommender system. We briefly discuss each component in this section.

We assume that users specify the gist of their interest in a few keywords. Therefore, we formulate a query from these keywords and summarize our recommendation task as “At time T , give a ranked list of most relevant and novel tweets about query Q”. We consider the language model-based retrieval using Dirichlet smoothing [ 13 ] as our baseline relevance model. This model retrieves a list of tweets ranked by their relevance to the search query. Verifying the relevance of retrieved tweets by active learning strategies can improve the accuracy of tweet relevance model (see Section 3 for some preliminary results). To take into account the novelty of the recommended tweets, we cluster the retrieved tweets and only include the tweet that was published earlier within each cluster in the final recommendation list [ 8 ]. We consider these sequence of steps as our baseline recommender system.

In the baseline model, we rank retrieved tweets based on their relevance score by the language model. It has been shown that social attributes such as the number of retweets, URLs or hashtags in the text of tweets, or followers of the tweets’ authors contain information about the credibility of tweets [ 6 ] and should also be considered in ranking of tweets. Having a set of features, learning-to-rank methods have been used for constructing ranking models for tweet recommendation [ 3, 4, 14, 9 ]. However, these methods need labeled data for training the ranking model. Assuming the availability of human labels or limited labeled data, active learning and semi-supervised techniques focus on selecting the best instances to be considered in the training data. Using these techniques in recommender systems in general has been studied before [ 7, 11, 5, 1 ]. However, adapting these strategies for Twitter message recommendation has not been explored much to the best of our knowledge.

DATASET AND INITIAL EVALUATION

In order to study how user involvement can improve the performance of our tweet recommender system, we performed some preliminary experiments using the TREC 2015 Microblog track (TRECMicro) dataset [ 8 ]. We collected tweets during the evaluation period of the TRECMicro using the Twitter’s streaming API. We gathered 40,264,332 tweets, out of which only English tweets (16,302,498) have been used in our experiments. We consider the judgment scores provided by the TRECMicro and the normalized Discounted Cumulative Gain (nDCG) [ 12 ] as our gold standard and evaluation metric respectively. nDCG is calculated over k (k = 10 here) top suggested tweets for each user profile per day. The final value is the average over the evaluation period and all 51 user interest profiles (also provided by TRECMicro). In cases where there are no relevant tweets to the user interest in a particular day, the recommender system should remain silent (achieving nDCG = 1) for that day, otherwise any tweet recommendation penalizes the system (nDCG = 0).

The focus of the evaluation metric on the precision of the recommended tweets requires including only tweets the system is confident about. Considering a threshold and discarding tweets with relevance score less than that threshold is one possible way. However, not having labeled data makes it difficult to tune this parameter for different users with various interests. Therefore, we consider the most basic strategy for selecting tweets for labeling (i.e. to be labeled by the information source), which is to select only the top recommended tweet for each user profile per day. If the top selected tweet is not relevant to the user interest, we discard all selected tweets for that day.

The results of applying this simple strategy to the relevance model is reported in Table 1. It also shows the results of top 3 teams that participated in TRECMicro, which indicates the potential of active learning techniques in improving the performance of the tweet recommender systems. TRECMicro categorized the systems based on the amount of human involvement into three different categories: automatic with no human input, manual preparation with human input only before the evaluation starts and manual intervention with human input all the time. Our proposed approach using active learning falls under the category of manual intervention.

Type manual intervention manual manual automatic automatic 4.

FUTURE PLAN

To overcome some of the challenges related to real-time tweet recommender systems, we plan to introduce and apply active learning strategies. These strategies select the instances to be labeled by an information source in a way that the quality of the recommended tweets improves the most. Initial analysis, reviewing state-of-the-art methods, and preliminary experiments has already been done. Our future plan for the next year is to integrate active learning into different retrieval and ranking models and investigate their effectiveness in improving the results of our baseline recommender system. In addition, we would like to analyze the effect of different groups of features including contentbased features and social attributes on the accuracy of tweet relevance and ranking models.

ACKNOWLEDGMENTS

This research was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and The Boeing Company.

References

[1] L. Bl ´edait´e and F. Ricci . Pairwise preferences elicitation and exploitation for conversational collaborative filtering . In Proceedings of the 26th ACM Conference on Hypertext & Social Media , pages 231 - 236 , 2015 .

[2] A. Bruns and J. Burgess . Researching news discussion on twitter: New methodologies . Journalism Studies , 13 ( 5-6 ): 801 - 814 , 2012 .

[3] E. Diaz-Aviles , L. Drumond , L. Schmidt-Thieme , and W. Nejdl . Real-time top-n recommendation in social streams . In Proceedings of the sixth ACM conference on Recommender systems , pages 59 - 66 , 2012 .

[4] Y. Duan , L. Jiang , T. Qin , M. Zhou , and H.-Y. Shum . An empirical study on learning to rank of tweets . In Proceedings of the 23rd International Conference on Computational Linguistics , pages 295 - 303 , 2010 .

[5] M. Elahi , F. Ricci , and N. Rubens . Active learning in collaborative filtering recommender systems . In E-Commerce and Web Technologies , pages 113 - 124 . 2014 .

[6] A. Gupta and P. Kumaraguru . Credibility ranking of tweets during high impact events . In Proceedings of the 1st Workshop on Privacy and Security in Online Social Media, page 2 , 2012 .

[7] D. Li , B. He , T. Luo , and X. Zhang. Selecting training data for learning-based twitter search . In Advances in Information Retrieval , pages 501 - 506 . 2015 .

[8] J. Lin , M. Efron , Y. Wang , G. Sherman , and E. Voorhees . Overview of the TREC-2015 microblog track . In Proceedings of the TREC , 2015 .

[9] F. Martins , J. Magalh˜aes, and J. Callan . Barbara made the news: mining the behavior of crowds for time-aware learning to rank . In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining , pages 667 - 676 , 2016 .

[10] M. G. Rodriguez , K. Gummadi , and B. Schoelkopf . Quantifying information overload in social media and its impact on social contagions . arXiv preprint arXiv:1403.6838 , 2014 .

[11] N. Rubens , D. Kaplan , and M. Sugiyama . Active learning in recommender systems . In Recommender systems handbook , pages 735 - 767 . 2011 .

[12] G. Shani and A. Gunawardana . Evaluating recommendation systems . In Recommender systems handbook , pages 257 - 297 . 2011 .

[13] C. Zhai and J. Lafferty . A study of smoothing methods for language models applied to information retrieval . ACM Transactions on Information Systems (TOIS) , 22 ( 2 ): 179 - 214 , 2004 .

[14] X. Zhang , B. He , T. Luo , and B. Li . Query-biased learning to rank for real-time twitter search . In Proceedings of the 21st ACM international conference on Information and knowledge management , pages 1915 - 1919 , 2012 .