Sports news are a special case in the eld of news recommendations as readers often come with a strong emotional attachment to selected sports, teams or players. Furthermore, the interest in a topic can suddenly change if, for example, an important sports event is taking place. In this work, we present a hybrid sports news recommender system that combines content-based recommendations with collaborative ltering. We developed a recommender dashboard and integrated it into the Sport1.de website. In a user study, we evaluated our solution. Results show that a pure content-based approach delivers accurate news recommendations and the users con rm our recommender dashboard a high usability. Nevertheless, the collaborative ltering component of our hybrid approach is necessary to increase the diversity of the recommendations and to recommend older articles if they are of special importance to the user.
Recommender systems (RSs) suggest items like movies,
songs or points of interest based on the user's preferences.
Traditional RSs have to face some challenges when
recommending such items. One of the most common problems
is the cold-start problem [
In this work, we want to examine how well content-based RSs work for recommending sports news. In addition, we extend our RS by collaborative ltering. We develop a recommender dashboard and integrate it into the website of the German television channel and Internet portal Sport11. We evaluate both algorithms and the usability of our prototype in a user study.
This paper is structured as follows: in Section 2 we present related work and highlight our contribution to the current state of research in content-based and hybrid news RSs. We explain how we combine a content-based and a collaborative ltering component to a hybrid sports news RS in Section 3. Our development is evaluated in a user study. The results of this study are summarized in Section 4. This work ends with a conclusion and an outlook on future work. 2.
Di erent approaches try to tackle the problem of
personalized news recommendations. One of the rst news RSs
was developed and evaluated by the GroupLens project [
Using article keywords to build user pro les for news
recommendations has already been researched. The
Personalized Information Network (PIN) creates user pro les by so
called interest terms which consist of one or more keywords
[
Some researchers used hybrid RS combining di erent
techniques to suggest news articles. Claypool et al. [
Asikin and Worndl [
To the best of our knowledge, no research focusing on the special case of sports news has been done. In this work we want to show how sports news can be recommended in a content-based approach. In addition, we extend this RS by a collaborative ltering component. In a user study, we evaluate both approaches to nd out if the hybrid algorithm improves the recommendations. We show how sports news can be suggested to real users by developing and testing a fully working recommender dashboard which can be integrated into existing webpages.
This section explains the algorithms we used in our RS, but also illustrates the user pro le modeling that is needed to provide personalized recommendations. Finally, the prototype is shown to point out how our concepts are implemented on a website. 3.1
The user's preferences with regard to sports news are expressed by keywords of articles that she or he is reading. Each article of our recommendation database is characterized by ve to ten keywords which are automatically generated by analyzing the article's text. We are storing a list of keywords and how often each keyword occurs in articles the user has read. The more articles the user is reading, the better the recommendations are optimized with regard to the user's preferences. In our rst prototype the counter for each present keyword is incremented by one when the user reads the article containing this keyword. In future works, the keywords in an article could be weighted according to the relevance and importance of the keyword to the article.
The new user problem a ects every user who did not read an article yet. As explained, sports news di er from other kinds of news in the emotional attachment to selected sports, teams or players. We use this nding to overcome the new user problem. Before starting the recommendation process, the user can specify her or his favorite sport and team. News can then be recommended based on this selection and will improve when the user is reading articles, thus providing implicit feedback. 3.2
Content-based recommender suggest items that are
similar to items the user liked in the past [
One of the most important things for a news RS is to provide articles that are not dated. Especially in the sports news domain the environment is fast changing and usually the user is not interested in news about a sports event or her or his favorite team that are not up-to-date. The main challenge for us was to determine how old sports news can be before they are not considered for recommendation anymore. For our content-based RS we only take news into account that are not older than three days. Besides only providing relevant articles, this decision promises a better performance of the algorithm. The more articles are considered, the longer the process of calculating the recommendations takes. Our system currently uses only one news provider, but if the system grows, this could lead to a signi cant loss of performance. Our hybrid algorithm which incorporates collaborative ltering is also able to provide older articles if they are of high importance to the user (cf. Section 3.4).
The formula below computes the similarity between two articles (g and h),
sim(g; h) = where
Pi2W (gi hi) q(Pi2W gi2
Pi2W hi2) ; (1) g,h are vectors representing articles with
weighted keywords,
i is a keyword and gi,hi are the weights of i in g and h, respectively.
In the computation of content-based similarity scores we only consider the relative dimension of the keyword weights. For the reason that user pro les have di erent dimensions compared to articles, the use of relative dimensions provides better results for our system. As an illustration of the main idea of the algorithm, let us consider the simple case where the user pro le contains two keywords with the weights 5 and 10. Additionally there is another article with these two keywords but the weights are 1 and 2, respectively. In this case the similarity is 1, because of the same relative dimensions of the article and the user pro le.
The algorithm considers every article as an element in a vector space, where the keywords are forming the base. The coordinate of an article in the direction of a keyword is given by the weight of this keyword. If the keyword does not occur, the weight will be 0.
We normalize each article relative to the standard scalar product by dividing it by its absolute value. Consequently, the standard scalar product of the two normalized vectors conforms to the desired comparison features. Even if there are negative weights, e.g. for active suppressed keywords, the algorithm calculates similarities correctly.
In order to understand the similarity calculation better, we explain how the algorithm works for an article with itself (or another article with the same weight proportions). In this case, the scalar product is 1, because of the way the vectors are normalized. But if two articles have disjunctive keyword sets, the result is 0, because such vectors are orthogonal to each other.
In the end, the system sorts the articles by similarity descending and returns the 50 articles with the highest score. 3.3
In contrast to content-based ltering, a collaborative RS
uses the ratings of other users to calculate the similarity of
articles [
sim(i; j) = cos(~i;~j) =
(2)
~i ~j
kik kjk
The rst algorithm is the vector-based, also called
cosinebased, similarity. In this algorithm, items are represented
as two vectors that contain the user ratings. The
similarity between item i and item j is calculated by the cosine of
the angle between the two vectors. The " " denotes the dot
product of vector ~i and vector ~j [
sim(i; j) = qP u2U (Ru;i
Pu2U (Ru;i
Ri)(Ru;j
Rj) Ri)2qPu2U (Ru;j
Rj)2 (3)
The rst part of this algorithm is to nd a set of users U that contains all users who rated both items i and j. These items are called co-rated items. Not co-rated items are not taken into consideration of this algorithm. This similarity calculation is based on how much the rating of a user deviates from the average rating of this item. Ru;i represents the rating of a user u on item i and Ri denotes the average rating of an item i.
sim(i; j) = qP u2U (Ru;i
Pu2U (Ru;i
Ru)(Ru;j
Ru) Ru)2qPu2U (Ru;j
Ru)2 (4)
Adjusted cosine similarity takes into account that the rating preferences of the di erent users di er. There are some user that always give low ratings, but on the other side there are users that rate highly in general. To avoid this drawback, this algorithm subtracts the average rating of a user Ru from each rating Ru;i and Ru;j on the items i and j.
The presented advantage is the reason why we apply the adjusted cosine similarity in our development. First, the system has to calculate the related articles list of all articles. To compute the related article list of an article, we iterate through the list of all articles. If the current article is not the same as the article to compare, we will calculate the similarity.
The function returns a value between minus one and one. Since the article rating range is from one to ve, we map the similarity to the rating range by using the linear function: sim = 2 sim + 3 (5)
There is one bigger problem in the adjusted cosine similarity calculation. When there is just one common user between articles, the similarity for those items is one, which is the highest value of the rating range. This is due to the subtraction of the average rating from the user's rating. To avoid the e ect that the best rated articles are the articles with just one common user, we speci ed a minimum number of users that two articles need to have in common. In our implementation the minimum number of common users is ve. When there are less than ve common users, the articles are not considered in our related article list.
Afterwards, we sort the list by similarity. Moreover, we set a limit of 50 related articles to avoid additional expenses due to articles that are not considered for computation. When the related article list is calculated, we can predict the top articles for a user. For each article in the related articles list, we check if the user has already read the article. If that is the case, the article is not recommended anymore and the system jumps to the next article. If it is a new article, the prediction is calculated and added to the recommendation list. After calculating the prediction for every article, we sort the recommendation list by the predicted value.
The prediction Pu;i can be calculated by the weighted sum
method [
Pu;i =
Pall similar items;N (si;N
Pall similar items;N (jsi;N j)
Ru;N ) (6)
This approach is "computing the sum of the ratings given
by the user on the items similar to i" [
A key advantage over content-based ltering techniques is the fact that collaborative RSs are able to provide a bigger variety of topics. Furthermore, with collaborative techniques, it is possible to provide event- or trend based recommendations, such as news about the World Cup. A pure content-based RS is not be able to recommend news about the darts championship if the user has just read football articles before. 3.4
In this section, we explain how we combine the contentbased and the collaborative components to a hybrid sports news RS.
As combination technique, we use the weighted hybrid
strategy as described in [
We determined that the weights are just applied if both components of our system recommend the corresponding article. Otherwise, additional requests have to be sent to calculate a combined score for each article. If just one component recommends the article, just the score of this component is taken with the full weight. Due to this procedure, we are able to provide recommendations of both components. 3.5
We developed a dashboard widget which can be integrated in existing websites to provide personalized sports news recommendations. For the development of the front-end, we used the JavaScript framework AngularJS, the style sheet language Less and HTML5 local storage. Figure 1 shows a current screenshot of our recommender dashboard. Nine recommendations are presented at one time. When the mouse is moved over one article, the user can read it ("Ansehen") or reject the recommendation ("Entfernen").
It is critical to identify the user every time she or he accesses the RS to provide personalized recommendations. We avoided to implement a mandatory login as this could be a big obstacle for new users visiting a sports news website. Instead, we calculate a Globally Unique Identi er (GUID) which is then stored in HTML5 local storage without an expiration date. This is an important advantage for our RS. Due to the fact that HTML5 local storage has no explicit lifecycle, we can use it not only for user identi cation, but also for generating a pro le of the user. Storing this data on client site is decreasing the amount of data stored on the server which makes the system more scalable. Only the item similarities and recommendations are calculated on the server, due to direct access to articles from our backend.
In order to get content-based recommendations, the client sends an Ajax request to a NodeJS server. Therefore, the user ID and the corresponding pro le are sent as parameters. We decided to use an Ajax request due to the fact that the computation causes no overhead at site loading if it is done from JavaScript code. At our backend, the weighted keyword pro le is sorted by the keyword name alphabetically. As mentioned, we receive articles published within the last three days. After obtaining those articles from our article repository, we add the suitable keywords to each of them. The weighted keywords are in the same form as the user pro le to make them comparable to each other. The system calculates the similarity of the user pro le with every article. Therefore, the union of the keyword sets is built. Subsequently, the similarity is computed using formula 1.
A JSON response sends the 50 articles with the highest score back to the client. The response is then processed by the Angular directive of the personalized dashboard. If the user removes an article, the next recommended article takes over its place. In addition, further statistics like the last read articles or last and next matches of the preferred team are displayed.
For the computation of collaborative recommended articles we use the same NodeJS server. In contrast to other systems, we do not store our data in a database. Due to the fact that we have to iterate through lists most of the time to compare ratings and users, we decided to use arrays to store our data within the application. The ratings provided by the user are collected in a rating variable that is kept in memory. It stores JSON objects with the user ID, the article path and the provided rating. Furthermore, the current date is used to distinguish current data from dated ratings that are not relevant for our system anymore.
To speed up the similarity computation, we adapt the average rating of a user every time providing a new rating. The average ratings are kept in an extra variable for performance reasons. The current average rating and the number of ratings provided by the given user is enough to adapt the average. Just a few basic arithmetic operations are necessary to avoid calculating the average from the rating variable every time from scratch. We minimize the accesses to the rating variable due to the fact that this variable is the main component of our server. Most of the requests read or write this variable. Every variable access that can be eliminated helps to improve the system's performance.
Moreover, we store a list of users as well as a list of articles to iterate through these arrays without generating them rst. Using a list of all articles is primarily important when the system computes related articles. The list of related articles is updated every hour. A cronjob is executed every hour to consider current news as well. After one hour there are more ratings provided and the new item problem of a pure collaborative RS is suppressed.
For similarity calculation of two items, we need to nd a set of users that contains all users who rated both items. Therefore, we generate a list of objects that contain the articles and all the users who rated the corresponding article. To compute the user set of two articles, we compare the two user lists and determine the intersection.
The combination of the content-based and the collaborative part of our RS is implemented in JavaScript. First, we send an Ajax request to our backend to collect the contentbased recommended articles. In addition, another request is sent to our NodeJS server where the collaborative ltered articles are computed. If the collaborative ltered recommendations are returned correctly, the system computes the combination of both article sets. Finally, the recommended articles are returned and the JSON response is sent to the application.
In the news domain the age of an article is de nitely one of the most important properties when the article's attractiveness is determined for a user. Because of the recency problem, we decided to implement a route in our NodeJS server to remove dated ratings and articles from our system. Every two weeks a cronjob is executed and every rating that is older than four weeks is removed from the ratings table. The removal of those ratings implies the secondary e ect that old users that do not exist anymore are removed as well. This is a very common scenario in our system, due to the fact that we identify the user by using HTML5 local storage. If the local storage is deleted, the old user ID does not occur anymore. We decided to use these time intervals, because our content-based version considers only articles published the last three days and we want to provide recommendations of articles older than a few days as well if an older article is getting popular again. In this case, our system is able to recommend those articles as well as long as ratings are provided in the last four weeks.
We conducted user studies to evaluate our algorithms and the usability of our recommender dashboard. In this section we present the goals, the procedure and the results of our evaluation. We interpret our ndings to answer the question how well content-based algorithms support user in receiving interesting sports news and if a hybrid algorithm can improve the performance of our RS. 4.1
In order to collect usage data of real users, we tested the content-based approach on the live version of the Sport1 website. For this purpose, the recommender dashboard prototype is presented to one percent of the users. Due to the fact that the website is visited by thousands of users every day, one percent of the users is enough to evaluate not only the functionality but also the usability of our RS. In future, we will increase the amount of test users from time to time and adapt our implementation accordingly. We used Google Analytics to measure relevant Key Performance Indicators (KPIs) that help us to evaluate our solution. We analyzed how much the users clicked on the read and the remove button, respectively. Moreover, we tested how often the users navigated to articles they have already read by using the last read articles widget. In addition to the event tracking, we analyzed if there is an impact on the article ratings due to the new personalized dashboard. This is why we compare the average ratings of di erent articles. Articles are just taken into account, if they are rated by the one percent of users that can use the personalized dashboard.
At the end of our live study 5132 user IDs were registered on our server. This does not mean that more than 5000 di erent users used the dashboard due to the fact that every device has its own GUID and if the history of the browser is deleted, a new ID is generated. But there were enough users producing events we can track.
The click behaviors of the users give information about the user acceptance of the di erent components. Figure 2 illustrates how many clicks are executed on the di erent components of the personalized dashboard.
Almost 50 percent of the clicks were executed on the remove button of the news recommendation widget. On the rst view, this number is quite high. But if we consider that at all the other buttons navigate the user to another page, it is obvious that the remove button is executed more often than all the other buttons. If the user clicks on remove, the article will disappear and a new one will be displayed in the dashboard. The user is then able to interact again with the dashboard. 27 percent of the clicks are executed on the view article button, which is a good proportion. Especially, if we consider that the RS is new, it is noticeable that after every third interaction, an article that potentially ts to the interests of the user, is recommended. To get better informations about the quality of the recommendations, we need to organize a long-term study. The sports news domain is very dynamic and the click behavior is changing depending on the current events. By that reason, the two week evaluation is not enough to ensure that the amount of clicks on an element is constantly similar. Around one quarter of all clicks are executed on links and buttons which are not part of the recommender dashboard but provide additional information such as last read articles and team-related statistics such as last and next matches and top scorers.
We expect that the quality of the recommendations increases with the time of use. To test this assumption, we analyzed the trend of the remove button clicks. Except for some days, the ratio of clicks on remove decreased with every day performing our testing (cf. Figure 3). The exceptions may base on new users or users that do not read many articles on the website. If none or just a few articles are read before using our dashboard, the quality of the recommendations will be low. Since the dashboard is just presented to one percent of the users, we are not able to give evidence that the subjective quality will be the same when publishing the dashboard to all users. With increasing the number of testers, the ratio of remove button clicks increases at the beginning and then falls again with the time of use. We detected this when we released the dashboard on the website.
To analyze if the dashboard has an e ect on the article ratings, we compared three types of articles. First, articles that are bad rated in general, second average rated articles and nally articles that are high rated in general. A generally bad rated article gets a better score from our RS users. This is mainly due to the fact that we use the implicit feedback of three stars if a user reads the article. Moreover, the bulk of the users is not providing any rating for an article. So the average rating of the testers is almost at three stars for bad and average rated articles. For high rated articles, the RS users scored a little lower in general. The chance to get such an article provided by our system is higher due to the fact that more comparable users are available for the most read articles. If the user clicks on remove, the lowest rating of one star is implicitly provided and the average rating is decreasing. Since the personalized dashboard is not established on the website, we can sum up that the recommendations have almost no e ect on the rating scores. This may change if the users will use the dashboard as their rst contact point on the website.
It was also noticeable that the users want to read already read articles again. The last read article widget helps them to navigate back and easily get an overview of the last interactions. We expect that the amount of clicks in this widget will decrease in the future. Since new articles are potentially more attractive for a user, we can not imagine that every tenth click is executed on an already read article. We believe that the users in the study were curious and wanted to test this new feature. 4.2
Our hybrid algorithm extends the presented content-based approach by a collaborative ltering component. This algorithm is not part of the live version of the RS yet. We tested the hybrid recommendations with a selected user group. We paid attention to choose persons from di erent backgrounds and with diverging interests to ensure comparability to our users.
The participants had to rate the RS in two scales on a scale from 1 (worst rating) to 5 (best rating): How well the recommendations t their interests and how diversi ed they are. The pure content-based solutions served as a baseline algorithm. In total, we received 40 completed questionnaires for the content-based approach and 20 for the extended, hybrid RS.
The results show that the recommendations are not diversi ed enough in our pure content-based approach ( 2.9), but they improve in our hybrid implementation where the average rating was 3.3. The content-based recommendations are representing the interests of the user ( 3.4) which shows us that the dashboard provides additional value. This value did not change in our hybrid version. It was noticeable that the more frequent a user is visiting the website, the more he is satis ed with the result of the recommendations. The users that visited the website every day gave an average rating of 3.6. This con rms that the quality of our recommendations increases over time. 4.3
Besides evaluating our recommendation algorithms, we
asked the study participants to rate the usability of our
recommender dashboard. We used the well-established System
Usability Scale (SUS) [
To calculate the SUS score, the answers for each question
are converted to a new score from 0 to 4 where 4 is the
best score and 0 is representing the worst possible answer of
this question. Afterwards the di erent scores are added
together and multiplied by 2.5 to get a ranking value between
0 and 100 [
The score is calculated by adding the scores and multiplying the sum with 2.5: score = sum
The score of 87 exceeded our expectations although we attached great importance to the design and usability of our system. This was required because the dashboard is implemented on the live website of Sport1. Nevertheless the users mentioned some desires concerning the usability. For example, some users wished to change the design by choosing their own colors. Since these informations have not an direct impact on our RS implementation, we will not deepen these suggestions here. 4.4
The user study results show that our content-based RS is a promising approach to suggest sports news to users. The recommendations t the users' interests and improve when the users provide more feedback. Nevertheless, the diversity of the recommended articles remains to be low. This is a typical problem of pure content-based RS and can be overcome by using a hybrid solution. We extended the RS by a collaborative ltering component which increased the diversity of the recommendations.
As described before, it is very important that a news RS provides current articles. Especially in sports, the environment is very dynamic and the news topics are changing all the time. For that reason the system can not be a pure collaborative RS. With collaborative ltering it is almost impossible to recommend new items. But this problem can be solved by using a content-based component as well. Contentbased RSs can provide content that ts to the general interests of the users. In addition, attention should be paid to event based interests, e.g. the Super Bowl, an event that is closely followed by many people. If a user has no interests in American Football in general, the content-based RS does not provide articles about the Super Bowl. So there must be a combination of both techniques to bene t from the strengths of each component. 5.
In this work, we tackled the problem of recommending sports news. Sports news are a special case in the eld of news recommendations as users often come with a strong emotional attachment to selected sports, teams or players. Furthermore, the interest in a topic is event-driven and can suddenly change. We developed a content-based RS that creates user pro les based on implicit feedback the user shares when reading articles. Using automatically created keywords, the similarity between articles can be measured and the relevance for the user can be predicted. This approach delivers accurate recommendations but lacks diversity. In a rst prototype, we designed and evaluated a hybrid algorithm that extends our content-based RS by a collaborative component. This hybrid approach increases diversity and also allows to recommend older articles if they are of particular interest for the user.
To improve the quality of the hybrid recommendations, we will adjust our implementation from time to time and test if the adaptions serve their purpose. First, we will test di erent weights for the two components. One idea is to increase the weight of the content-based version. The decrease of the weight of the collaborative version does not exclude the event-based recommendations. Even if the collaborative part does just count one third, it is able to provide recommendations because if the article is only recommended by our collaborative version, just the score of this component is taken into account. If both components provide this article recommendation, the content-based version is more adapted to the users interests. To nd out which weight ratio is the best for our case, we have to analyze the implicit and explicit user feedback for a longer time period. The evaluation of the weights is just meaningful if the feedback is collected for a few months to avoid temporally uctuation, which is quite common in the news domain.
Furthermore, we want to implement a switching hybrid as well. If there is a new item, the collaborative ltering method can not provide recommendations from the rst second. This is the strength of our content-based version. The RS has to switch to the content-based version if the article is newer than a speci c date. Recommendations for a new user are calculated by our collaborative ltering component to handle new users as well as the preferences at the rst use of the system are not su cient to compute pure content-based recommendations. If a larger user pro le is constructed and an article is not published in the last minutes, the combination of both techniques will be applied as described before.
We tested our rst developments in a two-week user study. Our content-based RS has been tested with live users while the hybrid approach was only accessible for a selected user group. In future, we want to conduct larger studies with more users for all algorithms we develop. Our rst results will serve as the baseline for future extensions and other algorithms.
I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system. I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly. I found the system very cumbersome to use.
I felt very con dent using the system.
I needed to learn a lot of things before I could get going with this system.