Predicting user behaviour has grown increasingly important in the context of news recommendation over the years. Knowledge of behavioural patterns o ers valuable insight for developing e cient and user friendly services, ensuring good user experiences for users and increased revenues for companies. Though very useful, these user recommendations can be di cult to make and the news domain poses its own, unique challenges for the task. As a result of the news domain's volatile nature, sparse user-item matrices in excessively high dimensional space is a prominent challenge in news recommendation. This makes the utilization of traditional user modeling and recommendation methods di cult, especially due to the domain's inherent need to recommend unseen items, known as the cold start problem. Extracting abstract content descriptions by traditional topic modeling, and the more recently developed method of topic embedding, on the rich text objects provided by the news domain, new articles can be compared to older articles with more collected context and data in a meaningful and e cient way. By organizing these articles in clusters based on topic proportions, useful topic combinations can be elicited. Using these clusters to create user models based on composite interests, newly introduced articles can be recommended to users based on their topic compositions. The issue of nding a compromise between optimization of performance in subtasks and the nal prediction is prominent in the experiments and results, and will be crucial in future works if the method described in this article is to be useful in a real life context. Despite this challenge, extensive experiments on two comparable real-world datasets did show improvements on user prediction results.
Recommender systems (RS) have been a topic of interest within research and customer services for
years, proving itself an e cient and popular method of improving user experiences and revenue [
The area of news recommendation also faces more unique challenges such as the frequency at which new items are added and the need to recommended items with little or no user history available, further amplifying the sparsity of the data. This is also re ected in the demand for item recency, the recommendation of older news articles is rarely of interest to the user. In a research eld which has often relied on extensive purchase histories for both user- and item-based models, these features requires certain adaptions in order to make useful recommendations.
Explicit user feedback is also a rarity, especially measured against the total number of users and
the frequency of article consumption per user [
NOBIDS 2018 whether a particular user has read the article or not, with no guarantee of whether the user enjoyed the article itself or a composition of the topics it encompasses. It is also important to note that a user may have enjoyed an article they have not read, and assumptions about articles that are not read are also di cult to make, especially due to the mentioned high turnover rate and volumes of articles. Additionally, certain articles, such as breaking news, are commonly popular across all groups and types of users regardless of their composite personal interests.
These features produces many variables and considerations that needs to be addressed when recommending news articles. This article provides a method that deals with these challenges by modeling topical, dimensionality reduced representations of documents. Extraction of abstract topics from text corpora enables generation of rich item information based on content at introduction time, and further allows for automated document comparison and user models built on expressed interests in composite topics, as well as alleviating the data sparsity and thus the cold item start problem. By connecting articles and users to higher level topical clusters, users may also be recommended articles in a broader range than if solely using each user's read history, facilitating exploration and avoiding monotone lter bubbles. The method shows improvements on user prediction results, but a key nding which should be emphasized is the con ict between optimizing goals for subtasks and optimal performance in the prediction task. 2 2.1
In the context of recommender systems, community detection through clustering has proven to be
an e cient tool for personalized recommendations extracted from interactions between users and
rather persistent items such as books [
However, a key component of a user model is the collected knowledge of user behaviour through
user feedback (explicitly or implicitly [
In order to deal with the sparsity issue as a consequence of the volatile news domain, a content based approach may be suitable, comparing content of the news articles and thus not mainly relying on user-item interaction. As simply comparing the articles' words will result in a highdimensional sparse matrix, dimensionality reduction may be applied in order to achieve a more general representation of the articles' content, and consequently user history in terms of topic interests.
Topic modelling allows for this, by discovering overall topics from unstructured data. While
discriminative models have been preferred for many natural language processing tasks due to their
clear relation to the traditional classi cation task of nding the label Y when given the features
X, generative models provide more exibility and can be more suited for cases lacking fully labeled
datasets or where the opportunity to generate sample data is advantageous [
A topic model which has inspired many further models is the generative statistical model Latent
Dirchlet Allocation (LDA). A key feature of the Dirichlet distribution in topic modeling is that
it assumes that each topic has a few high-probability words connected to them, and that each
document in turn has a few high-probability topics connected to them, a natural assumption for
many topics and documents which allows for each document to be presented as a composition of
topics [
An alternative method of topic extraction, addressing some of the these issues, is by the use of
representation learning. Representation learning is a manner to extract semantic similarities and
interesting relationships between entities, such as words, and has gained great attention within
deep learning research. It also allows for sparsity alleviation through compressed representation,
and e orts to exploit sparsity in the generation phase is common practice [
Some attempts have been made to incorporate the methods in the news domain, with several
clustering topic modeled news articles but never extending to recommendation [
With a more general and comparative representations of the documents, topical communities may
be detected through clustering, which has been applied to similar corpora with promising results [
In the news domain, a suitable clustering algorithm should be able to handle large datasets,
many clusters of varying sizes and preferably also non- at geometrical shapes. k -Means [
As described earlier in the article, the news recommendation area struggles with providing a data
foundation on which traditional recommendation methods of collecting user history on item
before recommending them, or denoting items by simple categorizations, can perform optimally. The
process presented in the following aims to improve recommendation of news articles by
performing predictions which mainly relies on content, and for this purpose utilizes topic extraction and
clustering methods to build user models and embed new articles into an existing space of set
dimensions. Letting the algorithms for topic extraction and clustering be denoted by parameters
and , a set of new articles D and -parameters are sent into , outputting dimensionally recuded
document vectors V , with dimensions corresponding to topics. V and -parameters are used by
to generate a set of topical clusters C of documents. Unseen documents may then be embedded by
, and connected to topical clusters in C. Using each user's document base Du, each user is
represented by a cluster membership vector !u, being input together with C, to the predictor generating
recommendations . Figure 3 shows the implementation at a conceptual level, also presented in in
algorithm 1. A brief description of the main objectives and setup, further described in [
In the context of this paper, the topic modeling will either infer the the full corpus likelihood in TopicVec:
p(Z; j ; D; A; V; T):
Here, the training objective is to discover the word-topic and document-topic distributions, Z
and , given the hyperparameters and documents, and xing the word embedding variables A and V
along with the topic candidates T which is inferred from the whole corpus. Extensive mathematical
proof can be found in [
In the case of LDA, the corpus likelihood is calculated as p(D j ; ) =
YM Z d=1 p( d j )
Nd ! Y X p(zdn j d)p(wdn j zdn; ) d d: n=1 zdn where N represents a word-topic combination, w is the word and z is the topic distribution of the word. M denotes the document in the collection and is the topic distribution of the document, with m representing the distribution for the document m. Both models utilizes Dirichlet priors to capture the relationship between words topics and documents and topics. (1) (2) 3.1
Due to lack of access to a complete dataset re ecting the news domain in terms of su cient attributes on both articles and user history, two public datasets were mainly used, complementing each other. The News Aggregator 1 dataset, containing categorized headlines, was used to simulate the categorized article documents, while a data collection of user submissions at the social content aggregation and discussion website reddit 2 was used to cover user interactions. Thus it should be noted that only the latter was available for recommendation testing, while the rst provided a comparison of the reddit data and real life news data.
Topic extraction is performed on both data sets, with each model utilizing either LDA [
is chosen. When TopicVec is used, the algorithm is run on top of the continuous word embeddings
provided by its predecessor PSDVec, as recommended from [
Extracted topics are used to represent documents as topic vectors. The topic vectors resulting from both LDA and TopicVec are then clustered by the k -means or HDBSCAN algorithm, applied to document vectors resulting from the varying numbers of topics in the previous step, and with a range of clustering parameters in order to observe trends for di erent numbers and sizes of clusters. Users are later connected to the resulting topical communities based on their interaction history.
The best parameters are chosen based on Silhouette coe cient and Calinski-Harabasz (CH) index, and the best clusterings are then used to create user models based on the users' interactions. Due to the inability to assert these evaluation metrics as a guideline for the best possible clusters for the recommendation task, some visualization and qualitative choices are used to pick out some complimentary models for further testing. These qualitative choices were aided by evaluating 2-D and 3-D visualizations of clusterings on topic modeled documents, also compared to clusterings resulting from the pre-de ned categories and subreddits. Another qualitative consideration when selecting models, was choosing models which were not top performing in terms of metrics but provided a compliment to quantitatively chosen models in terms of topic or cluster numbers, and the ratio between the two.
The user model is built by classifying all unique documents the user either has published or
commented on. The users memberships to each topical community is then decided by number of
interactions within each community. More precisely membership strength, !, is given by
!u;c = log(countu;c + 1)
(3)
where countu;c is the number of unique document user u has interacted with in cluster c. To
emphasize activity involving more rare users and clusters, inspired by [
During prediction, 230 documents are attempted predicted for each user, where 30 of these are known to be in the user's interaction history. The ranked list is Mean Average Precision (MAP) evaluated at position 3, 5, and 10, in addition to the full list and 30, being the number of articles we wish to recommend. A ranked list sorted by Ranking Accuracy (RA) is also generated, appearing to better capture the quality of recommendations, considering the characteristics of each prediction evaluation metric. Documents classi ed as noise are not predicted. Both metrics and their dependencies are displayed in table 1. 4
Evaluation Metric
Equation
Parameters Average Precision
Ri Mean Average Precision M AP =
Percentile Ranking rank = jQj P u;i
P u;i u;i ranku;i
u;i Ranking Accuracy
50% rank RA = 50% k = number of top elements
Q = set of all users u;i = 1 if user u interacted with article i, 0 otherwise
parameter in k -Means, and s and c the minimum sample and cluster parameters in DBSCAN3. The prediction result for each of the models presented previously in table 2 is displayed in gure 4. The combination of TopicVec and DBSCAN produced such excessive numbers of outliers and very small clusters that it seemed of little use for the intended purpose, and is thus not included.
Di erent combinations of parameters provides prediction results with very di erent qualitative characteristics, but comparing to predictions using original labels and random recommendations, the models obtain better MAP and RA than both. With models using both topic modeling and clustering, RA shows improvements on random, ranged from 34%-64%, for models which performed
(https://hdbscan.readthedocs.io/en/latest/parameter selection.html)
Fig. 4. Prediction results. well. Other models gains improvements ranging from 18%-278 % on top 3, 5, 10 and 30 positions for MAP, but declines in performance on RA by 18% 28%.
As can be observed, several models perform notably better when only evaluating the upper part of the ranked list (top 3, 4, 10, and 30), but have an excessive drop in performance when evaluating the full list. In Figure 5, the distribution of positive hits are highlighted, showing some interesting tendencies. The models which obtains the higher MAP-values, have a very high focus of positive articles in the very rst part of the list and in compact groups between the 50th and 75th percentile in the lower half of the recommended list. Those with lower MAP-values, generally obtain higher RA-scores. As seen in Figure 5, these models shows more evenly distributed positive hits in the list, and have a heavier focus in the rst half percentiles of the recommendations, but su ers in MAP evaluation due to the gap present between the very rst hits and the 10-25th percentile. Combining topic modeling and community detection, appears to shift a large bulk of the positive hits 25-50 percentiles lower down in the list. While this proves to increase MAP-scores, it appears that the concentration of positive hits in such high percentiles could indicate that the RA provides a more appropriate evaluation criteria.
An important and interesting nding, is that the models chosen for their promising evaluation scores in some ways perform worse than those chosen for comparison reasons or by qualitative consideration. Two such qualitatively chosen models are LDA+KMEANS4 and TV+KMEANS2, performing well within RA and thus presenting more of the relevant documents in the earlier part of the recommendation list. When using LDA, selections of lower-quality clusterings sometimes yielded better prediction results, highlighting that clustering may prioritize qualities which are not appropriate or optimal for prediction. This was also observed when using topic embedding, where some models with qualitatively chosen embeddings combined with lower-performing clusterings yielded prediction results that are preferable in a news recommendation case, compared to models with more optimal clustering evaluation metrics.
Consequently, one of the main challenges throughout the experiments has been optimizing unsupervised training models across di erent sub-goals to meet the prediction end goal. As observed, eliciting the best topical clusters for the explicit purpose of user recommendation is challenging as di erent evaluation metrics may be con icting, and in turn, the learning objectives for topic extraction may not provide the optimal results for the purpose of clustering either. This especially relevant for topic embedding which is traditionally tested on tasks such as classi cation. The alignment of goals across technologies in the di erent steps may be complicated, and discerning the optimal solutions for the next step while relying on unsupervised training can be di cult. 5
This article addresses the sparsity issue in the new domain by utilizing dimensionality reduction through semantic analysis and clustering. In order to overcome the lack of su cient user feedback and provide better recommendations, this is implemented in a pipeline conducting document embedding, extraction of high level topical clusters, user modeling, and prediction.
A key motivation for the model was to provide set-length topical representation of documents and users, which are possible to generate the instance a new document is introduced to the system. In doing this, we have provided a model which is capable of recommending previously unseen documents, a vital property for a news recommender system. All combinations of topic extraction and clustering does perform better than random and the use of original labels to recommend, both in terms of MAP and RA.
Though the system does support the recommendation of previously unseen documents through topic modeling and clustering, aligning the subgoals of optimal topic representation and clustering proved di cult. The quantitative evaluation metrics used to evaluate clusterings did not necessarily make for better recommendation models and the evaluation metrics for the recommendations themselves also proved to be misleading at times.
An important note is the noisiness and partially unrealistic comparison to the news eld the tested dataset produces. A more complete and directly news related dataset with the possibility of connecting a single users history over time is necessary to produce more realistic testing enviroments.
The model described in this paper, does not consider the e ect of long-term and short-term interests among users, nor the potential di erences in life span for articles of di erent types and topics. Adding considerations for these time sensitive attributes could provide a model which not only gives better recommendations, but which remains able to give valuable predictions over time.
Though these considerations and additions to the described model could further improve recommendations, a main challenge remains the alignment of subgoals in the tasks of topic modeling and clustering and the nal recommendations. In order for the methods to be useful, a way to automatically discover optimal variables across the process is important, and new evaluation metrics may need to be utilized or created for the task.
The authors would like to thank L. Zhang and P. Liu at the Department of Computer Science at NTNU for their support.