Graph-based approaches have become an efective strategy to model the users' preferences in recommender systems accurately; however, despite their excellent recommendation quality, the literature still needs to incorporate impressions (past recommendations) into existing approaches. By their definition, impressions contain the selection of the most relevant items for the user; enriching the users' profiles with those items may lead to higher-quality recommendations. In this work, we propose and empirically explore the efectiveness of two approaches that include impressions into graph-based recommenders. Both approaches are simple yet extensible as they do not change the definitions of the recommenders; but transform their main data structure: the graph's adjacency matrix. The results of our experiments suggest that our approaches may improve the recommendation quality of graph-based recommenders that do not use impressions; however, we also find that beyond-accuracy metrics may become negatively afected.
tional matrix factorization recommender. Recently, Pérez Maurera et al. [5, 6] performed a systematic literature Graph-based recommenders model the relationship be- review about impressions in recommender systems. In tween users and items of a recommender system using their review, they show how previous works have incorgraphs [1]. In this work, we focus graph-based recom- porated impressions into recommendation approaches. menders that use bipartite undirected graphs, such as For instance, the literature incorporates impressions into the ones used by Cooper et al. [2] and Christofel et al. [3]. a wide range of techniques, some papers [7, 8] include This type of graph represents users and items as its nodes, them to heuristics, while others include them to machine and the edges indicate a pair-wise relation between the learning [9, 10], deep learning [11, 12], or reinforcement user and the item. The graph is bipartite between the learning [13, 14] recommenders. user and item nodes, meaning edges only connect a user The literature has yet to include impressions on existnode to an item node. ing graph-based approaches or to develop new ones using
Impressions are a data source that contains the items impressions. In this work, we address this gap and proshown to the users. In other words, impressions are the pose two methods to incorporate impressions into this recommendations generated by the recommender. The type of recommenders. We propose adding impressions community’s awareness and interest in impressions has to the primary data structure used in graph-based recomincreased in recent years. The increased interest is sup- menders: the adjacency matrix. To test the efectiveness ported by papers incorporating impressions into existing of our approaches, we perform an evaluation study on recommenders or developing new recommenders that three public datasets with impressions and compare the use impressions. For instance, Aharon et al. [4] incorpo- recommendation quality of two strong graph-based recrated a latent factor that estimates the preference of users ommenders when building the adjacency matrix with the to repeated impressions with the same item to a tradi- traditional formulation and our approaches. We propose to use impressions as the users’ profiles to build the adjacency matrix. Such an approach contrasts with the traditional formulation that uses the interactions of users, as shown in Equation 2. The reasoning behind this method is that impressions include additional information to the users’ profiles, i.e., by definition of impressions, we can expand the users’ profiles to have the items the recommender system presented to the user.
Graph-based methods may leverage the additional information in users’ profiles to produce higher-quality recommendations.
Mathematically, we construct the adjacency matrix of (1) the graph using the instead of the : =
0 ⊺ )︂
0
We propose to change the definition of the graph; mainly, we define it as a
directed graph, where the edges go in we create an edge from the user node to the item node. Second, when a user is impressed with an item, we create an edge from the item node to the user node. The traditional characteristics of the graph remain unchanged, i.e., only user and item nodes are permitted, and the graph is bipartite.
As we change the definition of the graph, we also (2) change the definition of the adjacency matrix , which now reflects the directed edges: =
0 ⊺ )︂ 0 (4) (5) those subscripts are combined, we obtain the value on the cell indicated by the row and column.
user-item relations by first inspecting using the userrating matrix ∈ R||×|ℐ| . This matrix holds the interactions users performed with the system, where an interaction is any action the users performed on an item. When the recommender system uses explicit interactions, holds the ratings the users performed, e.g., numbers between 1-5. When the recommender system uses implicit interactions, holds binary indicators that tell whether the user interacted with an item or not: , = {︃1 if interacted with 0
otherwise
Graph-based recommenders model user-item interactions by building an undirected bipartite graph = (, , , ), where is the set of nodes, is the set of edges, are the attributes of nodes, and are the attributes of the edges. For graph-based recommenders, nodes are users and items, and the edges are the interacThe content of the gives the weight of the edges. As the graph is bipartite, the edges only connect user nodes to item nodes, i.e., no edge connects two item nodes or two user nodes. This graph is represented mathematiwhich encodes the information of the graph: cally using an adjacency matrix ∈ R(||+|ℐ|)× (||+|ℐ|) = ︂(
0 ⊺ )︂ 0
This section presents our two methods to incorporate impressions into graph-based recommenders. First, the definition of an impression is the recommendation list shown to the user by the recommender system. Second, we define the user-impressions matrix ∈ R||×|ℐ| as a matrix that indicates whether a user has been impressed with an item: , = {︃1 if was impressed with 0 otherwise
(3) tions (either explicit or not) between a user and an item. two directions. First, when a user interacts with an item,
As the impressions also contain the interacted items, of the and , we de-duplicate the user’s inthe contents of the traditional are also in . teractions with the same item by keeping the one with In other words, contain interacted and impressed but non-interacted user-item pairs. ︂( ︂(
In our experiments, we follow the same evaluation methodology as Pérez Maurera et al. [15, 16] as they already proposed a framework to evaluate recommenders that use impressions.
Datasets
in two recently published datasets with impressions:
Datasets Processing
the latest recorded date and time. As interactions are paired with impressions, the de-duplication process also removes the impressions associated with the removed interactions. Lastly, we remove users with less than three interactions from the datasets.
Recommendation Task For a given user, the recommenders are tasked to generate a recommendation list of items containing relevant items to the users. In our experiments, an item is considered relevant if the user interacted with such an item in the test split. We evaluate the recommendations using traditional accuracy and beyond-accuracy metrics. Regarding accuracy metrics, we evaluate the recommenders using the normalized Discounted Cumulative Gain (nDCG) [19], Precision [19], and Recall [19]. Regarding beyond-accuracy metrics, we evaluate the recommenders using the Coverage [20], the Diversity Gini [20], and the Novelty [20].
Datasets Splits We partition those datasets into three the first 16 times it selects random hyper-parameters; in splits train, validation, and test using a user-wise leave- the remaining cases, the optimizer exploits those hyperlast-out strategy: for each user, we sort their interactions parameters with the highest likelihood to become the and impressions by their recorded date and time. Then, optimal set of hyper-parameters. When tuning the hyperwe place the last interaction and impression in the test parameters, the recommender is trained using the train split, the second last interaction and impression in the split and is evaluated against the validation split. At the validation split, and the remaining interactions and im- end of the Bayesian search, the optimizer trains the recpressions in the train split. ommender on the union of the train and validation splits, it selects the hyper-parameters with the highest recommendation quality found, and evaluates the recommender on the test set.
Baseline Methods In our experiments, we train two well-known graph-based recommenders: 3 and 3.
Those recommenders have shown excellent competitive recommendation quality in previous works [21, 22] against other recommenders and represent the strongest graph-based baselines. 3 was proposed by Cooper et al. [2] and performs random walks through the adjacency matrix to model the users’ preference to items in the catalog. 3 was proposed by He et al. [23] and applies the same concept as 3; however, with the diference that the edges of the graph are weighted to account for the items’ popularity. Despite the simplicity and extension capabilities of our approaches, we do not evaluate graph-based recommenders using deep learning, e.g., LightGCN [23] due to their long training time.
From Table 1, we notice two patterns in the results dependent on the dataset we test against. On the MIND dataset, our proposed methods are efective and produce recommenders that achieve a higher recommendation accuracy when compared to the baseline method, i.e., 3 and 3 benefit from the inclusion of impressions in the adjacency matrix, either by considering only impressions (denoted as UP, see Equation 4) or by building a directed graph (denoted as DG, see Equation 5). Overall, the UP approach is the one that obtains the highest accuracy in five out of the six comparisons; in the remaining one, the DG approach achieves the highest Proposed Methods As presented in Section 3, our nDCG. The results suggest that the UP approach can approaches incorporate impressions to graph-based rec- model user preferences with finer granularity than the ommenders by modifying their adjacency matrix. To try DG and traditional approaches. Moreover, for the UP the efectiveness of our methods, we train 3 and 3 approach, the relative improvements in accuracy on this using the adjacency matrices defined in Equation 4 and dataset are higher when training a 3 recommender than Equation 5. We denote those approaches with the sufixes a 3 recommender. The opposite occurs for the DG “UP” and “DG”. In total, we perform our experiments with approach, where pairing it with a 3 recommender three variants of both 3 and 3: one is the traditional yields a higher relative improvement than pairing it with (see Equation 2), one uses our UP approach, and the other a 3 recommender. Lastly, for 3 recommender, we see uses our DG approach. minimum relative improvements in the accuracy of 50 %. In contrast, for the 3 recommender, we see minimum Hyper-parameter Tuning We perform Bayesian op- relative improvements in the accuracy of 100 %. timization [24] to search the hyper-parameter space of On the ContentWise Impressions dataset, the results the recommenders and select the combination of hyper- are not favorable; indeed, none of our proposed methparameters that yield the highest nDCG. Mainly, the op- ods achieve a higher recommendation accuracy when timizer trains each recommender 50 times, out of which compared to the baseline method, i.e., 3 and 3 obTop-20 accuracy and beyond-accuracy metrics of graph-based recommenders when evaluated on the MIND and ContentWise Impressions datasets. The recommender without sufix (i.e., 3 and 3) use the adjacency matrix shown in Equation 2. The sufix “-UP” indicates the recommender uses our proposed adjacency matrix shown in Equation 4. The sufix “-DG” indicates the recommender uses our proposed adjacency matrix shown in Equation 5. For each metric, we indicate in boldface the methods using impressions with higher accuracy than the recommender without impressions.
Precision
Coverage
Diversity Gini
Novelty Dataset MIND ContentWise Impressions
Method cally, between − 20 % and − 50 % less accuracy than the
When inspecting the beyond-accuracy metrics, we notice that our approaches generally yield lower values on all datasets and both 3 and 3 recommenders. Notably, the coverage and the diversity Gini metrics are the most afected, with negative relative improvements ranging from − 55 % to − 95 %.
When inspecting the Coverage metric, the results indicate that although our approaches yield higher accuracy, they recommend fewer unique items from the catalog. At the same time, when inspecting the Diversity Gini metric, the results indicate that our approaches have less diversity. Some exceptions to the behavior on the previous metrics occur when inspecting the Novelty metric; particularly, the metric measure the ability of recommenders to recommend unpopular items. In three cases, the novelty of the UP approach was higher than the DG and the traditional approaches; despite that result, the relative improvement ranges from 1 % to 5.5 %.
This work proposes two approaches to incorporate impressions into existing graph-based recommenders. We empirically validate the efectiveness of our approaches by conducting experiments on two datasets with impressions. These experiments show positive and negative outcomes: our approaches consistently obtain more accurate recommendations on one dataset; however, it comes with less diverse recommendations. Future works may analyze this efect and understand whether such results are due to the recommendation domain or characteristics of the datasets.
Our approaches do not currently model the nuances of users and their preference toward impressions. Notably, we assign the same importance to impressions and interactions on the definition of the adjacency matrix of our approaches (see Section 3). This is more evident in the DG approach, as we construct a directed graph where each edge is 0 or 1. A future direction is to tune the importance of impressions on the edges of the graph when constructing the adjacency matrix; this would be similar to the ideas behind the 3 recommender.
Another direction to pursue is to explore our approaches on other graph-based recommenders, particularly those that use deep learning and messagepassing [25, 26] to learn users’ preferences, e.g.,