I present a hybrid matrix factorisation model representing users and items as linear combinations of their content features' latent factors. The model outperforms both collaborative and content-based models in cold-start or sparse interaction data scenarios (using both user and item metadata), and performs at least as well as a pure collaborative matrix factorisation model where interaction data is abundant. Additionally, feature embeddings produced by the model encode semantic information in a way reminiscent of word embedding approaches, making them useful for a range of related tasks such as tag recommendations.
Building recommender systems that perform well in coldstart scenarios (where little data is available on new users and items) remains a challenge. The standard matrix factorisation (MF) model performs poorly in that setting: it is di cult to e ectively estimate user and item latent factors when collaborative interaction data is sparse.
Content-based (CB) methods address this by representing
items through their metadata [
Copyright remains with the authors and/or original copyright holders.
At Lyst, solving these problems is crucial. We are a fashion company aiming to provide our users with a convenient and engaging way to browse|and shop|for fashion online. To that end we maintain a very large product catalogue: at the time of writing, we aggregate over 8 million fashion items from across the web, adding tens of thousands of new products every day.
Three factors conspire to make recommendations challenging for us. Firstly, our system contains a very large number of items. This makes our data very sparse. Secondly, we deal in fashion: often, the most relevant items are those from newly released collections, allowing us only a short window to gather data and provide e ective recommendations. Finally, a large proportion of our users are rsttime visitors: we would like to present them with compelling recommendations even with little data. This combination of user and item cold-start makes both pure collaborative and content-based methods unsuitable for us.
To solve this problem, I use a hybrid content-collaborative model, called LightFM due to its resemblance to factorisation machines (see Section 3). In LightFM, like in a collaborative ltering model, users and items are represented as latent vectors (embeddings). However, just as in a CB model, these are entirely de ned by functions (in this case, linear combinations) of embeddings of the content features that describe each product or user. For example, if the movie `Wizard of Oz' is described by the following features: `musical fantasy', `Judy Garland', and `Wizard of Oz', then its latent representation will be given by the sum of these features' latent representations.
In doing so, LightFM unites the advantages of contentbased and collaborative recommenders. In this paper, I formalise the model and present empirical results on two datasets, showing that: 1. In both cold-start and low density scenarios, LightFM performs at least as well as pure content-based models, substantially outperforming them when either (1) collaborative information is available in the training set or (2) user features are included in the model. 2. When collaborative data is abundant (warm-start, dense user-item matrix), LightFM performs at least as well as the MF model. 3. Embeddings produced by LightFM encode important semantic information about features, and can be used for related recommendation tasks such as tag recommendations.
This has several bene ts for real-world recommender systems. Because LightFM works well on both dense and sparse data, it obviates the need for building and maintaining multiple specialised machine learning models for each setting. Additionally, as it can use both user and item metadata, it has the quality of being applicable in both item and user cold-start scenarios.
To allow others to reproduce the results in this paper, I have released a Python implementation of LightFM1, and made the source code for this paper and all the experiments available on Github2. 2.1
The structure of the LightFM model is motivated by two considerations.
1. The model must be able to learn user and item representations from interaction data: if items described as `ball gown and `pencil skirt' are consistently all liked by users, the model must learn that ball gowns are similar to pencil skirts. 2. The model must be able to compute recommendations for new items and users.
I ful l the rst requirement by using the latent representation approach. If ball gowns and pencil skirts are both liked by the same users, their embeddings will be close together; if ball gowns and biker jackets are never liked by the same users, their embeddings will be far apart.
Such representations allow transfer learning to occur. If the representations for ball gowns and pencil skirts are similar, we can con dently recommend ball gowns to a new user who has so far only interacted with pencil skirts.
This is over and above what pure CB models using dimensionality reduction techniques (such as latent semantic indexing, LSI) can achieve, as these only encode information given by feature co-occurrence rather than user actions. For example, suppose that all users who look at items described as aviators also look at items described as wayfarers, but the two features never describe the same item. In this case, the LSI vector for wayfarers will not be similar to the one for aviators even though collaborative information suggests it should be.
I ful l the second requirement by representing items and users as linear combinations of their content features. Because content features are known the moment a user or item enters the system, this allows recommendations to be made straight away. The resulting structure is also easy to understand. The representation for denim jacket is simply a sum of the representation of denim and the representation of jacket; the representation for a female user from the US is a sum of the representations of US and female users. 2.2
To describe the model formally, let U be the set of users, I be the set of items, F U be the set of user features, and F I the set of item features. Each user interacts with a number of items, either in a favourable way (a positive interaction),
or in an unfavourable way (a negative interaction). The set of all user-item interaction pairs (u; i) 2 U I is the union of both positive S+ and negative interactions S .
Users and items are fully described by their features. Each user u is described by a set of features fu F U . The same holds for each item i whose features are given by fi F I . The features are known in advance and represent user and item metadata.
The model is parameterised in terms of d-dimensional user and item feature embeddings efU and eIf for each feature f . Each feature is also described by a scalar bias term (bfU for user and bIf for item features).
The latent representation of user u is given by the sum of its features' latent vectors:
qu = pi = bu = bi =
X ej
U j2fu X ej
I j2fi X bjU j2fu X bI
j j2fi
The model's prediction for user u and item i is then given
by the dot product of user and item representations,
adjusted by user and item feature biases:
rbui = f (qu pi + bu + bi)
(1)
There is a number of functions suitable for f ( ). An identity
function would work well for predicting ratings; in this
paper, I am interested in predicting binary data, and so after
Rendle et al. [
1 1 + exp( x) :
The optimisation objective for the model consists in maximising the likelihood of the data conditional on the parameters. The likelihood is given by
L eU ; eI ; bU ; bI =
Y (u;i)2S+ rbui
Y (u;i)2S (1 rbui) (2)
I train the model using asynchronous stochastic gradient
descent [
The relationship between LightFM and the collaborative MF model is governed by the structure of the user and item feature sets. If the feature sets consist solely of indicator variables for each user and item, LightFM reduces to the standard MF model. If the feature sets also contain metadata features shared by more than one item or user, LightFM extends the MF model by letting the feature latent factors explain part of the structure of user interactions.
This is important on three counts. 1. In most applications there will be fewer metadata features than there are users or items, either because an ontology with a xed type/category structure is used, or because a xed-size dictionary of most common terms is maintained when using raw textual features. This means that fewer parameters need to be estimated from limited training data, reducing the risk of over tting and improving generalisation performance. 2. Latent vectors for indicator variables cannot be estimated for new, cold-start users or items. Representing these as combinations of metadata features that can be estimated from the training set makes it possible to make cold-start predictions. 3. If only indicator features are present, LightFM should perform on par with the standard MF model.
When only metadata features and no indicator variables are present, the model in general does not reduce to a pure content-based system. LightFM estimates feature embeddings by factorising the collaborative interaction matrix; this is unlike content-based systems which (when dimensionality reduction is used) factorise pure content co-occurrence matrices.
One special case where LightFM does reduce to a pure CB model is where each user is described by an indicator variable and has interacted only with one item. In that setting, the user vector is equivalent to a document vector in the LSI formulation, and only features which occur together in product descriptions will have similar embeddings.
The fact that LightFM contains both the pure CB model at the sparse data end of the spectrum and the MF model at the dense end suggests that it should adapt well to datasets of varying sparsity. In fact, empirical results show that it performs at least as well as the appropriate specialised model in each scenario.
There are a number of related hybrid models attempting to solve the cold-start problem by jointly modelling content and collaborative data.
Soboro et al. [
Saveski et al. [
Shmueli et al. [
LightFM ts into the hybrid model tradition by jointly
factorising the user-item, item-feature, and user-feature
matrices. From a theory standpoint, it can be construed as a
special case of Factorisation Machines [
FMs provide an e cient method of estimating variable interaction terms in linear models under sparsity. Each variable is represented by a k-dimensional latent factor; the interaction between variable i and j is then given by the dot product of their latent factors. This has the advantage of reducing the number of parameters to be estimated.
LightFM further restricts the interaction structure by only estimating the interactions between user and item features. This aids the interpretability of resulting feature embeddings. 4.
I evaluate LightFM's performance on two datasets. The datasets span the range of dense interaction data, where MF models can be expected to perform well (MovieLens), and sparse data, where CB models tend to perform better (CrossValidated). Both datasets are freely available. 4.1
The rst experiment uses the well-known MovieLens 10M
dataset3, combined with the Tag Genome tag set [
The dataset consists of approximately 10 million movie ratings, submitted by 71; 567 users on 10; 681 movies. All movies are described by their genres and a list of tags from the Tag Genome. Each movie-tag pair is accompanied by a relevance score (between 0 and 1), denoting how accurately a given tag describes the movie.
To binarise the problem, I treat all ratings below 4:0 (out of a 1 to 5 scale) as negative; all ratings equal to or above 4:0 are positive. I also lter out all ratings that fall below the 0:8 relevance threshold to retain only highly relevant tags.
The nal dataset contains 69; 878 users, 10; 681 items, 9; 996; 948 interactions, and 1030 unique tags. 4.2
The second dataset consists of questions and answers posted on CrossValidated4, a part of the larger network of StackExchange collaborative Q&A sites that focuses on statistics and machine learning. The dataset5 consists of 5953 users, 44; 200 questions, and 188; 865 answers and comments. Each question is accompanied by one or more of 1032 unique tags (such as `regression' or `hypothesis-testing'). Additionally,
user metadata is available in the form of `About Me' sections on users' pro les.
The recommendation goal is to match users with questions they can answer. A user answering a question is taken as an implicit positive signal; all questions that a user has not answered are treated as implicit negative signals. For the training and test sets, I construct 3 negative training pairs for each positive user-question pair by randomly sampling from all questions that a given user has not answered.
To keep the model simple, I focus on a user's willingness
to answer a question rather than their ability, and forego
modelling user expertise [
For each dataset, I perform two experiments. The rst simulates a warm-start setting: 20% of all interaction pairs are randomly assigned to the test set, but all items and users are represented in the training set. The second is an item cold-start scenario: all interactions pertaining to 20% of items are removed from the training set and added to the test set. This approximates a setting where the recommender is required to make recommendations from a pool of items for which no collaborative information has been gathered, and only content metadata (tags) are available.
I measure model accuracy using the mean receiver operating characteristics area under the curve (ROC AUC) metric. For an individual user, AUC corresponds to the probability that a randomly chosen positive item will be ranked higher than a randomly chosen negative item. A high AUC score is equivalent to low rank-inversion probability, where the recommender mistakenly ranks an unattractive item higher than an attractive item. I compute this metric for all users in the test set and average it for the nal score.
I compute the AUC metric by repeatedly randomly splitting the dataset into a 80% training set and a 20% test set. The nal score is given averaging across 10 repetitions.
I test the following models:
1. MF: a conventional matrix factorisation model with
user and item biases and a sigmoid link function [
In both LightFM (tags) and LightFM (tags + ids) users are described only by indicator features.
I train the LightFM models using stochastic gradient descent with an initial learning rate of 0.05. The latent dimensionality of the models is set to 64 for all models and experiments. This setting is intended to re ect the balance between model accuracy and the computational cost of larger vectors in production systems (additional results on model sensitivity to this parameter are presented in Section 6.2). I regularise the model through an early-stopping criterion: the training is stopped when the model's performance on the test set stops improving. 6. 6.1
Experimental results are summarised in Table 1. LightFM performs very well, outperforming or matching the specialised model for each scenario.
In the warm-start, low-sparsity case (warm-start MovieLens), LightFM outperforms MF slightly when using both tag and item indicator features. This suggest that using metadata features may be valuable even when abundant interaction data is present.
Notably, LightFM (tags) almost matches MF performance despite using only metadata features. The LSI-LR and LSIUP models using the same information fare much worse. This demonstrates that (1) it is crucial to use collaborative information when estimating content feature embeddings, and (2) LightFM can capture that information much more accurately than other hybrid models such as LSI-UP.
In the warm-start, high-sparsity case (warm-start CrossValidated), MF performs very poorly. Because user interaction data is sparse (the CrossValidated user-item matrix is 99.95% sparse vs only 99% for the MovieLens dataset), MF is unable to learn good latent representations. Content-based models such as LSI-LR perform much better.
LightFM variants provide the best performance. LightFM (tags + about) is by far the best model, showing the added advantage of LightFM's ability to integrate user metadata embeddings into the recommendation model. This is likely due to improved prediction performance for users with little data in the training set.
Results for the cold-start cases are broadly similar. On the CrossValidated dataset, all variants of LightFM outperform other models; LightFM (tags + about) again provides the best performance. Interestingly, LightFM (tags + indicators) outperforms LightFM (tags) slightly on the MovieLens dataset, even though no embeddings can be estimated for movies in the test set. This suggests that using both metadata and per-movie features allows the model to estimate better embeddings for both, much like the use of user and item bias terms allows better latent factors to be computed. Unsurprisingly, MF performs no better than random in the cold-start case.
In all scenarios the LSI-UP model performs no better than the LSI-LR model, despite its attempt to incorporate collaborative data. On the CrossValidated dataset it performs strictly worse. This might be because its latent representations are estimated on less data than in LSI-LR: as there are fewer users than items in the dataset, there are fewer rows in the user-feature matrix than in the item-feature matrix.
The results con rm that LightFM encompasses both the MF and the LSI-LR model as special cases, performing better than the LSI-LR model in the sparse-data scenario and better than the MF model in the dense-data case. This means not only that a single model can be maintained in either settings, but also that the model will continue to perform well even when the sparsity structure of that data changes.
Good performance of LightFM (tags) in both datasets
is predicated on the availability of high-quality metadata.
Nevertheless, it is often possible to obtain good quality
metadata from item descriptions (genres, actor lists and so on),
expert or community tagging (Pandora [
dimensional LSI-LR model even when using fewer than 32 dimensions.
This is an important win for large-scale recommender systems, where the choice of d is governed by a trade-o between vector size and recommendation accuracy. Since smaller vectors occupy less memory and use fewer computations during query time, better representational power at small d allows the system to achieve the same model performance at a smaller computational cost. 6.3
Feature embeddings generated by the LightFM model capture important information about the semantic relationships between di erent features. Table 2 gives some examples by listing groups of tags similar (in the cosine similarity sense) to a given query tag.
In this respect, LightFM is similar to recent word
embedding approaches like word2vec and GloVe [
LightFM embeddings are useful for a number of recommendation tasks.
1. Tag recommendation. Various applications use
collaborative tagging as a way of generating richer
metadata for use in search and recommender system [
The LightFM approach is motivated by our experience at Lyst. We have deployed LightFM in production, and successfully use it for a number of recommendation tasks. In this section, I describe some of the engineering and algorithm choices that make this possible. 7.1
Thousands of new items and users appear on Lyst every day. To cope with this, we train our LightFM model in an online manner, continually updating the representations of existing features and creating fresh representations for features that we have never observed before.
We store model state, including feature embeddings and accumulated squared gradient information in a database. When new data on user interaction arrives, we restore the model state and resume training, folding in any newly observed features. Since our implementation uses per-parameter diminishing learning rates (Adagrad), any updates of established features will be incremental as the model adapts to new data. For new features, a high learning rate is used to allow useful embeddings to be learned as quickly as possible.
No re-training is necessary for folding in new products: their representation can be immediately computed as the sum of the representations of their features.
Each of our products is described by a set of textual features as well as structured metadata such as its type (dress, shoes and so on) or designer. These are accompanied by additional features coming from two sources.
Firstly, we employ a team of experienced fashion moderators, helping us to derive more ne-grained features such as clothing categories and subcategories (peplum dress, halterneck and so on).
Secondly, we use machine learning systems for automatic feature detection. The most important of these is a set of deep convolutional neural networks deriving feature tags from product image data. 7.3
The biggest application of LightFM-derived item representations are related product recommendations: given a product, we would like to recommend other highly relevant products. To do this e ciently across 8 million products, we use a combination of approximate (for on-demand recommendations) and exact (for near-line computation) nearest neighbour search.
For approximate nearest neighbour (ANN) queries, we use
Random Projection (RP) trees [
In LSH, k-bit hash codes for each point x are generated by drawing random hyperplanes v, and then setting the k-th bit of the hash code to 1 if x v 0 and 0 otherwise. The approximate nearest neighbours of x are then other points that share the same hash code (or whose hash codes are within some small Hamming distance of each other).
While extremely fast, LSH has the undesirable property of sometimes producing very highly unbalanced distribution of points across all hash codes: if points are densely concentrated, many codes of the tree will apply to no products while some will describe a very large number of points. This is unacceptable when building a production system, as it will lead to many queries being very slow.
RP trees provide much better guarantees about the size of leaf nodes: at each internal node, points are split based on the median distance to the chosen random hyperplane. This guarantees that at every split approximately half the points will be allocated to each leaf, making the distribution of points (and query performance) much more predictable.
In this paper, I have presented an e ective hybrid recommender model dubbed LightFM. I have shown the following: 1. LightFM performs at least as well as a specialised model across a wide range of collaborative data sparsity scenarios. It outperforms existing content-based and hybrid models in cold-start scenarios where collaborative data is abundant or where user metadata is available. 2. It produces high-quality content feature embeddings that capture important semantic information about the problem domain, and can be used for related tasks such as tag recommendations.
Both properties make LightFM an attractive model, applicable both in cold- and warm-start settings. Nevertheless, I see two promising directions in extending the current approach.
Firstly, the model can be easily extended to use more
sophisticated training methodologies. For example, an
optimisation scheme using Weighted Approximate-Rank Pairwise
loss [
Secondly, there is no easy way of incorporating visual or audio features in the present formulation of LightFM. At Lyst, we use a two-step process to address this: we rst use convolutional neural networks (CNNs) on image data to generate binary tags for all products, and then use the tags for generating recommendations. We conjecture that substantial improvements could be realised if the CNNs were trained with recommendation loss directly.