User-user similarities in recommender systems are traditionally assessed on co-rated items. As ratings encode item preferences, similarities on co-rated items capture similarities in item preferences. However, a majority of similarities are undefined as particularly small profiles seldom overlap. We propose to use a similarity measure based on users' item reviews in order to estimate similarities in item preferences in the absence of co-rated items. Although it is commonly believed that item reviews are descriptive of a user's item preferences, it is not clear whether indeed and what about a user's item preferences item reviews describe. We present empirical results indicating that the proposed review-based similarity measure captures features in users' item preferences that are diferent from those captured on co-rated items. Astonishingly, we find that 10
In order to better understand what about users’ item preferences is reflected in reviews, we propose a similarity measure that compares users on the basis of their item reviews. We report findings on our pilot experi1. Concatenate Reviews
2. Drop Stop Words 3. Extract Weighted Keywords (YAKE)
4. Map Keywords to Word Vectors 5. Calculate Word Mover’s Distance (WMD) 6. Transform WMD to Similarity Measure + + … + = + + … + = “awesome” : 0.65 “functional” : 0.35 “performance” : 0.8 “usability” : 0.2 s: [0.51, -0.04,.., 0.91] : 0.6 [1.12, 0.46,.., -0.85] : 0.4 t: [0.11, 0.59,.., 0.01] : 0.8
[3.41, 0.66,.., -0.33] : 0.2
WMD(s,t) simYAKE & WMD(u,v) ments indicating that the proposed similarity measure (a) indeed captures similarity in users’ item preferences, and (b) captures features that are diferent from those captured by co-rated-items-based similarity measures.
Independently from the above results, we find that the design of the proposed review-based similarity measure motivates its use in decentralized recommender systems for three design properties. First, it can be calculated between any pair of users who hold item reviews. Second, it can be calculated bilaterally without involvement of a third party. And third, it does not require to reveal a user’s plain review text.
sure proposed by Eichinger et al. [13]. It has originally been proposed as a general-purpose similarity measure on texting data. In the paper at hand, we instead apply it to item reviews and show that it particularly captures similarity in users’ item preferences.
Similarity comparison can be summarized as a six-step approach as shown in Figure 1. We first elaborate on Steps 4.-6. in Section 2.1, which constitute the core of the similarity measure. Afterwards in Section 2.2, we focus on optional steps such as text preprocessing and keyword extraction comprising Steps 1.-3. Eichinger et al. originally proposed keyword extraction on the basis of tf-idf features. In contrast to the original work, we instead apply a state-of-the-art keyword extractor, which additionally allows users to run keyword extraction independently from other users.
4. Map Keywords to Word Vectors (Figure 1): Kusner et al. [14] propose the Word Mover’s Distance (WMD), a distance metric between text documents that are each represented by a subset of their words.1 The WMD is made such that text documents that hold semantically similar words – and thus not necessarily the same words – are close. Semantic similarity between words is captured by word embeddings. Word embeddings map words to word vectors such that word vectors of semantically similar words are close. Words need not necessarily be keywords. Note that all users need to use the same word embedding model, wherefore we use a publicly available pre-trained word embedding model. 5. Calculate Word Mover’s Distance (Figure 1): The WMD leverages word vectors that condense semantic similarity
(EMD), where the EMD is in turn a special Wasserstein metric. between single words, in order to measure semantic WMD has not found wide adoption. Eforts to lower the similarity between sets of words. More precisely, the computational complexity include approximation [16, 17] WMD compares so-called signatures.2 Signatures are and the reduction of the signature size by keyword exsets of word vectors in which every word vector is traction [13]. In a previous paper, we applied keyword associated with a word weight. The number of word extraction on the basis of the tf-idf word relevance meavectors in a signature is called the signature size. The sure [13]. Note that keyword extraction via tf-idf requires distance between two signatures, associated with to keep track of the global usage of terms in all users’ the distance between two text documents, is then reviews. A more convenient alternative is Yet Another determined by solving a transportation problem (see Keyword Extractor (YAKE) by Campos et al. [18, 19]. [14] for details). The WMD can be calculated bilaterally Their keyword extractor is document-based and works and independently of other users upon the exchange of on textual features of single documents. It does not resignatures. quire information on other documents. YAKE is a weighted keyword extractor.5 It attaches 6. Transform WMD to Similarity Measure (Figure 1): We positive keyword weights > 0 to every keyword of transform the WMD distance metric into a similarity a text document. Keywords in YAKE are considered more measure. Note that the WMD distance between two simi- important in describing their underlying text document lar text documents is close to zero, whereas dissimilar text the smaller their associated keyword weights are. documents may yield arbitrarily large WMD distances. Conversely, WMD word weights are considered more Hence, we first limit the co-domain to WMD (, ) ∈ important the larger they are. We therefore reverse the [0, 2] for any pair of signatures and . We do so by order of the keyword weights for use as word weights using the cosine distance3 to measure distances between in the WMD. We do so via the linear transformation word vectors and normalize word weights in a signature such that they sum to 1.4 We then obtain a similarity measure upon the following linear transformation: defined by := + − ∈ [, ], and consecutive normalization of the word weights such that they sum to 1, where and are the minimum and maximum keyword weights respectively.
1 WMD(, ) := 1 − 2
WMD(, ) ∈ [0, 1].
The signature size is the sole hyperparameter of the transformation on a user’s item reviews yields signatures WMD, and thus also of the associated similarity mea- that serve as input to the WMD. We denote by YAKE() sure WMD. We will specify the signature size where the thus associated signature of some user ’s item rerequired, yet omit it in the notation for reasons of brevity. views. Combining this with the results of Section 2.1, we can now define the review-based similarity measure as 2.2. Key Word Extraction 1. Concatenate Reviews (Figure 1): In order to arrive at a user-specific text document, we first concatenate all item reviews authored by a user in arbitrary order with blanks between reviews.
review(, ) := WMD(YAKE(), YAKE()), where and are some users that hold item reviews. 3. Evaluation 2. Drop Stop Words (Figure 1): We drop stop words as a We present results that contrast the cosine similarity as basic text preprocessing step. We do not perform any a traditional co-rated-items-based similarity measure further preprocessing in order to mitigate the impact with the proposed review-based similarity measure due to preprocessing on the evaluation of the proposed review. We emphasize that the goal we pursue by review-based similarity measure. this comparison is not to argue that the review-based similarity measure is superior to co-rated-items-based 3. Extract Weighted Keywords (Figure 1): The computa- similarity measures. Instead, it is our goal to find a solid tional complexity of the WMD is often prohibitive as indication that the review-based similarity measure it is supercubic in the signature size. For this reason, the indeed captures similarity in terms of item preferences between users. We employ the rationale, that if the review-based similarity measure captures similarity 2The term signature has been coined by Rubner et al. [15] in the domain of computer vision as abstractions of color histograms.
3cos(, ) = 1 − ‖<‖·,‖>‖ , where < · , · > denotes the dot product and ‖·‖ the Euclidean norm.
4If the Euclidean distance is preferred, we can alternatively normalize vectors to length 1 and normalize word weights such that they sum to 1.
keywords and omit treatment of keyphrases for reasons of simplicity. Although it is also possible to convert keyphrases into vectors, distinct scientific reasoning is required to justify a comparison between signatures that combine word vectors and phrase vectors. between users’ item preferences, then it necessarily must sample. Some descriptive statistics are shown in Table perform well in user-based Collaborative Filtering (CF). 1. Note that, if similarity is measured on the basis of co-rated items, only 0.03% and 0.01% of all pairwise
We calculate review with the help of the follow- similarites can be calculated for users in the Medianing software contributions.6 We use Pele and Werman’s 100 and Low-100 data sets respectively. As we compare Python implementation of the WMD [20].7 We use Bo- review-based with co-rated-items-based similarity meajanowski et al.’s publicly available pre-trained fasttext sures, we omit an analysis on the samples Median-100 word embedding model cc.en.300.bin [21].8 We use the and Low-100 as they simply provide too little ground for stop word list provided by Bird’s The Natural Language comparison.
Toolkit (NLTK) [22]. Finally, we use Campos et al.’s Python library yake [18].9 3.2. Baselines
Splits into training and test sets are at a ratio of 80 to 20, where particularly every user’s entries are split into We apply the following standard mean-centered rating portions of training and test entries. We report average estimation equation for user-based CF: results over 5 distinct training-test splits on the usual Root Mean Squared Error (RMSE) accuracy metric. ∑︁ ˆ, = ¯ + ∈, (, )(, − ¯) , ∑︀∈, (, ) (1)
where ˆ, denotes an estimated rating for user We present results on two small samples of the Amazon on item , ¯ the mean rating of user , , some Reviews 5-core (2014) data set [23].10 The original data set neighborhood of users of user that have rated item , holds roughly 41 million entries on 24 product domains, a user-user similarity measure, and , the true where every user has at least 5 rating-review pairs. The rating of user on item . For reasons of brevity we say two samples considered in the paper at hand cover two that a similarity measure outperforms another, when in distinct scenarios of (a) an artificially high and (b) a more fact we mean that rating estimation as per Equation (1) realistic density. We now describe their construction. equipped with the one similarity measure outperforms
We draw the first data set Head-100 by selecting the that equipped with the other. 100 largest user profiles. It simulates an artificially high density of ratings and reviews with particularly large amounts of review text per user. As for the second data set Mix-100, we draw 2 additional data sets Median-100 and Tail-100, of medium and low density, by selecting the profiles of 100 median and 100 tail users respectively.
We finally sample Mix-100 from the datasets Head-100, Median-100, and Tail-100 at a ratio of 33 to 34 to 33. We construct Mix-100 in this way in order to guarantee the presence of large, medium-sized, and small profiles in the
comparison with review. First, cosine similarity as a similarity measure based on mutually rated items. And second, a simple arithmetic mean having equal similarity weights mean(, ) = 1/|,| for all users ∈ ,. If review outperforms mean, it is an indication that review does capture similarity in item preference between users, that is more than an estimate without prior knowledge on reviews. If further review outperforms cosine, it is an indication that the review-based similarity measure captures similarity in item preference at least on a par with co-rated-items-based similarity measures.
7https://pypi.org/project/pyemd/ 8https://fasttext.cc/docs/en/pretrained-vectors.html 9https://pypi.org/project/yake/ 10https://doi.org/10.7910/DVN/V7X3VE review cosine 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 similarity similarity
We present findings that indicate that the review-based
similarity measure review captures similarity in item
preference. We first constrast the statistical properties of
review and cosine, where we assume that cosine
already captures some similarity in item preference. We
then measure their respective impact on rating estimation
as per Equation (1), acting in the role of (a) similarity
weights, and (b) a neighborhood selection criterion. In
order to study the impact due to (a) and (b) individually,
we first omit neighborhood selection by setting , as
the set of all other users ̸= and applying review
similarity weights, and then conversely omit
similaritybased weighted averaging by using mean similarity
weights and setting , as the set of users that are
most similar to user with respect to review and have
rated item .
3.3.1. Statistical Properties
both High-100 and Mix-100. In contrast, review’s
distribution follows the shape of a normal distribution in
High-100, and a mix of normal distributions with distinct
modes in Mix-100. Review-based similarities in High-100,
Median-100, and Low-100 seem to have distinct modes
that interfere in Mix-100.
The similarity measures review and cosine capture
distinct aspects of similarity between users’ item
preferences. We find that review and cosine are only Mix-100
weakly positively correlated with respect to the Spear- min 1 2 3 5 10
smMtraioxnn-1rg0al0yn)k.poIcfsoiirtnriveceloalyntitocrnoasr(tr0eb.l3ao6ttehdos,nitmhHiiislgawrhi-to1yu0lm0d,eianansdudicr0eas.t2ew5theoranet rcmeovesiaiennwe 111...000888754 111...000321329* 000...999988785* 000...999380285 –––
both similarity measures capture similar aspects of
similarity. In that case, we would also expect that both
similarity measures yield similar recommendation performance.
disWtinecftusritmhielra rfinitdytdhiasttributiornevsie.wFiagnudre2shcoosiwnespthroedsuimce- 3.3.2. Similarity-based Weighted Averaging
ilarity distributions in the data sets High-100 and Mix- We find that review similarity weights provide
signif100. We observe that the cosine similarity’s distribu- icantly better recommendation performance, if profiles
tion follows the shape of an exponential distribution in are not exclusively large such as in Mix-100. If in
conmean
review
cosine
1.2
trast, profiles are exclusively large such as in Head-100, Item Reviews for Item Recommendation: There
review similarity weights, cosine similarity weights, is a wealth of work that aims to leverage item
reand mean similarity weights perform similarly as views in order to improve recommendation performance.
shown in Table 2. None performs significantly better on Sachdeva and McAuley [12] recently presented a review
Head-100. On Mix-100 however, review significantly of state-of-the-art recommender algorithms that leverage
outperforms the alternatives for min ∈ {2, 3}. For review data. They categorize them into two tracks. First,
min = 5, superiority is not statistically significant de- algorithms that use reviews for regularization at
algospite the large absolute margin due to review’s high rithm training time [24, 25]. And second, algorithms that
empirical standard deviation. use review-based features for use at recommendation
time [
We apply a fasttext word embedding model since it can
leveraging review text for recommendation in general. Second, estimating similarity in item preference without using ratings. And third, alternatives to the proposed YAKE keyword extractor and fasttext word embedding models for use in review. map word tokens that have not been seen at training time by leveraging subword information [21]. An alternative that also leverages subword information is for instance LexVec [35]. A majority of word embedding models does not leverage subword information and can thus only map word tokens available at training time [36, 37, 38, 39].
We propose a review-based user-user similarity measure that presents an alternative to traditional co-rated-itemsbased similarity measures. It is particularly suitable, if two users do not have co-rated items and would thus default to an undefined user-user similarity. Similarities can now be calculated on the basis of item reviews instead of co-rated items.
We find that the proposed review-based similarity measure captures similarity in users’ item preferences. Interestingly, the proposed review-based similarity measure captures diferent features from those captured on corated items. The diference can be linked implicitly to the diference in their statistical features such as similarity distribution and Spearman rank correlation. However, the diference cannot be characterized explicitly as the review-based similarity measure is based on unsupervised word embeddings. More precisely, word embeddings find semantic similarity between words, yet do not tell how and in which sense the words are similar.
We conclude that the proposed review-based user-user similarity measure presents a promising feature for recommender system design, when item reviews are available. We do not argue that the proposed review-based similarity measure is in any sense superior to co-rateditems-based similarity measures. On the contrary, our ifndings indicate that they are complementary in modeling users’ item preferences. ceedings of the 32nd Int. Conf. on Machine Learn- [25] R. Catherine, W. Cohen, Transnets: Learning to ing, volume 37, JMLR.org, 2015, pp. 957–966. doi:10. transform for recommendation, in: Proceedings 5555/3045118.3045221. of the 11th ACM Conference on Recommender [15] Y. Rubner, C. Tomasi, L. Guibas, A metric for dis- Systems, ACM, 2017, pp. 288 – 296. doi:10.1145/ tributions with applications to image databases, in: 3109859.3109878.
Proceedings of the 6th IEEE International Confer- [26] L. Zheng, V. Noroozi, P. S. Yu, Joint deep modeling ence on Computer Vision, IEEE, 1998, pp. 59–66. of users and items using reviews for recommendoi:10.1109/ICCV.1998.710701. dation, in: Proceedings of the 10th ACM Interna[16] M. Cuturi, Sinkhorn distances: Lightspeed compu- tional Conference on Web Search and Data Mining, tation of optimal transport, Advances in Neural ACM, 2017, pp. 425–434. doi:10.1145/3018661. Information Processing Systems 26 (2013) 2292– 3018665. 2300. URL: https://papers.nips.cc/paper/2013/hash/ [27] Y. Bao, H. Fang, J. Zhang, Topicmf: Simultaneously af21d0c97db2e27e13572cbf59eb343d-Abstract. exploiting ratings and reviews for recommendation, html. in: Proceedings of the 28th AAAI Conference on [17] K. Atasu, T. Mittelholzer, Linear-complexity data- Artificial Intelligence, AAAI Press, 2014, pp. 2–8. parallel earth mover’s distance approximations, doi:10.5555/2893873.2893874. in: Proceedings of the 36th International Confer- [28] K. Bauman, A. Tuzhilin, Discovering contexence on Machine Learning, volume 97, PMLR, 2019, tual information from user reviews for recompp. 364–373. URL: http://proceedings.mlr.press/v97/ mendation purposes., in: Proceedings of the atasu19a.html. 1st CBRecSys Workshop @RecSys, CEUR-WS, [18] R. Campos, V. Mangaravite, A. Pasquali, A. Jorge, 2014, pp. 2–9. URL: http://ceur-ws.org/Vol-1245/ C. Nunes, A. Jatowt, Yake! keyword extraction cbrecsys2014-paper01.pdf. from single documents using multiple local features, [29] P. G. Campos, N. Rodríguez-Artigot, I. Cantador, ExInformation Sciences 509 (2020) 257–289. doi:10. tracting context data from user reviews for recom1016/j.ins.2019.09.013. mendation: A linked data approach, in: In Proceed[19] R. Campos, V. Mangaravite, A. Pasquali, A. Jorge, ings of the 1st ComplexRec Workshop @RecSys, C. Nunes, A. Jatowt, Yake! collection- CEUR-WS, 2017, pp. 14–18. URL: http://ceur-ws. independent automatic keyword extractor, in: Ad- org/Vol-1892/paper3.pdf. vances in Information Retrieval, volume 10772 of [30] A. De Spindler, M. C. Norrie, M. Grossniklaus, ColLNCS, Springer, 2018, pp. 806–810. doi:10.1007/ laborative filtering based on opportunistic informa978-3-319-76941-7_80. tion sharing in mobile ad-hoc networks, in: Pro[20] O. Pele, M. Werman, Fast and robust earth mover’s ceedings of the 2007 OTM Conf. Int. Conf., Springer distances, in: Proceedings of the 12th IEEE Interna- Berlin Heidelberg, 2007, pp. 408–416. doi:10.5555/ tional Conference on Computer Vision, IEEE, 2009, 1784607.1784643.
pp. 460–467. doi:10.1109/ICCV.2009.5459199. [31] S. Rose, D. Engel, N. Cramer, W. Cowley, Automatic [21] P. Bojanowski, E. Grave, A. Joulin, T. lov, Enriching keyword extraction from individual documents, in: word vectors with subword information, Trans- Text Mining, John Wiley & Sons Ltd., 2010, pp. 1–20. actions of the Association for Computational Lin- doi:10.1002/9780470689646.ch1. guistics 5 (2017) 135–146. doi:10.1162/tacl_a_ [32] S. R. El-Beltagy, A. Rafea, Kp-miner: A keyphrase 00051. extraction system for english and arabic docu[22] S. Bird, NLTK: The Natural Language Toolkit, in: ments, Information Systems 34 (2009) 132–144.
Proceedings of the COLING/ACL 2006 Interactive doi:10.1016/j.is.2008.05.002. Presentation Sessions, ACL, 2006, pp. 69–72. doi:10. [33] R. Mihalcea, P. Tarau, TextRank: Bringing order 3115/1225403.1225421. into text, in: Proceedings of the 2004 Conference on [23] J. McAuley, C. Targett, Q. Shi, A. van den Hen- Empirical Methods in Natural Language Processing, gel, Image-based recommendations on styles and ACL, 2004, pp. 404–411. URL: https://aclanthology. substitutes, in: Proceedings of the 38th Interna- org/W04-3252. tional ACM SIGIR Conf. on R&D in Information [34] A. Bougouin, F. Boudin, B. Daille, TopicRank: Retrieval, ACM, 2015, pp. 43–52. doi:10.1145/ Graph-based topic ranking for keyphrase extrac2766462.2767755. tion, in: Proceedings of the 6th International [24] J. McAuley, J. Leskovec, Hidden factors and hid- Joint Conference on Natural Language Processden topics: Understanding rating dimensions with ing, AFNLP, 2013, pp. 543–551. URL: https:// review text, in: Proceedings of the 7th ACM Con- aclanthology.org/I13-1062. ference on Recommender Systems, ACM, 2013, pp. [35] A. Salle, A. Villavicencio, Incorporating subword 165–172. doi:10.1145/2507157.2507163. information into matrix factorization word embed