A significant amount of past literature has shown that it is dificult to leverage plain-text reviews to improve recommendation efectiveness. Since then, Large Language Models (LLMs) have shown unprecedented ability to capture natural language semantics, which has been applied to multiple domains with good results. However, re-purposing them for recommendation is not straightforward, due to their high computational cost and the risk of hallucinations. For these reasons, rather than using LLMs as models to directly generate recommendations, we investigate if LLM embeddings of plain-text reviews can be a useful input to improve the quality of traditional review-based recommendation algorithms, by adapting their architecture to process said embeddings rather than word-level ones. We structure an empirical analysis using two Amazon Review Datasets and three LLMs to produce embeddings: OpenAI, Wang's Mistral and VoyageAI. The results show that LLM embeddings can be efectively used in review-based models developed for word-level embeddings, yet one baseline model still achieves greater accuracy.
CEUR ceur-ws.org
Recommender systems (RS) are widely adopted to help people navigate the vast and expanding
catalogues of digital platforms, most of which encourage users to also leave reviews for the items
they select. Despite their abundance, said reviews are underutilised by current RS [
Nonetheless, textual reviews hold untapped potential for RS, since many users rely on them in their browsing. It can be argued that the shortcomings of previous integration attempts are related to Natural Language Understanding, a field in which recent advances, particularly through Large Language Models (LLMs), have been noteworthy. LLMs are transformer-based neural architectures with billions of trainable parameters. nEvelop-O
There have been several attempts to use LLMs as RS, primarily exploiting them as generative
RS. However, such attempts show a number of limitations, mostly tied to the need for extensive
refactoring of interaction data, the high computational cost of training and fine-tuning LLMs,
and model reliability problems, such as hallucinations [
In this study, leveraging LLM-generated embeddings from plain-text product reviews, we investigate integrating the semantic capabilities of LLMs with traditional review-based RS. We then compare the efectiveness of existing algorithms using word-level embeddings to those using LLM-generated embeddings.
Chen et al. [
HFT is a rating weighting RS based on the combination of two techniques: a classic global
efects-aware matrix factorisation (MF) model [
NARRE [
HRDR [
We adapted NARRE and HRDR to use LLM embeddings of reviews as input to compare their efectiveness when using word-level embeddings versus LLM embeddings. Originally, their input would be a matrix of word embeddings representing a single review, which was aggregated into a single vector through the use of a CNN module. As LLM-based embeddings already represent the reviews in whole, such CNN module was eliminated in their adapted versions, which we respectively named NARRE-LLM and HRDR-LLM. HFT was considered as a baseline model and not adapted in its architecture, since it does not incorporate review embeddings, but instead exploits LDA to do review topic modelling. We also slightly modified all the models, including HFT, in their training process, in order to evaluate them on top- recommendation tasks instead of rating prediction. For NARRE, HRDR, NARRE-LLM and HRDR-LLM, we changed the loss function from squared loss to Bayesian Personalised Ranking (BPR). The HFT algorithm defines a custom loss function which we did not change; however, for the top- recommendation task, the model needs to distinguish positive user-item interactions from negative ones, i.e., those that did not occur, associated to a rating value of 0. Thus, our version of HFT samples negative interactions during training, with uniform probability defined by a hyperparameter.
We use two datasets, consisting of diferent categories of the Amazon Reviews Dataset [
The plain-text reviews contained in both datasets were embedded using multiple LLMs, selected from the MTEB Leaderboard [12]: OpenAI’s text-embedding-ada-0021, Wang’s e5mistral-7b-instruct [13], and VoyageAI’s voyage-lite-02-instruct2. While Wang’s and VoyageAI’s embedders were chosen for their high position within the leaderboard, OpenAI’s embedder was selected for its widespread use.
The models were trained iteratively through stochastic gradient descent, using Bayesian Optimization [14, 15] to optimize hyperparameters with respect to the NDCG@10 metric over the validation set. To mitigate the risk of overfitting, we employ early-stopping, performing an evaluation over the validation set every 5 epochs and terminating the training if NDCG@10 has not improved throughout the 5 latest evaluations. We evaluated the models on the top- recommendation task with NDCG at cutof 10. We also report two beyond-accuracy metrics to measure how the recommendations are distributed: Item Coverage, which measures the portion of items in the catalogue that were recommended at least once, and Item Coverage Hit, which represents the portion of items that were correctly recommended at least once.
Evaluation results are shown in Table 1. For both datasets, using the LLM embeddings as input for HRDR and NARRE resulted in visible improvements in recommendation efectiveness. NARRE-LLM improved up to 47.2% in NDCG with respect to NARRE over the Amazon Music Dataset, and up to 100.6% over the Amazon Fine Foods Dataset, employing embeddings from OpenAI and Wang, respectively. Similarly, HRDR-LLM outperforms HRDR by up to 25.5% and 5.84% over the two datasets, employing embeddings from VoyageAI and OpenAI. All models also visibly improve with respect to Item Coverage and Item Coverage Hit, over both datasets. In particular, NARRE-LLM achieves the best Item Coverage when using Wang’s embeddings, while HRDR-LLM’s Item Coverage improves the most when using OpenAI’s embeddings. 1https://openai.com/blog/new-and-improved-embedding-model 2https://docs.voyageai.com/docs/embeddings
Nevertheless, the HFT model, when trained with implicit interaction data and negative interaction sampling, achieved the best NDCG values. It outperforms HRDR-LLM by a margin of 11.02% over the Amazon Music Dataset and by 4.86% over the Amazon Fine Foods Dataset. NARRE-LLM shows the poorest recommendation accuracy, being outperformed by HFT by a margin of 67.95% when tested on the Amazon Music Dataset, and of 136.8% over the Amazon Fine Foods Dataset. HFT performs solidly also in terms of both Item Coverage and Item Coverage Hit. In terms of Item Coverage, HRDR-LLM with OpenAI embeddings is the best performer over the Music Dataset, improving HFT’s baseline by 5.33%; over the Fine Foods Dataset, HFT performs the best, being 16.44% better than HRDR-LLM with OpenAI embeddings. In terms of Item Coverage Hit, HFT is on top over both datasets, while HRDR-LLM with OpenAI embeddings is second best in both cases.
Although LLM embeddings are indeed an improvement over the ones used by NARRE and
HRDR, exploiting reviews as means for regularization like HFT still appears the better choice.
Since HFT only exploits reviews through LDA-based topic modelling, thus not making use of
review embeddings at all, its recommendation accuracy also comes at a lower computational
cost. These findings on top- accuracy are consistent with those of [
Possible continuations to our investigation might include the use of review embeddings for item feature extraction or feature weighting, and the development of RS specifically engineered to exploit LLM embeddings of textual information as input data, possibly going beyond reviews. [12] N. Muennighof, N. Tazi, L. Magne, N. Reimers, MTEB: Massive text embedding benchmark, in: A. Vlachos, I. Augenstein (Eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, Association for Computational Linguistics, 2023, pp. 2006–2029. doi:10. 18653/V1/2023.EACL-MAIN.148. [13] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, F. Wei, Improving text embeddings with large language models, CoRR abs/2401.00368 (2024). doi:10.48550/ARXIV.2401.00368. arXiv:2401.00368. [14] P. I. Frazier, Bayesian Optimization, in: E. Gel, L. Ntaimo, D. Shier, H. J. Greenberg (Eds.), Recent Advances in Optimization and Modeling of Contemporary Problems, INFORMS, 2018, pp. 255–278. doi:10.1287/educ.2018.0188. [15] M. Ferrari Dacrema, S. Boglio, P. Cremonesi, D. Jannach, A troubling analysis of reproducibility and progress in recommender systems research, ACM Transactions on Information Systems 39 (2021) 20:1–20:49. URL: https://doi.org/10.1145/3434185. doi:10. 1145/3434185. [16] N. Cecere, A. Pisani, M. Ferrari Dacrema, P. Cremonesi, Leveraging semantic embeddings of user reviews with of-the-shelf llms for traditional recommender systems, in: E. Maddalena, S. Mizzaro, K. Roitero, M. Viviani (Eds.), IIR2024: 14th Italian Information Retrieval Workshop, 5th - 6th September 2024, Udine, Italy, CEUR-WS.org, 2024.