Recent research has shown that language models can be used efectively for recommendation. We are interested in using language models to create natural-language (NL) profiles based on users' listening habits for the purpose of providing accurate, interpretable, and steerable recommendations. In this paper, we explore the usage of these NL profiles for recommendation with both a large commercial language model and a smaller open-source model. We find that, though these methods do not perform as well as traditional recommender systems (e.g., matrix factorization) in terms of accuracy, they do produce meaningful recommendations and provide the user the ability to control their recommendations using natural language.
recommendation interfaces struggle to surface them in a way that users trust and are able to influence.
tions. Building on prior work that characterized the dificulty of recommending long-tail artists and
the additional challenge of contextualizing those music recommendations for users [
Specifically, we investigate whether the natural language (NL) user profiles (see Figure
1)
produced by language models can (1) recover the quality of the recommendation
comparable to more direct approaches that use a list of seed artists as a item-based profile, and
(2) remain interpretable [
CEUR
ceur-ws.org
(LLMs), either as complete recommender systems (using a variety of embedding approaches) [
examine) and steerability (the ability to alter) through the reading and manual editing of these profiles.
techniques like providing examples (one/few-shot prompting), and providing contrasting item sets to
improved the quality in terms of both readability and recommendation quality [
The evaluation of the experiments in this paper was carried out using data from Localify.org1 users who signed up with their Spotify accounts. This data is comprised of the “heavy rotation” artists (artists to which a user listens frequently) of 192 Localify users, and contains 3551 artists. Each user must have at least 10 artists in their heavy rotation, but there is no upper limit on the number of artists in this set2.
During evaluation, each user’s seed artists were split into two groups at random. The first group was
used as their seed artists in the recommendation, and the second group was placed in the candidate
set. A set of artists (i.e. distractors) of the same size as the split seed groups was then taken from the
seed artists of other users and placed in the candidate set as well, and the language model was asked to
rank the candidate set. The score for each recommendation was found by calculating the area under
the ROC curve (AUC) [
as a useful algorithm for local music recommendation, and the algorithm used currently by Localify.
4. Direct Recommendations Without NL Profiles In order to establish a baseline for recommendation performance using language models with our evaluation method, we first generate recommendations by providing a language model directly with a list of seed artists for a user. We then generated recommendations by providing both artist names and their associated genres. The prompt used for these approaches can be seen in B.1 and the results can be seen in Table 1.
of data. This ensures that the sets of user seeds cannot be traced back to the user with which they are associated, even by system administrators.
We ran these experiments on two models; gpt-4o- Table 1: Average AUC (with standard error) for mini via the OpenAI API and gemma-3-4b-it run- 192 Users with Item-based Profiles ning on a local GPU workstation. When we provided the model with artist genres to contextualize Model Artists Names Ar&tisGteNnarmeses the names, neither model’s accuracy improved by a substantial amount. In the case of gemma-3-4b-it, gpt-4o-mini 0.75 (± 0.02) 0.77 (± 0.02) providing genres improved the accuracy in some gMeFm(mbaas-e3l-i4nbe-)it 00..6892 ((±± 00..0022)) 0.68 (±– 0.02) iterations of the experiment and degraded the accuracy in others.
Our first experiment with embedding user seeds into a natural language (NL) profile used a naive prompt as a starting point for these experiments. We asked the model to construct a natural language profile of a user’s listening habits given a list of artists that this user is known to listen to. The prompt for this can be found in B.3. We then passed this profile, along with a list of candidate artists, to the model in a new context window and asked it to rank the candidate set. Though we expected the quality of the recommendations to degrade by removing the set of seed artists, the experiments with these initial natural language listening profiles showed no substantial change in accuracy from the experiments that provided the seed artists directly in the prompt (see Table 2).
Research shows that LM-generated
summaries can be improved by providing the
model with contrasting examples [
model with an example output (one-shot prompting) can improve the quality of the response [
example, and improved slightly when provided with the optimal generated example. However, both of these changes were within two standard errors of the original score, and are not statistically significant.
than recommendation quality. We wanted to see if the quality would hold up if the listening profile was formatted as a list, and if it was written in first person. This is because lists are easy to edit and augment, and users will likely want to write in first person if they are editing and adding to their listening profile.
Our final experiments with prompt engineering combined all of these approaches. We also wrote the prompts while keeping in mind OpenAI’s guidelines3 for prompt engineering using their API. In this experiment, we found an accuracy of around 0.77, which is not a statistically significant improvement over our previous LLM-based approaches. We also tried this experiment with an accompanying system prompt, but did not see a significant diference in the AUC score. As such, although we will continue to use these prompt engineering approaches due to the slight improvement in accuracy, prompt engineering does not improve accuracy by a statistically significant amount in our case. The prompt for this approach can be seen in B.6.
tion, we tracked the performance for diferent users’ seed artist sets across our multiple experiments.
the development of more efective recommendation techniques. As such, we recorded the scores for individual sets of user seeds across six experiments; the original recommendation experiment with no
(a) Average Popularity vs. Average AUC for Natural
Language Recommendations (b) Average Popularity vs. AUC for ALS-MF Recommendations embedded listening profile, the prompt engineering experiment using examples, the prompt engineering experiment using contrasting items, and the three combined prompt engineering experiments (one of which used a human-written example, one of which used a solid generated example, and one of which used a generated example and a system prompt). The goal was not to see the accuracy of these experiments, but to analyze how individual users’ recommendations were performing under a variety of conditions.
We found that the average score for recommendations using a specific user’s seed artists was correlated positively (r = 0.337) with the mean Spotify popularity4 for those artists (i.e., the more popular artists a user listens to, the more accurate their recommendations will be). A graph illustrating this finding can be found in figure 2a. Each point in the graph represents the average artist popularity for a user’s item-based profile vs. the average accuracy of their recommendations over 6 LLM prompts (see Appendix B). For our baseline ALS-MF evaluation, we found that the correlation coeficient for average artist popularity and AUC was only r = -0.015. A graph illustrating this finding can be found in ifgure 2b. This suggests that our LLM-based recommendation approach is more susceptible to popularity bias than our MF baseline.
Given the potential for popularity bias re- Table 3: Accuracy with NL Profiles by Persona vealed by these findings, we perform an ex- Experiment Accuracy (gpt) periment specifically focused on determining the influence of artist popularity on LLM- Full Test Set 0.76 (± 0.02) 0.82 (± 0.02) based recommendations. LMoewdiPuompuPloapruitlyarity 00..7746 ((±± 00..0034)) 00..8831 ((±± 00..0022))
To start, we divide the set of artists in High Popularity 0.81 (± 0.03) 0.82 (± 0.02) our dataset into three roughly equal-sized groups based on their Spotify popularity; low popularity artists with a score below 64, medium popularity artists with a score between 64 and 73, and high popularity artists with a score above 73.
which is considered an individual user persona for this experiment. After filtering these personas by size, we were left with 110 low popularity personas, 82 medium popularity personas, and 109 high popularity personas. We then used these personas to run our best-performing approach, ”All (Generated
Accuracy (Baseline MF)
and language models. It is clear that, with these methods, recommendations made with language models do not perform as well (in terms of recommendation accuracy) as recommendations made with traditional methods. However, these listening profiles provide a potential for user control that would not be possible with conventional methods. As such, we believe that language models could be a useful tool for scrutable and steerable recommendation.
number of approaches to prompt engineering we could have tried, and there are many more language models (both commercial-grade and open-source) that we could have used for the experiments. These are both avenues that could lead to better accuracy and more readable listening profiles.
model training and fine-tuning. We are also interested in exploring hybrid recommendation which uses
traditional recommendation algorithms to rank order candidate artists and then uses language models
to help explain recommendations [
authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.
The code used for this paper can be found on GitHub at https://github.com/JimiLab/localify-nlp.
This section enumerates prompts used for LLM artist recommendation. B.1 is used for item-based profile recommendation. B.2 is the generic prompt for natural language (NL) profile recommendation. B.3 - B.6 are the prompts used to generate NL profiles using diferent prompt engineering techniques. B.7 is used in conjunction with the recommendation prompts to achieve a model response that can be parsed by our evaluation procedure.
You are an expert in music recommendation. Your specialty is in ranking a list of artists by how similar each one is to a diferent set of artists that someone already knows.
You are presented with a client who frequently listens to the following artists: You are asked to use your expert knowledge of these artists to rank the following artists (on which you are also an expert) in order from most recommended to least recommended: B.2. Prompt used for Recommendations with Listening Profiles You are an expert in music recommendation. Your specialty is in ranking a list of artists based on a textual listening profile that you are provided.
You are able to perfectly rank these artists in order of preference, where preference is defined by how much a user matching your provided listening profile will enjoy that artist.
You are presented with a client with the following listening profile:
You are an expert in describing people’s music listening habits. You are presented with a client who listens to the following artists:
Give a textual description of this person’s listening habits, without using artist names.
You are an expert in describing people’s music listening habits. You are presented with a client who listens to the following artists: Give a textual description of this person’s listening habits, without using artist names. Write this description according to the following example, but the details should correspond to the user I have provided. You are asked to use your expert knowledge of musical artists and your complete understanding of the listening profile to rank the following artists (on which you are also an expert) in order from most recommended to least recommended: B.4.1. Human-Written Example
My music taste is a mix of pop, musical theatre, r&b and dance/electronic. I used to be in theatre, so I love a great singer, and I really value the quality of voice and the skill of the band. As such, I love when artists use real instruments and studio musicians. I do also like music at the other side of the spectrum that’s electronic, but not synth trying to be instruments. I like upbeat music, so I don’t usually listen to sad songs but I will if the vocals are amazing, and I love a dreamy relaxing r&b track. I also love a rap feature on a pop song.
My musical taste weaves velvety jazz-infused neo-soul with driving, synth-heavy electronic grooves that efortlessly blend warmth and futurism. I gravitate toward smoky, upright-bass-led lounge numbers that evoke intimate club corners, alongside pulsating house tracks that ignite late-night dancefloors. I refresh my sets with experimental ambient textures and lo-fi hip-hop beats that layer nostalgic vinyl crackle over head-nodding rhythms, while occasional avant-garde free-jazz injections add daring dissonance. This balance of cozy soulfulness and boundarypushing sonic exploration speaks to my love of music that soothes and stimulates in equal measure. The result is a listening profile rooted in sophistication and spontaneity, comfort and curiosity, ofering a journey that feels both familiar and thrilling.
You are an expert in describing people’s music listening habits. You are presented with a client who listens to the following artists: {other_seeds} You are also presented with the following other users’ familiar artists: Give a textual description of this person’s listening habits, without using artist names. Focus on how your client’s listening habits difer from the habits of the other users (what makes them unique). Do not directly mention these other users, and do not pander to the client. Give an accurate description that gives the best possible summary of the artists that they like. The description should be designed such that a third party could use it to make artist recommendations.
B.6. Prompt used for Listening Profiles by All Techniques You are an expert in describing people’s music listening habits. You are experienced in writing clear, concise, and descriptive summaries of people’s listening habits based on a list of artists that they like.
You are presented with a client who listens to the following artists: You are also presented with the following other users’ familiar artists: Give a textual description of this person’s listening habits, without using artist names. Focus on how your client’s listening habits difer from the habits of the other users (what makes them unique). Do not directly mention these other users, and do not pander to the client. Give an accurate description that gives the best possible summary of the artists that they like. The description should be designed such that a third party could use it to make artist recommendations, and it should follow the following example: as a structure and guide, but the details should pertain to the client it is written for. The description should be written in first-person, from the perspective of the client.
You must present these recommendations in a very specific way. Each candidate artist that you are recommending has an integer ID associated with them. The key is as follows: You must finish your response by listing your recommendations only using these ids, separated by commas, and surrounded by <>. An example recommendation would be as follows: <#,#,#,#,#,#,#,#,#,#,...> With each hashtag replaced by the artist you recommend in that position. You must abide by the following rules: - Rank all of the artists in the candidate set I provided you, not just however many are in my example recommendation - Do not include any artists not in that candidate set - Your response must be formatted as I have said, in a list of comma separated ids (governed by the key I provided) and surrounded by angle brackets/chevrons (<>)