Recent advances in large language models have extended their potential use cases to diferent domains. Models such as ChatGPT have an extensive internal knowledge base that enables them to provide answers to various domain-specific queries. In this paper, we explore the potential use of OpenAI's GPT3.5 model as a conversational recommender system. We designed a user-friendly chatbot capable of recommending items in three domains: books, movies, and music. Our study involved collecting explicit feedback from 517 users, and we report the results obtained. The average usefulness of our bot is 4.15 / 5. Our experimental results demonstrate the efectiveness of GPT3.5 as a personalised recommendation system. We hope that our work will inspire further research in this area. Our chatbot is available on the popular messaging platform Telegram ∗Corresponding author.
recommender systems, large language models, user study Although this is an incredible improvement over non- such as GPT3.5 can provide good recommendations to tent on almost every topic. Since people are usually in- some heuristics or complicated architectures [5, 6], but
terested in a limited number of topics, recommender systems play an important role in generating personalised interfaces. They try to understand the user’s interests and intentions and select suitable items to be displayed at the top.
There are many types of recommender systems. Some
models work for registered users, using the user’s past
interactions to predict the user’s next actions. Other
methods, such as session-based recommenders, work even
when the user is not authenticated. Such systems
analyse the actions in the current session and try to predict
the actions in the next session. Currently,
state-of-theart next-item recommender systems such as BERT4Rec
correctly guess the next item 10%-30% of the time [
nEvelop-O report the results. Our experiments demonstrate 1https://openai.com/blog/chatgpt
the efectiveness of GPT3.5 as a personalised rec- phone) and service (travel). The authors found some ommendation system. limitations, such as the use of outdated data, the gen• We integrated our GPT3RecBot chatbot into Tele- eration of false and misleading text, and ignorance of garm, one of the most popular messengers in the some facts. The researchers concluded that this model world. has great potential for supporting recommendation tasks.
In contrast to this article, we conduct a user study that allows the calculation of qualitative metrics. In [27] the authors use ChatGPT for personalised recommendations via chat. They provide a case study for cross-domain and cold-start recommendations. In addition, this paper includes ofline experiments on MovieLens-100k where the proposed ChatRec outperforms a state-of-the-art conventional recommender LightGCN. Another paper [28] is less related to our research, but still suggests using Chat
papers that present the results of a user study on the quality of ChatGPT’s personalised recommendations. We hope that our findings will inspire new research directions in this area.
Conversational Recommenders. The authors of the survey [11] presented a typical conversational recommender system (CRS) architecture, which includes five components dedicated to diferent purposes. Due to their complexity, CRSs face many challenges [12], such as question-based user preference elicitation and dialogue understanding and generation. The authors of [6] distinguish between diferent types of utterances in CRSs. One of the simplest approaches is ”System is Active, User is
users respond. We choose this approach as a first step in using ChatGPT for recommender systems, and leave the other types of utterances as future work.
Approaches that follow the SAUR paradigm consist of
several complicated components. Paper [
ChatGPT. Thanks to the rapid development of large- The chat flow applied to all users in GPT3RecBot is shown scale language models [19], the quality of dialogue un- in Figure 1. We have implemented a SUAP paradigm derstanding has improved. ChatGPT is a good example. where only GPT3RecBot asks questions and the user It has attracted the attention of researchers from physics answers. During the interaction process between the [20], mathematics [21], computational biology [22], etc. user and our application, we have several steps. The first Researchers have adopted ChatGPT for various fields of step is initialisation, where the bot randomly assigns the machine learning such as natural language processing user to one of three test groups. These groups allow us to [23], stock market prediction [24], binary classification test three diferent prompts for GPT3.5 in the user study. [25] etc. ChatGPT was published on 30 Nov 2022, and the After initialisation, we are ready to communicate with number of papers on Google Scholar related to ChatGPT the user and ask them to select the type of content. We do is about 10000 as of 9 May 2023. Therefore, it is dificult not ask for specific types of content and the users choose to describe all the cases, but it is definitely a topic that is for themselves. The next step is to ask explicitly about the intensively studied by the research community. user’s interests. This helps ChatGPT to understand the
The possible application of ChatGPT to recommender user’s preferences. The next question is asked to users systems has not yet been well studied, due to the short in the second and third test groups. This is the question time since publication. A very recent paper [26] provided a case study of ChatGPT recommendations in three typ- 2BPMN - Business Process Model and Notation diagram ical domains: entertainment (music), high cost (smart- 3https://telegram.org/
about the content the user does not like. We expect the model to use this information to further restrict the types of content genres that are recommended.
After the questions have been asked, the prompt generation phase begins, whereby ChatGPT is supplied with generated prompts. During this phase, a unique prompt is generated for the bot, making it specific for each test group , by filling in pre-existing templates with user responses. We keep three diferent prompt templates, and each subsequent group’s prompt extends the previous one. The example is shown in Figure 2. After generating the prompt, we send it to gpt-3.5-turbo (training data until September 2021) using the OpenAI API, which provides us with a convenient interface to use the power of
answer is sent back to the user.
We then ask the user to rate the reality of the recommendation, which measures how realistic the recommended items and descriptions seemed to the user. The next question is about relevance to the user, which is a measure of how well the model matches the user’s preferences. The last is usefulness, where we measure whether GPT3RecBot can help users in their daily lives.
attendees. If we reach this limit before the conference, we will provide an opportunity to try it via another OpenAI account belonging to another author of this paper. In case of force majeure, anyone can run our GPT3RecBot as soon as we release an implementation code under the MIT licence5.
interests without providing any personal information. form. As this was done through a crowdsourcing platform, participants knew that their ratings would be used for 3.2. Implementation anonymised aggregation. We informed them that this information would only be used for research purposes. The The architecture described in this section is typical of Research Ethics Committee (REC) of the HSE university messaging bots. It’s shown in Figure 3. We have a simple approved this study. client-server application, where the client is Telegram Description. We conducted a user study to evaluate and the server is our own application. Let us describe the potential of GPT3RecBot. As GPT3RecBot is a chatthe communication flow between the main components bot that recommends items on request, real users were we have in the whole system. involved in the experiments. We used a crowdsourcing
A user starts the bot and sends a start command using platform to get high quality feedback on the bot. All the Telegram client when he decides to try our applica- crowdworkers were asked to read the questions carefully tion and participate in the experiment. The Telegram and share their thoughts on the quality of the recomclient sends calls to our server application via the Tele- mendations and the formatting of the recommendations. gram API. This process continues until there are no more We asked students from an HSE university to test our requests for messages. The main application, the core chatbot. It is important to note that respondents chose of the server, stores all conversations and talks to Ope- the type of content they wanted to be recommended. For nAI to get recommendations via ChatGPT based on the example, some people don’t like to read books, but they user’s specific requests when it’s time. All responses, are music fans, so they are likely to be interested in music messages and technical information (logs) are stored in recommendations. We believe that a set of music, book the database, which is based on SQLite [29], a relational and movie recommendations can cover the interests of database popular for small projects. The reason for choos- the majority of people. However, in future research, we ing this database is that it can handle the full load of the aim to expand the applicability of our methodology to system. It is convenient for research purposes to down- encompass a wider range of domains. load the database from the server and monitor the exper- To gain some insight into prompt engineering for GPTiment online, as this database stores all the information based recommendation models, we randomly divided the in a single file. At the end of the full run, we ask the user users into three groups, with each user being assigned if they want to try again. This creates a loop and allows to a group for the duration of the study. The first group the user to do a few iterations. was only asked about their preferences in one selected
Implementing a bot in Telegram for messaging with domain. In the second group, the users had to indicate users is free. The messenger has reached 550 million the items they did not like. This information could theomonthly active users in 2022 [30] and it is quite easy retically help the model to filter recommendations and to register new users. Use of the OpenAI API is paid. avoid suggesting obviously irrelevant content. AccordHowever, OpenAI grants $18 for new accounts. The price ing to recent research [26], ChatGPT are said to be good is $0.002 / 1K tokens actual on 9 May 2023. The average explainers. In preliminary experiments, we found that length of our requests is about 100 tokens, according to GPT3.5 could explain its recommendations to users, even the user study. This means that we can get about 5000 linking items to their interests. So we added a third group responses from GPT3.5 for free and continue to get them in which GPT3.5 was instructed to explain its recommenfor $0.0002 / response. dations. We assumed that explaining recommendations
We will maintain this bot until it reaches the limit could improve the user experience, based on the results for free responses. At the time of submission, we have of other studies [31, 32]. about 4500 responses available to the OpenAI API, which seems suficient to demonstrate the bot to conference
book texts. The worst performance is for music recom517 people completed our task and left feedback. They mendations. When analysing free-form questions, we gave explicit feedback on three diferent metrics, such were able to catch fake music titles generated by Chatas the reality of the suggestions, the relevance of the GPT. It may try to synthesise additional information if it recommended items and the usefulness of GPT3RecBot. is not available in suficient quantity in the training data. Note that we do not count users who did not give explicit If we analyse the results within groups, we can see feedback. The summarised results are listed in Table 1. that GPT3.5 significantly improves its recommendations There are a few things to note. when it is aware of negative user experiences, which
Firstly, people chose films more often than books and may help to filter recommendations. Asking GPT3.5 to music combined. We shufled the domain selection but- explain its recommendations improved the results for mutons during the experiments, so this bias is not introduced sic and film recommendations, which may have helped by the GPT3RecBot interface. It is presumed that among to convince users of the appropriateness of the recomthe users who participated in our system’s testing, films mendations generated. watching is a more prevalent hobby compared to books To test the hypothesis that GPT only recommends a or music. The other suggestion is that people are most limited number of content titles, we measured content dilikely to want to try out movie recommendations, as they versity, which is the proportion of unique recommended have experienced with movie recommendation engines content to the total number of content titles [34]. We in the past. We leave the investigation of this efect to can clearly see in Table 1 that ChatGPT is able to provide future work. users with content that is diferent and new to them. In
Secondly, it is clear that there is a diference in user rat- Figure 4 we show 5 of the most popular recommended ings depending on the type of content. GPT3.5 may have movies and books. The model recommended Nineteen a diferent knowledge capacity for diferent types of con- Eighty-Four6 16% of the time. Other items were recomtent. The best results were obtained when recommending mended less than 4% of the time. The top recommended books, which received the highest ratings in all questions items appear to be popular and this may be due to popuasked. The reason for this may be that ChatGPT has larity bias [35]. more knowledge about text-based content and is able to For the open-ended question, we had to come up with process this type of content. This is followed by films, where it also performed well. The poorer performance 6Nineteen Eighty-Four is a dystopian social science fiction novel and may also be explained by the information available for cautionary tale by George Orwell.
provides a good selection of books for the future. It would be great if it also described the genre. The first three recommendations were to the author’s liking, but they chose the wrong genre. The bot makes good recommendations, but most of the choices are those that have already been read. It is useful as a handy assistant. The recommendation of a collection of poems is based on preference, but not on the theme of war
mended films, but it would be nice to be able to select films released by year and add additional criteria to get a more satisfactory recommendation.
from IMDB to the bot. The AI should be more selective and understand general preferences to make recommendations for films, not just popular world films, but also lesser known good films.
make a recommendation based on the person’s interests The idea of the bot is interesting, but the description of the recommended compositions is imprecise. The format and quality are good, but the recommendations need to be refined with interest filters. This service is useful for discovering new music and new artists, but it does not match the user’s interests.
The narrator is looking for more thrillers and detective stories. They have a good selection of recommendations, 4/5 of which are known and read. They like the format of the recommendations, which is simple and clear. They hope that the next recommendation will be more interesting.
clear way of providing information, the clarity and format to meet the needs were all appreciated. The service was good and the bot made good recommendations for each genre, making it convenient and easy to register.
The most important details are that a bot can be used to select music to listen to, and that the recommendations are of high quality and match preferred genres. The service is convenient and eficient, and the recommendations are of high quality and in preferred genres. a special method to process such a large amount of feedback. First, we split all the feedback into domains and then used the Distilbert model7 for sentiment analysis [36] and selected only those reviews where the model was at least 70% sure about the sentiment of the feedback. We then combined the feedback into a single text and used the Quillbot8 to summarise it. The results are shown in Table 2. The responses in the negative and positive reviews seem a little contradictory. Users who left negative feedback point out that GPT3RecBot does not correctly guess the user’s intersets. However, users who left positive feedback wrote the opposite opinion.
tive feedback and 18% negative feedback according to
gingface.co, revision af0f99b
Nineteen Eighty-Four
The Little Prince To Kill a Mockingbird
Three Comrades One Hundred Years of Solitude 2 3 3 4 Interstellar 3.6 A Beautiful Mind
La La Land
Shutter Island Game of Thrones
In this paper, we present GPT3RecBot, which can personally recommend movies, books or music. This bot is available to Telegram users for research purposes, and each user has 10 free requests. We share a chat flow, implementation code, and a prompt used to facilitate future research. Our paper is the first to present the results of a user study on the recommendation quality of ChatGPT. 517 users participated in the study and gave an average rating of 4.01 / 5 for reality of recommended content, 3.86 / 5 for relevance and 4.15 / 5 for usefulness of GPT3RecBot. We hope that such high ratings will increase the interest of the research community in future work. timent analysis: A survey, Wiley Interdisciplinary
(2018) e1253.