Recommender systems play a vital role in many application settings, ranging from online shops over streaming services to websites that want to suggest other relevant content based on their users' browsing history. Although sequential recommendation has been extensively studied and several model architectures [1,2,3] and several ways to incorporate additional information about users or items [4,5,6] have been proposed, they are not directly applicable in settings where the context is not given in the form of click sequences. Recently, however, applications based on chat or dialogue data have become increasingly relevant. Here, context is provided in the form of text messages rather than previous clicks. The use cases for this kind of recommender are manifold, ranging from chatbots that can observe chat rooms and enrich them with additional information about the topics being discussed to conversational recommender systems, where the bots actively engage in the conversation and try to find the users' interest by asking questions. Although the setting is, at first glance, very different from traditional sequential recommenders, we show that we can adapt arbitrary sequential recommender models to this task using a modeling approach for non-item pages proposed in Fischer et al. [7,8]. The main idea of this approach is to incorporate additional information into sequential recommender models by including non-item pages in the sequences. These non-item pages can be, for example, list pages, category pages, or blog posts describing articles of a certain category. They are inserted into the sequence and represented by an embedding vector like regular items, but are never the target of the recommender. In the same way, we can build a sequence consisting of chat messages and items (representing relevant content in the sequence), treating the chat messages like non-item pages. In this paper, we evaluate the suitability of sequential recommender models in their basic version and when using non-item page modeling on four datasets of conversations about movies, website content Figure 1: A schematic visualisation of our task and approach. Our goal is to identify items that are mentioned in a conversation, possibly by very noisy names (e.g. "RADI" referring to the item Radiance and "RAX" referring to Buildings#Barracks). To this end, we represent each message in a conversation as a "non-item" and train a model to recommend pages with information about the items referred to in the messages.
or sports events. In all of these datasets, our goal is to detect which items (e.g., players, movies) are currently discussed in the conversation and to recommend relevant links to the users. Figure 1 gives a visualization of our task and approach.
Our contributions in this paper are as follows: (a) We adapt sequential recommender models to our task of providing additional content relevant to the conversation. (b) We introduce four datasets for this task, partially derived from existing datasets, enabling further research. (c) We compare our approach with baseline recommender models, showing that it is effective in finding relevant content for conversations.
As neural networks have gained widespread popularity, they have become a popular choice for modeling and learning user behavior sequences. Sequential recommender models have been developed based on recurrent neural networks [1] or convolutional neural networks [2,9,10]. Numerous architectures have since adapted the attention mechanism [11] for sequential recommendation [3,12,13,14,15].
All of these works utilized only the item sequences itself, but the inclusion of additional item or user information has also been investigated by several studies. Item features have been modeled in RNNs [4,16], CNNs [5], and transformer models [6,17,18,19,20]. Some approaches merge additional item information directly in the attention layer [21,22]. There are also works that study the inclusion of user representations in a similar way for different architectures [2,23,24,25,26]. Information that is not directly related to a user or an item, yet still part of the sequence, has not been investigated as extensively. Some works for the Coveo challenge [27] are an exception, as they include interactions that are not tied to specific items. These non-item interactions are included in sequential recommender models by [28,29,30], but they are represented the same way as item interactions. An explicit modeling and a formal definition of non-item pages was first introduced for transformer-based models by Fischer et al. [7]. A more extensive study [8] expands the setup to a wider set of sequential recommender models and allows the integration of non-items with generic embedding representations (see Section 3.1).
Providing relevant content items based on textual information can also be viewed as a form of Entity Linking (EL). Usually, the task involves the recognition of entities explicitly mentioned in texts and their disambiguation [31]. Since many conversations feature nondirect references to entities, where context clues must be leveraged to identify the target, many EL approaches face difficulties recognizing and resolving such references. This special setting -Implicit Entity Linking -has been investigated primarily in the setting of Twitter posts [32,33,34]. These approaches, however, rely heavily on manual feature selection rather than current deep learning models. Additionally, due to the setting, they often only consider limited conversational context directly. In this work, we investigate how sequential recommendation models can be adapted to this task. Applying these models to implicit entity linking is beneficial, since it enables us to directly transfer any progress in sequential recommendation to implicit entity linking.
Our setting is also related to conversational recommendation [35]. However, while the goal there is to lead a conversation with a user, ask questions designed to narrow down the space of possible items, and finally provide a recommendation, our goal is to detect the items that are mentioned in a conversation to recommend pages providing additional content relevant to the conversation.
Our model can be used as one component of a conversational recommender system, for example to identify the movies/items that the user mentions and provide them to the recommender model in charge of determining new suggestions.
Non-Item Pages have been proposed recently as a way to incorporate sequence elements other than the items that are potential targets for recommendation into sequential recommender systems [7,8]. In essence, they can be of arbitrary nature, representing additional content related to the items, for example, a page listing items of a specific category, a search query, or a blog post about a set of items. [8] propose different ways of representing these non-item pages, depending on their nature. The two representations relevant for our setting are: (1) Unique Page-ID (UPID): A unique identifier can always be assigned to any non-item page and integrated into the sequence as an item. Possible drawbacks are high sparsity and a growing vocabulary. (2) Page Embedding (PE): A single placeholder id is used to represent all non-items. The information about the non-item is included by adding an embedding representation of the non-item to the id in the embedding layer.
In this work, we use this non-item page modeling to include chat messages as information about the current topic in a conversation.
Our task is to enrich conversations in chat systems with content that is relevant to the current conversation. We model this task as a sequential recommendation problem, treating chat messages as non-item pages and relevant content as items, for example, in the form of links to Wiki pages.
More formally, we closely follow the setup of [8] and introduce our notation as follows: Our work aims to solve the task of recommending items (i.e., relevant content) in a conversation (modeled as a session) given a sequence of chat messages and previously mentioned items. We define the item set as 𝒱 = {𝑣 1 , 𝑣 2 , … , 𝑣 |𝒱 | } and the set of non-items as ℳ = {𝑚 1 , 𝑚 2 , … , 𝑚 |ℳ| }. The set of conversations is defined as 𝒞 = {𝑐 1 , 𝑐 2 , … , 𝑐 |𝒞 | }. For each conversation 𝑐, we can denote the sequence of interactions as 𝑠 𝑐 = [𝑖 1 , 𝑖 2 , … , 𝑖 𝑛 ] ⊆ {𝒱 ∪ ℳ} 𝑛 . Using these notations, our goal is to solve the task of predicting the next item 𝑖 𝑛+1 ∈ 𝒱 for every interaction 𝑖 𝑛 in each sequence 𝑠 𝑐 ∈ 𝒮.
We model our task as a sequential recommendation problem with sequences consisting of items and possibly chat content. This enables us, in principle, to use any sequential recommendation architecture. In this section, we describe our model architectures, our different ways of representing conversations as sequences and our subsequence sampling.
Here, we discuss our representation of conversations with mentioned items as sequences for a recommender model. Assume this short conversation as an example, where items are marked in bold:
User 1 I like The Godfather User 2 Do you like The Matrix?
We compare different variants of modeling this as a sequence of items and possibly non-items:
Items Only As a baseline, we apply standard sequential recommendation models directly to the sequence of items mentioned in the conversations, ignoring the chat messages. In this model, the conversation above becomes 𝑠 items = (The Godfather, The Matrix). This can work on some datasets that are close to the traditional sequential recommender setting (e.g., in the ReDial-Mention dataset, cf. Section 5.2). However, we expect that this setting does not carry enough information in other datasets and that the content of the conversation is required to adequately identify the items that are referred to in the conversation.
Unique Token ID (UTID) Therefore, we propose to switch to a non-item modeling setting to include chat content. As a first step towards this, we split the messages into tokens corresponding to words and introduce a new "word-item" 𝑚 𝑖 to the set of non-items ℳ for each unique token occurring in the dataset. We then map each occurrence of a token to the corresponding word-item and include these word-items in the sequence. Now, our sequence becomes 𝑠 UTID = (I, like, The, Godfather, The Godfather, Do, you, like, The, Matrix, ?, The Matrix). This means that the model only has to identify the target items that have been referred to in the conversation rather than predict which item could be brought up next. Note that the modelling here is still very close to a traditional sequential recommender, with the only difference being that our models are only trained to predict actual items from 𝒱, that is, word-items from ℳ are never the target for recommendation, as we always build our sequences to end with actual items. This setting is identical to the Unique Page-ID (UPID) setting in [8].
Token Embedding (TE) Since the previous setting introduces a very high number of items to the vocabulary (each word in the texts is mapped to a separate item), making it more difficult to train the models, we explore the Page Embedding setting from [8] next: Here, we map all words to a single placeholder token T, leading to a set of non-items with only one entry ℳ = {T}. Each occurrence of this token is then associated with a pre-computed embedding r 𝑚 for the word it corresponds to. These embeddings can, for example, be extracted from language models or simple word embeddings. With this approach, our example conversation is represented by the id sequence 𝑠 TE = (T, T, T, T, The Godfather, T, T, T, T, T, T, The Matrix). The actual content of the tokens will only be represented by the corresponding embeddings.
Message Embedding (ME) While using Token Embeddings solves the problem of producing a very large vocabulary, it still leads to a high sequence length, since each word is included in the conversation as a separate element. However, the non-item page modeling also allows us to see entire messages as non-items. Therefore, in our final setting, we map each message to a placeholder token M, again leading to a singleton set of non-items ℳ = {M}. As in the previous setting, this placeholder token is shared for all messages and the content of the message is represented by a precomputed embedding r 𝑚 , for example from a language model. In this setting, our sequence becomes 𝑠 ME = (M, The Godfather, M, The Matrix).
We utilize a set of popular recommender models which all leverage the sequence of item ids. While UTID representations can be used without model changes, we need to adjust the embedding layer to include additional embedding representations for non-items. The common setup for the embedding layer, with embedding size 𝑑, looks as follows:
(i) an id embedding 𝐸 𝒱 ∈ ℝ |𝒱 |×𝑑 (ii) an optional embedding 𝐸 𝐺 ∈ ℝ 𝑁 ×𝑑 to encode the position of the items in the sequence, with 𝑁 as the maximum input sequence length.
To include any additional precomputed embedding representation r𝑖 ∈ ℝ |𝑅| for an interaction 𝑖, we add (iii) a linear layer 𝑙 𝑅 (𝑥) = 𝑊 𝑥 + 𝑏 with the weight matrix 𝑊 ∈ ℝ 𝑅×𝑑 and bias 𝑏 ∈ ℝ 𝑑 which scales the representation to the embedding size 𝑑.
The final output of the embedding layer is created by summing up all layers. For each sequence step 𝑛 we embed the id of interaction 𝑖 𝑛 with 𝑒 𝑛 = 𝐸 𝒱 (𝑖 𝑛 ), the position with 𝑔 𝑛 = 𝐸 𝐺 (𝑛) and the scaled precomputed representation r𝑖 with 𝑟 𝑛 = 𝑙 𝑅 ( r 𝑖 𝑛 ). We compute ℎ 0 𝑛 = 𝑒 𝑛 + 𝑔 𝑛 + 𝑟 𝑛 as the input for the following layers of the sequential recommender. This allows us to add additional latent representations for any interaction, for non-items as well as items. Specifically, it also enables the use of the Token Embedding and Message Embedding strategies (cf. Section 4.1), which use only one shared non-item id and represent the content of the non-items using pre-computed embeddings.
In our setting, not only predicting the last item in a sequence is of relevance, but rather predicting all items given the chat context up to this point. Therefore, we generate a set 𝑆 𝑖 of subsequences from each sequence 𝑠 𝑖 in our datasets in the following way: We select all items 𝑣 𝑗 ∈ 𝒱 in 𝑠 𝑖 and, for each of these items, generate a subsequence 𝑆 𝑖,𝑗 = 𝑠 𝑖 [1 ∶ 𝑗] containing the sequence up to item 𝑣 𝑗 . Should a subsequence exceed the maximum sequence length set for the model, we truncate it from the left side. We evaluate our models on each of these subsequences to get an accurate estimate of their performance.
Since some of our datasets are small, we optionally also employ the same subsequence generation scheme for the training data, training on each subsequence and therefore increasing the amount of training samples.
We use four datasets in our experiments:
1. ReDial-Mention: conversations between two users, where the task is to recommend links to the movies mentioned in the conversation, 2. HandballSynth: synthetic conversations in streams of Handball games from the European Championship, where the goal is to link the profiles of mentioned players and teams,
Bestatter: conversations between users and a chatbot, with the task of recommending pages with additional information about the discussed topic and 4. Twitch: chat sessions in e-sports streams, aiming to recommend wiki pages with information about the players, teams, in-game characters and other entities mentioned by the users.
Descriptive statistics for all annotated datasets are provided in Table 1.1
Generally, we construct samples for our task from conversations in the following way: Each conversation is converted into a sequence 𝑠 consisting of items 𝑣 ∈ 𝒱 and potentially text messages 𝑚 ∈ ℳ. For Items Only, we build a sequence from every item 𝑣 mentioned in the conversation, discarding text messages. In all other variants, we build a sequence by representing each message 𝑚 either as the sequence of its words or as a single token M as described in Section 4. If one or multiple items 𝑣 are relevant to a message 𝑚, we append the items directly after 𝑚 as targets for our recommender in the order in which they were discussed in 𝑚.
We adapt the ReDial dataset for conversational recommendation [36] to our task. ReDial consists of conversations between two users, one "seeker", who is looking for movie recommendations, and one "recommender", who tries to find fitting movies to suggest to the seeker. This dataset is suitable for our task because it contains conversations annotated with movies that are mentioned in the messages. We construct samples for our task from ReDial as described above. ReDial contains movie IDs in the messages, which we replace by the full movie title to which they refer. Additionally, we crawled information such as the plot summary and director from the OMDb API2 , from which we build our item embeddings.
ReDial-Mention-Noise Since the first version of the ReDial dataset always mentions the exact movie title in the messages, potentially making the task too easy, we construct an additional noisy version of the dataset. To this end, we employ a pre-trained Large Language Model, specifically Llama 3.1 8b3 to get modified versions of the movie titles. We query the model for 20 alternative titles for each movie with this prompt template: Return a name for this movie as it would likely be used in a conversation. For example, if the movie is called "The Matrix", you might return "Matrix". For "The Lord of the Rings: The Fellowship of the Ring", you might return "The first Lord of the Rings movie". Return only one possible name, nothing else: { Movie Title } We keep the list of all 20 alternative titles, including duplicates. Since this sometimes yields titles that are too noisy or even wrong, we apply an additional filtering step, where we query the same model with this prompt template:
You will be passed a pair of original movie title and a noisy version of it. Your task is to determine if the noisy version is a valid reference that could be used in a conversation. This can be either the original title, a slightly modified version or a description.
If you think the noisy version is a valid alternative, return "yes". If you think the noisy version is not a valid alternative, return "no". Do not return anything else. The original title is: { Movie Title }. The noisy title is: { Noisy Title }. We reconstruct the dataset by sampling an alternative title from this list for each mention of a movie when building the message representation.
We generate a synthetic dataset for a controlled scenario with accurate information on all intended references. It simulates a chat with live messages from ongoing handball matches, where entities (e.g., players and coaches) are discussed and we want to recommend pages with additional information about the entities that are mentioned by the users. We used the pre-trained Large Language Model Claude 3.5 Sonnet4 to generate conversations based on play-by-play information from live-ticker data 5 .
The LLM was prompted to create artificial chat messages as reactions to the events in the ticker. In order to keep the generated messages realistic, we instructed the model to generate emotional, short and informal messages, as well as occasional off-topic comments. We included the relevant team and player data 6 in the prompt and instructed the model to discuss specific entities in roughly every third message. This information is also used to build the embeddings for the items. Additionally, the model includes the link to the website containing the entity's information -obtained from the sources included in the prompt -with every generated reference. We remove these links from the messages and use them as target items as before.
We use a dataset of user conversations from a service chat bot on a German website for information about funerals 7 . The bot provides users with information for funeral related inquiries. In addition to general information on the topic, the bot also recommends links to information on the website, external web sources, as well as relevant contact information.
From these conversations, we construct a dataset for our task as before. As target items, we extract links mentioned by the bot, as well as contact information such as email addresses and phone numbers and replace their occurrences in the dialogues with a corresponding placeholder string.
We generate item representations by embedding the text contents of the linked websites for links. For contact information we construct the representation for the item by embedding the local context of the reply message containing the email address or phone number, including up to 250 character before and after the mentioned contact information.
We collected a dataset of messages from Twitch.tv 8 , a popular streaming website, primarily used for e-sports streams. Along with each stream, Twitch provides a chat where users can comment on the content of the stream. The purpose of this data set is to recommend links that provide background information on entities disscussed in the chat. Messages on Twitch.tv generally have an extremely noisy language with a high amount of spelling errors and slang [37], making this dataset especially challenging.
We downloaded chat messages from several streams of the popular e-sports game Dota 2 using TwitchDownloader 9 and manually annotated them with relevant links from Liquipedia 10 and Wikipedia. Liquipedia provides information about both in-game characters and mechanics as well as real-world entities (players, teams) and tournaments in the context of Dota 2.
In total, we selected seven broadcasts from between September 2023 and June 2024, featuring both official streams from tournaments of different sizes as well as personal streams by individual, well known streamers. From these chat logs we randomly sampled segments of 100 consecutive messages for annotation. The messages where annotated independently by two annotators, with 1900 messages judged by both and an additional 3000 messages annotated by either one alone. Using the messages annotated by both participants, we observe a moderate inter annotator agreement of Cohen's kappa 𝜅 = 0.47. To further ensure quality of the annotations, an additional curator merged the two sets of annotations, resolving any discrepancies in the process.
Annotators were instructed to go through the chat log segments one message at a time, providing links to any entity directly mentioned or implied within that message. These links will serve as targets for our recommendation task.
In case of a direct mention of an entity, we can map them directly to their wiki page. However, many elements of the game do not have their own article. In this case, we link to the parent wiki page: For example, if a named ability is mentioned, we link the wiki page of the character with this ability. Apart from direct mentions, memes and in-jokes are common within the community. These often serve as a reference to a particular player or incident, from which we again derive appropriate link targets.
Again, we use the content of the linked pages to compute embeddings for the target items. As with the other datasets introduced above, sequences are constructed from the annotated messages.
Sequences are constructed from the annotated messages as above.
Evaluation Metrics For all sequences, we ensure that the last interaction in a sequence is an actual item 𝑣 ∈ 𝒱 for training and testing. We calculate the Hit Rate@k (HR@k) for 𝑘 ∈ {1, 5} and the Normalized Discounted Cumulative Gain@5 (NDCG@5), as we are only interested in recommending a small number of items at once. We calculate our metrics only on the top k recommended items, ignoring non-items. Metrics are computed for all subsequences, as described in Section 4.3.
For each dataset, we evaluate our setup on the following popular sequential recommender architectures:
• GRU4Rec [38]: a simple recurrent neural network with Gated Recurrent Units.
• CASER [2]: a convolutional network with horizontal and vertical convolution filters.
• NARM [3]: a neural network combining recurrent neural networks with attention.
• NextITNet [10]: a model built upon CASER, using dilated convolutions.
• SASRec [12]: a popular baseline utilizing self-attention for sequential recommendation. Here, it is used with a cross entropy loss.
• BERT4Rec [13]: an adaption of BERT [39] with bidirectional transformer layers and masked training [40].
• CORE [15]: an attention-based model, which couples the representation space for both encoding and decoding by representing the session as a linear combination of the items.
• LightSANs [14]: an attention-based model with a low-rank decomposed self-attention and a decoupled positional encoding.
Model Hyper-Parameters For the configuration of our experiments, we follow the setup in [8]. All of our models are trained with the Adam optimizer, and all models have a hidden and/or embedding size of 64 and an inner size of 256. The number of heads and layers is set to 2 for transformer layers and the dropout to 0.2. GRU4Rec also has 2 layers. The mask ratio for BERT4Rec is set to 0.2, while the fine-tuning ratio is set to 0.1. For CASER we set dropout = 0.4, Markov chain length = 5, vertical filters = 4 and horizontal filters = 8. For NextItNet, we set the kernel size = 3, the convolutional filter width = 5 and the dilations = [1,4]. We use [212, 10,42,404,6] as fixed seeds and report the average and standard deviation over 5 runs. We set the maximum sequence length to the ≈90% -quantile; see Table 1. We train all models for 50 epochs with a batch size of 64 on a cluster of L40-GPUs. We utilize the code from [8] and provide our own code and configurations in our repository 11 .
Embedding Models We compute non-item embeddings from the text of the messages and item embeddings from the content of the linked websites to provide the models with additional information about the targets. All content is embedded using Sentence Transformers [41,42] models. For the English datasets (ReDial-Mention, ReDial-Mention-Noise, Twitch), we use a monolingual English model. 12For the German datasets (HandballSynth, Bestatter), we use a multilingual model. 13 Both models provide embeddings of size 384.
The main results for all datasets are provided in Tables 2 to 5. We selected the two best-performing models, SASRec and LightSANs, and refer to the appendix for results of the other models. All results in this section are for models trained and evaluated on subsequences (cf. Section 4.3). We include results for models that are not trained, but only tested on subsequences, in the appendix. These models consistently perform worse, which is expected due to the lower amount of training data available in this setting. We compare our different sequence representation strategies (Items Only, UTID, TE, ME; cf. Section 4.1), as well as training and evaluating the models with or without item embeddings (non-items are always represented with embeddings). We can summarize our findings as follows:
Overall Performance Overall, our models yield reasonable to very good performance on all datasets. Message Embeddings (ME) perform best consistently. This is expected since the message embeddings provided by the Sentence Transformers models should capture the content of the messages well, while also keeping the sequence length manageable. This makes it relatively easy for the model to determine which entities are referred to in the messages. The models trained on word-level sequences perform worse, but still manage to provide good recommendations in most cases. Interestingly, using item embeddings only has a small influence on the results. This may be caused by the rather long content of the linked items: Sentence Transformer models are primarily trained to represent sentences and short paragraphs well, while we use them to represent entire websites. We leave this investigation, as well as the development of better representations for the item content, for future work.
LightSANs no HR@1 0,046± 0,002 0,299± 0,003 0,310± 0,003 0,619± 0,007 HR@5 0,144± 0,004 0,468± 0,010 0,487± 0,003 0,755± 0,013 NDCG@5 0,097± 0,002 0,390± 0,007 0,406± 0,002 0,698± 0,010 yes HR@1 0,041± 0,003 0,308± 0,004 0,315± 0,009 0,624± 0,005 HR@5 0,132± 0,004 0,485± 0,006 0,490± 0,016 0,771± 0,004 NDCG@5 0,088± 0,003 0,404± 0,004 0,411± 0,013 0,708± 0,002 SASRec no HR@1 0,046± 0,001 0,541± 0,055 0,533± 0,033 0,631± 0,006 HR@5 0,144± 0,002 0,733± 0,036 0,729± 0,022 0,792± 0,004 NDCG@5 0,097± 0,001 0,650± 0,045 0,644± 0,027 0,724± 0,005 yes HR@1 0,042± 0,003 0,521± 0,006 0,505± 0,017 0,619± 0,007 HR@5 0,132± 0,003 0,716± 0,004 0,706± 0,017 0,764± 0,005 NDCG@5 0,088± 0,002 0,630± 0,004 0,618± 0,017 0,702± 0,005 Noisy LightSANs no HR@1 0,046± 0,002 0,271± 0,007 0,276± 0,007 0,594± 0,008 HR@5 0,144± 0,004 0,442± 0,010 0,454± 0,006 0,743± 0,004 NDCG@5 0,097± 0,002 0,364± 0,008 0,372± 0,006 0,680± 0,005 yes HR@1 0,041± 0,003 0,280± 0,009 0,278± 0,007 0,595± 0,009 HR@5 0,132± 0,004 0,457± 0,012 0,446± 0,013 0,750± 0,004 NDCG@5 0,088± 0,003 0,376± 0,011 0,369± 0,010 0,684± 0,005 SASRec no HR@1 0,046± 0,001 0,536± 0,050 0,515± 0,020 0,598± 0,007 HR@5 0,144± 0,002 0,720± 0,038 0,710± 0,016 0,765± 0,005 NDCG@5 0,097± 0,001 0,640± 0,045 0,625± 0,017 0,694± 0,003 yes HR@1 0,042± 0,003 0,489± 0,035 0,498± 0,039 0,590± 0,005 HR@5 0,132± 0,003 0,686± 0,023 0,697± 0,026 0,750± 0,005 NDCG@5 0,088± 0,002 0,600± 0,028 0,610± 0,031 0,681± 0,005 ReDial-Mention The Message Embedding-models are able to identify the movies mentioned in the conversations very well. Word-level models (UTID, TE) still perform somewhat well but worse than the ME models. Item Only models are not able to solve the task well, since they are missing the content of the conversations.
ReDial-Mention-Noise Results on the noisy version of the dataset follow exactly the same trends as on ReDial-Mention. Since the titles of the movies are no longer contained exactly in the messages, the task becomes harder and, consequently, the performance drops slightly. However, the models are still able to identify the mentioned movies well if given access to the content of the conversation.
HandballSynth The general trends also hold on this dataset. Here, the Items Only strategy yields acceptable results for HR@5, which is likely caused by the somewhat low number of entities in the dataset. However, for both HR@1 and HR@5, including the conversation's content into the models still leads to a clear improvement.
Bestatter We were unable to train word-level models in the Bestatter dataset, as word-level modeling leads to very long sequences on this dataset (cf. Table 1). Again, the Items Only strategy works relatively well here. In addition to the low number of entities in the dataset, as in HandballSynth, the conversations in this dataset are also somewhat schematic: Many of the conversations follow the Table 3 Average HR@k with 𝑘 ∈ {1, 5} and NDCG@5 with standard deviation on the HandballSynth dataset over five random seeds for LightSANs and SASRec. ItemEmb indicates the use of Item Embeddings.
ItemEmb Metrics Items Only UTID TE ME LightSANs no HR@1 0,164± 0,043 0,369± 0,011 0,346± 0,018 0,618± 0,006 HR@5 0,309± 0,030 0,479± 0,019 0,492± 0,007 0,638± 0,006 NDCG@5 0,242± 0,031 0,425± 0,013 0,422± 0,013 0,629± 0,004 yes HR@1 0,148± 0,025 0,369± 0,011 0,346± 0,009 0,618± 0,006 HR@5 0,449± 0,017 0,487± 0,033 0,500± 0,000 0,638± 0,006 NDCG@5 0,308± 0,011 0,429± 0,017 0,424± 0,003 0,629± 0,005 SASRec no HR@1 0,078± 0,018 0,351± 0,039 0,339± 0,025 0,618± 0,006 HR@5 0,273± 0,009 0,477± 0,033 0,500± 0,029 0,631± 0,006 NDCG@5 0,175± 0,011 0,416± 0,030 0,424± 0,026 0,625± 0,007 yes HR@1 0,203± 0,025 0,351± 0,039 0,323± 0,040 0,615± 0,000 HR@5 0,455± 0,000 0,487± 0,042 0,513± 0,033 0,636± 0,007 NDCG@5 0,336± 0,011 0,421± 0,034 0,422± 0,028 0,626± 0,004
Average HR@k with 𝑘 ∈ {1, 5} and NDCG@5 with standard deviation on the Bestatter dataset over five random seeds for LightSANs and SASRec. ItemEmb indicates the use of Item Embeddings.
LightSANs no HR@1 0,519± 0,042 0,854± 0,006 HR@5 0,828± 0,008 0,942± 0,001 NDCG@5 0,686± 0,019 0,906± 0,003 yes HR@1 0,544± 0,017 0,882± 0,007 HR@5 0,823± 0,003 0,944± 0,000 NDCG@5 0,695± 0,005 0,919± 0,003 SASRec no HR@1 0,509± 0,056 0,845± 0,008 HR@5 0,794± 0,020 0,941± 0,000 NDCG@5 0,659± 0,034 0,901± 0,003 yes HR@1 0,533± 0,027 0,872± 0,011 HR@5 0,826± 0,010 0,943± 0,002 NDCG@5 0,691± 0,010 0,914± 0,005 same structure, meaning that items mentioned in the future can often be inferred from past items without including information about the conversation.
Twitch This is the most challenging dataset due to the very noisy nature of messages on twitch.tv. Consequently, the scores are lower than for other datasets, but the models still yield reasonable performance when using Message Embeddings. Otherwise, the results follow the same trends as on the other datasets.
We have shown that we can adapt arbitrary sequential recommender models to the task of recommending relevant content in chat conversations between users. To this end, we have adapted the recently proposed non-item page modeling to represent the content of chat messages in multiple ways. We have evaluated these models on several datasets, showing that our modeling is successful. We find that the models perform best when given access to pre-trained message embeddings to represent the content of the chat messages.
Our modeling can be applied in a variety of settings, ranging from the direct application of showing relevant content in chat rooms to being used as an intermediate component in a conversational
Average HR@k with 𝑘 ∈ {1, 5} and NDCG@5 with standard deviation on the Twitch dataset over five random seeds for LightSANs and SASRec. ItemEmb indicates the use of Item Embeddings.
ItemEmb Metrics Items Only UTID TE ME LightSANs no HR@1 0,163± 0,009 0,201± 0,006 0,193± 0,020 0,438± 0,020 HR@5 0,402± 0,013 0,293± 0,011 0,286± 0,013 0,575± 0,018 NDCG@5 0,288± 0,008 0,249± 0,010 0,242± 0,012 0,515± 0,008 yes HR@1 0,141± 0,015 0,210± 0,015 0,199± 0,011 0,419± 0,013 HR@5 0,351± 0,012 0,301± 0,010 0,284± 0,026 0,554± 0,032 NDCG@5 0,253± 0,007 0,255± 0,007 0,242± 0,017 0,492± 0,017 SASRec no HR@1 0,156± 0,007 0,207± 0,016 0,202± 0,014 0,422± 0,009 HR@5 0,379± 0,026 0,327± 0,010 0,338± 0,008 0,573± 0,016 NDCG@5 0,275± 0,015 0,270± 0,009 0,272± 0,008 0,507± 0,010 yes HR@1 0,157± 0,020 0,202± 0,011 0,215± 0,011 0,412± 0,028 HR@5 0,397± 0,015 0,322± 0,022 0,346± 0,025 0,563± 0,018 NDCG@5 0,286± 0,013 0,264± 0,015 0,280± 0,015 0,495± 0,012 recommender system.
Our findings imply that any improvements in sequential recommender models can be directly transferred to our task of recommending relevant content to chat conversations. In particular, we see the opportunity to adapt recommender models with access to knowledge graphs to our task. There is potential to improve our results further by providing better content representations for both recommended items and messages (especially in the case of Twitch.tv, where standard embedding models may struggle with the noisy language), which we see as a promising direction for future work.
Average HR@5 with standard deviation on the ReDial-Mention dataset for all models over five random seeds. SubSeq indicates training on subsequences, ItemEmb the use of Item Embeddings.
SubSeq ItemEmb Items Only UTID TE ME BERT4Rec no no 0,074± 0,005 0,035± 0,003 0,037± 0,003 0,046± 0,012 yes 0,072± 0,005 0,035± 0,003 0,036± 0,003 0,037± 0,002 yes no 0,082± 0,003 0,274± 0,027 0,039± 0,006 0,419± 0,007 yes 0,090± 0,005 0,275± 0,030 0,038± 0,006 0,369± 0,024 Caser no no 0,028± 0,003 0,062± 0,009 0,046± 0,006 0,066± 0,008 yes 0,045± 0,005 0,061± 0,006 0,042± 0,009 0,093± 0,011 yes no 0,080± 0,003 0,362± 0,017 0,379± 0,010 0,495± 0,010 yes 0,074± 0,004 0,359± 0,012 0,370± 0,011 0,504± 0,011 Core no no 0,135± 0,003 0,681± 0,008 0,677± 0,010 0,443± 0,027 yes 0,066± 0,004 0,656± 0,012 0,645± 0,012 0,575± 0,016 yes no 0,169± 0,002 0,860± 0,002 0,844± 0,003 0,851± 0,002 yes 0,115± 0,004 0,860± 0,001 0,848± 0,004 0,857± 0,002 GRU4Rec no no 0,028± 0,003 0,382± 0,050 0,409± 0,007 0,331± 0,009 yes 0,046± 0,002 0,441± 0,032 0,441± 0,016 0,334± 0,021 yes no 0,092± 0,003 0,812± 0,005 0,803± 0,003 0,703± 0,003 yes 0,095± 0,004 0,819± 0,004 0,812± 0,003 0,709± 0,006 LightSANs no no 0,117± 0,003 0,213± 0,006 0,218± 0,006 0,618± 0,002 yes 0,096± 0,003 0,214± 0,008 0,215± 0,004 0,616± 0,002 yes no 0,144± 0,004 0,468± 0,010 0,487± 0,003 0,755± 0,013 yes 0,132± 0,004 0,485± 0,006 0,490± 0,016 0,771± 0,004 NARM no no 0,012± 0,006 0,234± 0,017 0,243± 0,019 0,181± 0,019 yes 0,056± 0,003 0,216± 0,012 0,226± 0,019 0,124± 0,018 yes no 0,006± 0,000 0,752± 0,007 0,757± 0,007 0,660± 0,007 yes 0,087± 0,005 0,757± 0,004 0,760± 0,004 0,672± 0,006 NextItNet no no 0,021± 0,005 0,036± 0,005 0,038± 0,007 0,040± 0,003 yes 0,038± 0,009 0,038± 0,006 0,035± 0,007 0,049± 0,012 yes no 0,064± 0,004 0,036± 0,003 0,034± 0,001 0,548± 0,006 yes 0,077± 0,004 0,035± 0,001 0,034± 0,001 0,546± 0,006 SASRec no no 0,117± 0,003 0,258± 0,003 0,244± 0,005 0,617± 0,001 yes 0,097± 0,002 0,251± 0,010 0,257± 0,010 0,613± 0,003 yes no 0,144± 0,002 0,733± 0,036 0,729± 0,022 0,792± 0,004 yes 0,132± 0,003 0,716± 0,004 0,706± 0,017 0,764± 0,005 BERT4Rec no no 0,074± 0,005 0,035± 0,003 0,037± 0,003 0,046± 0,012 yes 0,072± 0,005 0,035± 0,003 0,036± 0,003 0,037± 0,002 yes no 0,082± 0,003 0,274± 0,027 0,039± 0,006 0,419± 0,007 yes 0,090± 0,005 0,275± 0,030 0,038± 0,006 0,369± 0,024 Caser no no 0,028± 0,003 0,062± 0,009 0,046± 0,006 0,066± 0,008 yes 0,045± 0,005 0,061± 0,006 0,042± 0,009 0,093± 0,011 yes no 0,080± 0,003 0,362± 0,017 0,379± 0,010 0,495± 0,010 yes 0,074± 0,004 0,359± 0,012 0,370± 0,011 0,504± 0,011 Core no no 0,135± 0,003 0,681± 0,008 0,677± 0,010 0,443± 0,027 yes 0,066± 0,004 0,656± 0,012 0,645± 0,012 0,575± 0,016 yes no 0,169± 0,002 0,860± 0,002 0,844± 0,003 0,851± 0,002 yes 0,115± 0,004 0,860± 0,001 0,848± 0,004 0,857± 0,002 GRU4Rec no no 0,028± 0,003 0,382± 0,050 0,409± 0,007 0,331± 0,009 yes 0,046± 0,002 0,441± 0,032 0,441± 0,016 0,334± 0,021 yes no 0,092± 0,003 0,812± 0,005 0,803± 0,003 0,703± 0,003 yes 0,095± 0,004 0,819± 0,004 0,812± 0,003 0,709± 0,006 LightSANs no no 0,117± 0,003 0,213± 0,006 0,218± 0,006 0,618± 0,002 yes 0,096± 0,003 0,214± 0,008 0,215± 0,004 0,616± 0,002 yes no 0,144± 0,004 0,468± 0,010 0,487± 0,003 0,755± 0,013 yes 0,132± 0,004 0,485± 0,006 0,490± 0,016 0,771± 0,004 NARM no no 0,012± 0,006 0,234± 0,017 0,243± 0,019 0,181± 0,019 yes 0,056± 0,003 0,216± 0,012 0,226± 0,019 0,124± 0,018 yes no 0,006± 0,000 0,752± 0,007 0,757± 0,007 0,660± 0,007 yes 0,087± 0,005 0,757± 0,004 0,760± 0,004 0,672± 0,006 NextItNet no no 0,021± 0,005 0,036± 0,005 0,038± 0,007 0,040± 0,003 yes 0,038± 0,009 0,038± 0,006 0,035± 0,007 0,049± 0,012 yes no 0,064± 0,004 0,036± 0,003 0,034± 0,001 0,548± 0,006 yes 0,077± 0,004 0,035± 0,001 0,034± 0,001 0,546± 0,006 SASRec no no 0,117± 0,003 0,258± 0,003 0,244± 0,005 0,617± 0,001 yes 0,097± 0,002 0,251± 0,010 0,257± 0,010 0,613± 0,003 yes no 0,144± 0,002 0,733± 0,036 0,729± 0,022 0,792± 0,004 yes 0,132± 0,003 0,716± 0,004 0,706± 0,017 0,764± 0,005 BERT4Rec no no 0,135± 0,024 0,138± 0,019 0,128± 0,029 0,115± 0,071 yes 0,140± 0,039 0,138± 0,019 0,128± 0,029 0,115± 0,071 yes no 0,135± 0,012 0,062± 0,033 0,162± 0,028 0,172± 0,017 yes 0,140± 0,011 0,062± 0,033 0,159± 0,030 0,169± 0,014 Caser no no 0,036± 0,050 0,090± 0,000 0,044± 0,039 0,072± 0,038 yes 0,010± 0,017 0,051± 0,043 0,021± 0,030 0,023± 0,032 yes no 0,148± 0,020 0,149± 0,028 0,136± 0,050 0,228± 0,042 yes 0,203± 0,033 0,113± 0,042 0,136± 0,025 0,282± 0,091 Core no no 0,127± 0,037 0,054± 0,049 0,036± 0,049 0,062± 0,041 yes 0,018± 0,041 0,054± 0,049 0,036± 0,049 0,000± 0,000 yes no 0,138± 0,030 0,131± 0,070 0,162± 0,028 0,126± 0,019 yes 0,158± 0,038 0,156± 0,067 0,149± 0,058 0,123± 0,056 GRU4Rec no no 0,047± 0,034 0,018± 0,040 0,082± 0,047 0,100± 0,091 yes 0,031± 0,028 0,003± 0,006 0,077± 0,043 0,072± 0,064 yes no 0,156± 0,033 0,197± 0,084 0,554± 0,091 0,538± 0,016 yes 0,221± 0,036 0,213± 0,066 0,492± 0,111 0,518± 0,063 LightSANs no no 0,013± 0,000 0,074± 0,021 0,090± 0,000 0,172± 0,007 yes 0,052± 0,000 0,082± 0,017 0,090± 0,000 0,169± 0,006 yes no 0,164± 0,043 0,369± 0,011 0,346± 0,018 0,618± 0,006 yes 0,148± 0,025 0,369± 0,011 0,346± 0,009 0,618± 0,006 NARM no no 0,039± 0,037 0,056± 0,053 0,059± 0,055 0,056± 0,053 yes 0,065± 0,046 0,046± 0,053 0,059± 0,042 0,051± 0,047 yes no 0,120± 0,017 0,328± 0,090 0,379± 0,064 0,428± 0,035 yes 0,218± 0,035 0,367± 0,067 0,403± 0,055 0,421± 0,017 NextItNet no no 0,068± 0,038 0,036± 0,049 0,018± 0,040 0,090± 0,000 yes 0,086± 0,007 0,072± 0,040 0,036± 0,049 0,090± 0,000 yes no 0,127± 0,050 0,180± 0,000 0,154± 0,035 0,174± 0,007 yes 0,148± 0,059 0,180± 0,000 0,167± 0,029 0,154± 0,035 SASRec no no 0,016± 0,006 0,059± 0,042 0,090± 0,000 0,177± 0,017 yes 0,073± 0,046 0,064± 0,048 0,090± 0,000 0,177± 0,017 yes no 0,078± 0,018 0,351± 0,039 0,339± 0,025 0,618± 0,006 yes 0,203± 0,025 0,351± 0,039 0,323± 0,040 0,615± 0,000
Average HR@5 with standard deviation on the Bestatter dataset for all models over five random seeds. SubSeq indicates training on subsequences, ItemEmb the use of Item Embeddings.
BERT4Rec no no 0,224± 0,050 0,324± 0,022 yes 0,159± 0,047 0,322± 0,011 yes no 0,393± 0,036 0,433± 0,014 yes 0,287± 0,036 0,402± 0,021 Caser no no 0,194± 0,040 0,488± 0,025 yes 0,190± 0,037 0,541± 0,007 yes no 0,373± 0,032 0,623± 0,021 yes 0,353± 0,032 0,699± 0,007 Core no no 0,170± 0,017 0,620± 0,014 yes 0,291± 0,032 0,653± 0,009 yes no 0,160± 0,018 0,766± 0,003 yes 0,492± 0,006 0,791± 0,029 GRU4Rec no no 0,236± 0,025 0,594± 0,008 yes 0,228± 0,049 0,621± 0,013 yes no 0,411± 0,029 0,799± 0,028 yes 0,490± 0,024 0,847± 0,026 LightSANs no no 0,309± 0,030 0,620± 0,031 yes 0,289± 0,013 0,626± 0,012 yes no 0,519± 0,042 0,854± 0,006 yes 0,544± 0,017 0,882± 0,007 NARM no no 0,206± 0,017 0,568± 0,003 yes 0,217± 0,030 0,590± 0,020 yes no 0,403± 0,043 0,738± 0,012 yes 0,434± 0,025 0,783± 0,016 NextItNet no no 0,222± 0,039 0,194± 0,077 yes 0,180± 0,035 0,194± 0,077 yes no 0,387± 0,029 0,056± 0,000 yes 0,363± 0,024 0,057± 0,002 SASRec no no 0,235± 0,056 0,585± 0,020 yes 0,288± 0,045 0,641± 0,024 yes no 0,509± 0,056 0,845± 0,008 yes 0,533± 0,027 0,872± 0,011
Average HR@5 with standard deviation on the Twitch dataset for all models over five random seeds. SubSeq indicates training on subsequences, ItemEmb the use of Item Embeddings.
BERT4Rec no no 0,242± 0,014 0,148± 0,038 0,205± 0,031 0,228± 0,025 yes 0,232± 0,020 0,148± 0,035 0,205± 0,031 0,228± 0,025 yes no 0,348± 0,008 0,198± 0,017 0,232± 0,011 0,225± 0,046 yes 0,336± 0,040 0,203± 0,013 0,237± 0,014 0,197± 0,042 Caser no no 0,126± 0,007 0,132± 0,010 0,123± 0,034 0,117± 0,014 yes 0,091± 0,019 0,135± 0,020 0,141± 0,026 0,098± 0,052 yes no 0,332± 0,057 0,150± 0,047 0,153± 0,065 0,200± 0,052 yes 0,267± 0,043 0,151± 0,053 0,103± 0,052 0,188± 0,085 Core no no 0,368± 0,039 0,138± 0,019 0,139± 0,019 0,147± 0,028 yes 0,182± 0,031 0,148± 0,024 0,138± 0,017 0,166± 0,010 yes no 0,485± 0,007 0,373± 0,020 0,346± 0,058 0,494± 0,045 yes 0,341± 0,103 0,414± 0,039 0,368± 0,014 0,438± 0,024 GRU4Rec no no 0,128± 0,024 0,111± 0,048 0,122± 0,026 0,149± 0,015 yes 0,115± 0,036 0,120± 0,034 0,130± 0,015 0,141± 0,014 yes no 0,360± 0,014 0,332± 0,029 0,462± 0,011 0,511± 0,011 yes 0,384± 0,024 0,332± 0,029 0,455± 0,008 0,486± 0,025 LightSANs no no 0,188± 0,016 0,118± 0,022 0,103± 0,019 0,178± 0,014 yes 0,121± 0,019 0,111± 0,035 0,111± 0,017 0,186± 0,017 yes no 0,402± 0,013 0,293± 0,011 0,286± 0,013 0,575± 0,018 yes 0,351± 0,012 0,301± 0,010 0,284± 0,026 0,554± 0,032 NARM no no 0,167± 0,013 0,097± 0,034 0,085± 0,051 0,108± 0,031 yes 0,178± 0,023 0,114± 0,035 0,088± 0,064 0,149± 0,016 yes no 0,353± 0,040 0,345± 0,047 0,352± 0,029 0,396± 0,045 yes 0,359± 0,036 0,356± 0,043 0,369± 0,024 0,420± 0,048 NextItNet no no 0,105± 0,035 0,108± 0,035 0,164± 0,029 0,096± 0,027 yes 0,101± 0,024 0,117± 0,031 0,160± 0,022 0,119± 0,009 yes no 0,287± 0,045 0,231± 0,021 0,234± 0,018 0,219± 0,035 yes 0,211± 0,071 0,244± 0,005 0,236± 0,021 0,227± 0,007 SASRec no no 0,203± 0,015 0,118± 0,023 0,113± 0,018 0,181± 0,015 yes 0,141± 0,014 0,106± 0,031 0,102± 0,017 0,181± 0,009 yes no 0,379± 0,026 0,327± 0,010 0,338± 0,008 0,573± 0,016 yes 0,397± 0,015 0,322± 0,022 0,346± 0,025 0,563± 0,018
This work is partially funded by the German Federal Ministry of Education and Research (BMBF) under grant number 01IS22051B in the KILiMod project.