The rising popularity of conversational agents for accessing information stems from their natural language dialogue-based interaction, facilitating usability for a broad spectrum of users, including the elderly, children, and visually impaired individuals. Among others, two tasks that benefit the most conversational agents are search and recommendation: in the former, the user receives factual information by asking the agent; in the latter, the system refines its knowledge of the user's needs by posing them a sequence of questions. This work discusses the observations and findings of the first CAMEO (Conversational Agents: Mastering, Evaluating, Optimizing) project retreat. The retreat focused on similarities and diferences of conversational search and recommendation to identify the path to construct a joint conversational search and recommendation system. Our observations highlight how all the conversational search/recommendation systems can be categorized using two axes: “explorationdisambiguation” and “search-recommendation”. The first axis describes whether the question aims to gain knowledge over something unknown or allows to refine already available knowledge. The second axis describes if the user's interest is in gaining knowledge or obtaining a recommendation. Additionally, we provide insights on obtaining a dataset that can be used to train/test such a joint system. Finally, we describe how the CAMEO project will address the product search task, which we believe is the scenario where the joint conversational search and recommendation system would be the most efective.
The conversational paradigm is increasingly used both for search and recommendation. For
what concerns Conversational Search (CS), instead of issuing a single query, the user satisfies
their information need by refining it through a conversation with the agent. At the same
time, concerning Conversational Recommendation (CR), instead of receiving a one-shot static
recommendation, users can establish a multi-turn dialogue with the CR to express their needs
best. The conversational paradigm presents several advantages. It can be easily used by visually
impaired people, elderly and children, who can interact with the system using their voice. At the
same time, the possibility of expressing their need using natural languages alleviates the mental
encumbering of formulating efective queries for generic users. Nevertheless, conversational
systems also present major challenges that must be efectively addressed. For example, the
system must understand and process complex natural language structures, such as anaphoras,
ellipses, and co-references. At the same time, the system should be able to keep track of the
conversation’s state to provide efective and satisfactory answers for the user, also considering
previous interactions. Finally, the evaluation of these systems is particularly challenging for
several reasons. First, being a relatively new paradigm, we still lack rich and extensive evaluation
collections. Secondly, we lack proper evaluation measures and paradigms. Indeed, most of
the current evaluation approaches [
Currently, CS and CR are intended and applied as two orthogonal tasks: either the system is
meant to search, or it is designed to recommend. We argue that this approach is limiting and
does not exploit the full potential of a conversational system to interact seamlessly with a user.
Whether to answer with a document (search) or an item (recommendation) should depend on
the user’s interaction and needs rather than the system’s nature. Furthermore, past eforts in
the joint recommendation and search [
The “Conversational Agents: Mastering, Evaluating, Optimizing (CAMEO)” project1 addresses the abovementioned limitations while fostering the development of conversational agents designed for joint search and recommendation. CAMEO is being developed under the Progetti di Rilevante Interesse Nazionale (PRIN) framework and involves partners from four institutions: The National Research Council (CNR), the Polytechnic University of Bari (POLIBA), the Sapienza University of Rome and the University of Padua (UNIPD, coordinator). In this discussion paper, we present the outcomes of the first brainstorming meeting of CAMEO, organized as a retreat of three days at San Vito di Cadore, to explore the notion of joint recommendation and search for conversational and to define a common conceptual framework. Our discussion focused on formalizing the similarities and diferences between CS and CR systems. We also discussed which type of architecture would fit the best. We observed that using two diferent engines (one for recommendation and one for search) that share some information, either in the form of a knowledge base or representation space, would be the most efective solution in our case. Finally, we discussed how a joint search and recommendation dataset could be collected.
The remainder of this paper is organized as follows: Section 2 introduces the background on CR and CS, Section 3 details the observations and analyses of the current state-of-the-art resulting from the first CAMEO’s project meeting. Section 4 introduces conversational product recommendation, the first task CAMEO will focus on. Finally, Section 5 draws the conclusion and introduces our future work.
We describe here CS and CR paradigms, highlighting their similarities and diferences.
Our definition of CS follows the one adopted by the TREC Conversational Assistance Track (CAsT)
evaluation campaign [
System (response): D1: apples are a very healthy type of winter fruits.
D2: oranges are winter fruits.
D3: strawberries are not winter fruits.
D4: winter is the coldest season of the year.
User (utterance 2): tell me more about which types of apples exist
System (response): D1: Golden Delicious is a very famous variety of apples.
D2: There are over 7500 cultivars of apples.
D3: Navel is a popular orange cultivar.
The user issues the first utterance (utterance 1), and the system retrieves documents in
response. Among such documents, some will be relevant, some will not. This allows us to
compute IR evaluation measures such as nDCG, AP, and Precision, and measures designed for
CS [
More recently, the mixed-initiative task has emerged [
Regarding CR, the task presents similar challenges to CS (i.e., handling natural language and
exploiting the context), but follows a quite diferent search paradigm. Also in this case we can
have three diferent interaction strategies: system driven, user driven, and mixed initiative [
The user asks, the system responds The system asks, the user responds, or viceversa There are no domain-dependent operations: the user The system’s questions depend on the domain. conditions the domain with their questions. several conversation goals: several diferent information needs, one per utterance. single conversation goal: items to be recommended.
In the most typical scenario, no user profile can be used It is possible to condition the search space based on to condition the answers. both the conversation and the user profile User: fruits User: I prefer winter ones System: winter fruits or summer fruits?
System: vegetables or fruits? System: you should try apples
When applying the slot-filling CR paradigm, it is possible to extend the task with additional
context, e.g., knowing the current season, the system might avoid questions. Some information
about users can also be extracted from their profiles that can store long- and/or short-term
preferences (e.g., the user likes red fruits). A possible slot-filling approach could be based on
“backtracking”: with each system’s question, the search space is further partitioned [
The evaluation concerns efectiveness, eficiency (or interaction cost), and linguistic aspects
[
Diferences Based on the paradigms described in Section 2, we analyse here the main diferences between CS and CR that should be taken into consideration when designing a joint CS and CR framework. Table 1 summarizes our observations.
The first two diferences reported in Table 1 concern what the user expects to do with the system. With the archetypal CS system, the user is expected to lead the conversation. On the contrary, the typical CR system could have a mixed interaction. When a user interacts with a CS system, they aim at “exploring” the corpus to gain more information, without having any previous knowledge of it. On the other hand, when a CR system interacts with a user, the system is aware that the user has a sort of ideal item in mind and needs to “disambiguate” among many diferent possibilities to identify the right one. This leads to the third diference between the two classes of systems. The CS approach does not require specific knowledge of the domain: as the user does not have any knowledge of the relevant documents and their characteristics, it is not possible to use domain knowledge to restrict the search space. Vice-versa, when it comes to the recommendation task, the CR system needs to exploit its domain knowledge to refine the answer. For example, if the CR system works on the movie domain, the question “with actors should have acted in it” might be particularly efective. The same question is useless in other domains, such as music or product recommendation.
Finally, the CS system seldom employs a user profile, while it is natural for the CR system to employ characteristics of the user to provide an efective recommendation.
Similarities The first similarity between the two tasks is linked to the challenges that arise when natural language is involved. Both CS and CR systems need to deal with anaphoras, coreferences, and other complex linguistic structures.
The conversation’s structure is, in our opinion, the most relevant point of contact between
CS and CR. In both cases, the conversation could be formalized as a conversation tree. When
formalizing a conversation in the CS scenario, it is possible to use a tree [
Categorizing the conversational systems Based on our analysis, we believe that each conversational system can be characterized according to two axes: • exploration/disambiguation. A pure exploration system is based on a user that asks and the system that responds. The information is available only to the system and the user has to extract from it, by “exploring”. A pure disambiguation system asks questions, and the user responds to help the system disambiguate their interest. • search/recommendation: in the search scenario, the user is interested in collecting information and discovering something on a topic of interest. Vice-versa, when it comes to the recommendation, the focus is on the system providing suggestions based on a set of predefined options (i.e., the catalogue). At the same time, in search, the research is guided by the user’s query, while in recommendation there is no actual query, and the query corresponds to the user’s profile.
Following this schema, we can identify five scenarios: • full exploration, full search. This scenario corresponds to the current CS: the user has a piece of information and explores the corpus (guided by the system) to satisfy it. • full disambiguation, full recommendation. This case is the current CR: the user has the (latent) information, and the system tries to disambiguate it by asking questions. • full exploration, full recommendation. The user is not interested in receiving a recommendation but rather in exploring the catalogue of the RS. In a sense, in this scenario, the user is interested in meta-data, such as what are the most popular songs, which movies belong to a certain genre, and so on. • full disambiguation, full search. The user knows a relevant document exists and needs to access it via disambiguation. An example of this scenario is represented by navigational queries used in the CS context: the user knows with certainty which page they are trying to access and guides the system in understanding it as well. • Mixed scenarios. This class of systems includes for example the mixed-initiative in IR: the system is allowed to ask questions to disambiguate their information need. Other mixed situations could be a system that, besides providing recommendations, accompanies them with pieces of text explaining the item(s) in detail. For example, a user searching for a song might receive recommendations and documents describing the story of the recommended songs, details on genres, documents on the artist, and other information derived from a corpus of documents.
Due to its objective, models developed by CAMEO belong to the last category.
During our brainstorming at CAMEO retreat, we identified three main strategies to design a
joint conversational search and recommendation system:
• use a unique module. This approach would rely on using a Large Language Model (LLM),
an approach that proved to be efective for both CS and CR tasks. This approach would
be relatively straightforward to realize, but it is hard to make it efective and evaluate it.
• use two separate modules, with a classifier deciding the type of module to use. In
this case, some of the requests will be redirected to the IR engine while some to the RS.
The major advantage is the simplicity of the approach: it is easy to define, train, and
operationalize the two distinct models, as they are more or less the currently available
conversational IR and RS systems. The major challenges regard i) the construction of the
classifier and ii) the absence of shared knowledge as it decreases the power of the tool: it
is impossible to improve a component based on the interactions of the other.
• use two separate modules, with shared elements. The two modules could, for
example, be based on the same knowledge base or rely on the same latent representation
space where documents and items are represented jointly. Then, when queried by the user,
the system uses a classifier to decide which module to use and filter the latent space based
on this. Notice that this approach shares with the previous one the challenges linked
to obtaining the classifier. Furthermore, several eforts highlighted greater efectiveness
thanks to the usage of a joint search and recommendation space [
One of the major challenges in realizing a joint conversational search and recommendation system is the fact that there are no currently available datasets.
The first possibility would be to extend an existing dataset meant for either search or
recommendation with additional ground truth for the other task. For example, it could be possible
to integrate the CAsT dataset with recommendation components. This requires to partially
re-annotate the dataset. Nevertheless, this has the advantage that, while collecting new
annotations, we could extend the existing dataset with additional meta-information such as linguistic
aspects (the presence of anaphoras), links between utterances (what is the tree structure of the
conversation), and external elements such as DBpedia [
On the other hand, we could assume to extend a conversational recommendation dataset. The problem is less encumbering than in conversational search, as, by using the paradigm described in Section 2, it is possible to use any currently available RS dataset to simulate a conversation. This reduces the realism of the evaluation, but it grants access to a large amount of data. Therefore, it would be necessary to combine the items and categories available in a recommendation dataset, with documents available in a corpus. Then, we can annotate documents for relevance about questions arising on the recommendation dataset’s domain.
Considering the limitations linked to annotating the conversational search dataset for the recommendation task – namely the absence of a user’s profile, we consider the strategy based on turning a recommendation dataset into a set of simulated conversations and annotating such conversations the most efective approach.
RESEARCHER DEVELOPERS
O3 search and conversational recommendation engines. The dialog manager interacts with the user, granting high quality answers from the linguistic point of view. Ideally, it could be implemented using a Large Language Model. Below the dialog manager, two separate components, the CS and RS models, are responsible of identifying the contents to be presented to the user. This is in line with our observations during the first CAMEO retreat, where we considered a system with two separate internal subsystems more efective in our specific scenario as well as more easy to evaluate. These three components share an internal knowledge, containing short- and long-term user preferences as well as the state of the current dialog and its dependencies. Furthermore, such shared internal knowledge will contain the joint space where represent recommendation items and search documents. The external knowledge includes all the external data used by the system, among which corpora of documents, catalogs of items, knowledge bases, and ontologies.
The visual analytics environment (O2) will allow researchers and developers to visually and interactively explain the behaviour of the conversational agent, as the dialog progresses. Moreover, it will serve to explore, analyze, and make sense of the performance of the conversational agent, easing the optimization process.
After taking into account the aspects mentioned above, we consider product search the scenario where CAMEO might provide the biggest advantages.
Product search consists of guiding a user into finding a specific product: the user knows what they need, but does not know which specific instance of that product satisfies their needs. The user needs something possibly present in the catalogue and the system, with its knowledge, guides them in finding the most suited item.
Assume for example a user searching for a generic product. The RS can contextualize the recommendation based on the user profile (e.g., if they prefer low-cost products, colour, material, etc). We can imagine an exploration component which provides information on the search of the user. The system can provide information to expand the user’s knowledge, such as detailing the varieties of products available, the diferences between brands and so on.
In the conversational scenario, we could consider a two-phase interaction. During the ifrst (expectoration) phase the user gathers knowledge about the product: the user leads the conversation and the system responds. In the second (disambiguation) phase the system guides the conversation through questions that allow to reduce the search space (the system leads the conversation and the user responds), driving the user toward a specific instance of the product.
The annotations protocol is similar to the one in IR and RS. The system provides the documents and the annotator assesses whether such documents are relevant to the user’s query. For example, if the user’s query is “I would like to buy fruit”, relevant documents are those concerning fruits, such as which types of fruit exist and what are nutritional properties of diferent fruits. For RS, the evaluation focuses on the fact that the user was recommended an item relevant to them.
Additionally, the system could be integrated with analytics that might help in diversifying the sold products, thus increasing fairness. Assume a scenario in which a producer has certain items for which they wish to bust selling. Given two almost equivalent items – at least based on what the user considers relevant –, it would be desirable if the conversation could nudge the user towards the item that the seller wishes to sell more. Thus, the analytics integrated within CAMEO conversational systems could help in answering questions such as: • Are there words or sets of words that better lead the user to pick one item or the other? • Are there sequences of questions that guide the user in choosing one item over the other? • Are documents that, if presented during the first phase, better nudge the user toward choosing a specific product? • Is the sentiment of the questions and documents retrieved capable of influencing the “appeal” of an item to a specific user?
In conversational search, users obtain factual information through direct queries, while in conversational recommendation systems, user preferences are refined through a series of inquiries stimulated by the agent. The CAMEO project retreat served to go in-depth into these tasks, exploring their similarities and disparities. Our findings highlighted the utility of categorizing conversational systems along axes of exploration-disambiguation and searchrecommendation for the roles played by the user and the intelligent agent. Moreover, we discussed the need to acquire datasets to train and evaluate joint conversational search and recommendation systems. Looking forward, the CAMEO project aims to tackle the product search domain, envisioning a scenario where the amalgamation of conversational search and recommendation capabilities will prove most impactful.
A lot of future work for CAMEO emerged from our discussion. Firstly, the development of joint conversational search and recommendation systems necessitates a deeper understanding of user intent and system capabilities. Additionally, creating comprehensive datasets in the product search scenario remains paramount. Finally, the methodology and the algorithmic approaches to seamlessly integrate search and recommendation functionalities will be a focal point, emphasizing eficiency, efectiveness, and appropriate evaluation metrics.
This work was partially supported by CAMEO, PRIN 2022 n. 2022ZLL7MW.