Conversational Recommender Systems (CRSs) leverage multi-turn interactions to assist users in decision-making. The quality of these systems is heavily dependent on the datasets used for their training. Existing datasets, often created by crowdworkers or proprietary Large Language Models (LLMs), frequently lack conversational coherence, fail to model diverse user behaviors, and sufer from reproducibility issues. To address these limitations, we introduce DistillRecDial, a conversational recommendation dataset that incorporates varied user personas and interaction patterns. The dataset is generated using a Large-to-Small Language Model Distillation pipeline, enabling dialogue synthesis without reliance on closed, resource-intensive LLMs. We model user heterogeneity in preferences, goals, and dialogic behavior, resulting in a more realistic and diverse corpus. Human and automated evaluations indicate that DistillRecDial surpasses existing datasets in dialogue quality and diversity. To promote reproducible research, the dataset and generation code are publicly released and integrated into the CRSLab framework. This article is a discussion paper for our work accepted to the Reproducibility Track at ACM RecSys 2025; the code and resources can be found in [12].
Conversational Recommender Systems (CRSs) enhance user engagement by providing personalized
recommendations through multi-turn dialogue [
Early CRS datasets like ReDial [
Synthetic data generation using LLMs has emerged as a scalable alternative. Initial eforts converted
single-turn interactions to dialogues [
Our work difers by systematically modeling user variation through stereotypes. We utilize a
knowledge distillation pipeline [
The generation of DistillRecDial was designed to produce high-quality dialogues reflecting diverse user behaviors. The pipeline involves grounding conversations in real user data, defining user stereotypes to guide dialogue flow, and using knowledge distillation for scalable generation. A summary of dataset statistics is provided in Table 1.
To ground dialogues in realistic preferences, we utilized the Amazon Reviews dataset, focusing on the Movies and TV category. After applying a 12-core filter to ensure user profiles were suficiently dense, the data contained 23,456 users and 15,597 items. We enriched item metadata with information from The Movie Database (TMDB), including genres, keywords, and cast, to enable content-based conversations.
A core feature of our work is the modeling of user heterogeneity. We defined five user stereotypes by combining two behavioral dimensions: Preference Expression (None, Implicit, Explicit) and Intention Clarity (None, Vague, Specific), based on prior user modeling literature [ 14]. (1) Curious Newcomer: No history, vague goal. (2) Focused Newcomer: No history, specific goal. (3) History-Based Browser: Has history, no specific goal. (4) Guided Explorer: Has history, seeks novelty. (5) Preference-Driven Searcher: Has history, expresses explicit and complex preferences.
For each of the 23,456 users, we randomly assigned one of the five stereotypes. A structured prompt was created containing: (1) a description of the user’s persona and goal, (2) a target item to be recommended, and (3) the user’s interaction history (if applicable), with features conditioned on the assigned stereotype (Table 2). 1. Teacher Generation: A teacher model (LLaMA 3.3 70B) generated high-quality dialogues for 10% of the prompts. These dialogues served as exemplars. 2. Knowledge Distillation: A smaller student model (LLaMA 3.1 8B) was fine-tuned on the teacher-generated dialogues. This ofline distillation transfers the teacher’s stylistic and structural capabilities to the student [13]. 3. Scalable Generation: The fine-tuned student model generated dialogues for all 23,456 prompts.
After filtering for hallucinations, the final dataset contains 21,039 high-quality dialogues. Example snippets are shown in Table 3.
We evaluated DistillRecDial to address three research questions regarding dialogue quality (RQ1), diversity (RQ2), and downstream task performance (RQ3). We compared our dataset with ReDial, INSPIRED, and PEARL.
We conducted a head-to-head human evaluation where 10 evaluators compared 100 dialogues from DistillRecDial against 100 from each baseline. As shown in Figure 1, DistillRecDial was consistently preferred across all criteria, including naturalness, relevance, and consistency. It outperformed humangenerated datasets (ReDial, INSPIRED) by a wide margin and also demonstrated higher quality than the LLM-generated PEARL dataset.
To assess diversity, we computed the average pairwise cosine similarity of utterances at each turn for DistillRecDial and PEARL. As illustrated in Figure 2, DistillRecDial exhibits significantly lower turn-level similarity (0.422) compared to PEARL (0.598), indicating greater linguistic diversity. PEARL’s high similarity in early turns is due to repetitive openings, whereas our stereotype-driven approach generates more varied initial user requests.
We benchmarked standard CRS models on DistillRecDial using the CRSLab framework.
Recommendation Performance: As shown in Table 4, BERT, which leverages rich textual context, achieved the strongest recommendation performance (Hit@10 = 0.1728). Sequential models like SASRec performed less favorably, suggesting that the context-rich dialogues in our dataset are well exploited by pre-trained language models. Integrated CRS models like KBRD, INSPIRED, and ReDial struggled, indicating that they do not generalize well to the complex conversational structures present in DistillRecDial. This highlights the challenge of jointly optimizing for recommendation and dialogue on realistic, user-aware data and points to opportunities for developing more advanced CRS architectures.
We introduced DistillRecDial, a large-scale, diverse, and reproducible conversational recommendation dataset. By modeling distinct user stereotypes and employing a knowledge distillation pipeline with open-source models, we address key limitations of prior datasets related to behavioral homogeneity and reliance on proprietary LLMs. Our evaluations confirm that DistillRecDial exhibits higher dialogue quality and diversity. By integrating it into the CRSLab framework, we provide a robust benchmark to spur the development of next-generation CRSs capable of adapting to a wide range of user interaction styles.
This research is partially funded by the PNRR project FAIR—Future AI Research (PE00000013), Spoke 6—Symbiotic AI, under the NRRP MUR program supported by NextGenerationEU (CUP H97G22000210007), and the PHaSE project — Promoting Healthy and Sustainable Eating through Interactive and Explainable AI Methods, funded by MUR under the PRIN 2022 program - Finanziato dall’Unione europea - NextGeneration EU, Missione 4 Componente 1 (CUP H53D23003530006).
The models are developed using the Leonardo supercomputer with the support of CINECA-Italian Supercomputing Resource Allocation, under class C projects IscrC_LLM-REC (HP10CTEUGX) and IscrC_SYMBREC (HP10C1A4P8).
During the preparation of this work, the author did not use any AI tool. [13] Anup Shirgaonkar, Nikhil Pandey, Nazmiye Ceren Abay, Tolga Aktas, and Vijay Aski. Knowledge distillation using frontier open-source llms: Generalizability and the role of synthetic data. arXiv preprint arXiv:2410.18588, 2024. [14] Junjie Zhang, Ruobing Xie, Yupeng Hou, Xin Zhao, Leyu Lin, and Ji-Rong Wen. Recommendation as instruction following: A large language model empowered recommendation approach. ACM Trans. Inf. Syst., December 2024. [15] Kun Zhou, Xiaolei Wang, Yuanhang Zhou, Chenzhan Shang, Yuan Cheng, Wayne Xin Zhao, Yaliang Li, and Ji-Rong Wen. Crslab: An open-source toolkit for building conversational recommender system. arXiv preprint arXiv:2101.00939, 2021.