Multimodal recommender systems integrate diverse information sources, such as text and visual data, into predictive models for personalization. While multimodality promises richer representations and potentially improved relevance, it remains an open question whether multimodal inputs genuinely enhance recommender systems. In this talk, I present findings from an evaluation of reproducible multimodal RecSys models, designed to address this question. The results show that multimodality does not consistently lead to better performance. I argue that while multimodal inputs can yield incremental gains, their efectiveness must be considered in relation to user interaction dynamics, task objectives, and system context - their value is inherently contextual.
Recommender systems (RecSys) are foundational technologies for digital platforms ranging from ecommerce to media streaming. Traditional systems primarily rely on user-item interactions, leveraging collaborative filtering or content features extracted from text. However, the proliferation of rich multimedia content has spurred a growing body of research on multimodal recommendation, where heterogeneous signals — such as text descriptions, product images, and short videos — are fused into the recommendation pipeline.
As a trending topic in RecSys, multimodal recommendation has attracted significant attention. Yet,
a central question remains: Does multimodality truly improve recommender systems? Interestingly,
three groups have conducted independent studies around the same time, each aiming to answer this
question from diferent perspectives and experimental setups [
While the community increasingly emphasizes reproducibility, not all papers release their source
code or datasets for various reasons. Among the 41 papers, 12 were considered reproducible — meaning
both code reproducible (source code publicly available and functioning correctly) and dataset available
(publicly accessible datasets or raw data with preprocessing scripts). For benchmarking, the team
used three datasets: two e-commerce datasets (Amazon and Taobao) and one short-video dataset (DY).
Largely following the experimental settings of the reproducible papers, each dataset was randomly
partitioned into training, validation, and test sets with an 8:1:1 ratio. Although this random split can
introduce data leakage [
The team conducted multiple evaluations focusing on the role of multimodality. To assess its benefits, two classic baselines — ItemKNN and UserKNN — were employed, both relying solely on interaction data. Experimental results show that these traditional KNN-based methods perform comparably to, or even better than, several sophisticated multimodal recommendation models. Moreover, when comparing results across the three datasets, no consistent advantage of multimodality emerges. In particular, very few methods outperform the KNN baselines on the DY dataset.
The team then conducted experiments comparing the recommendation accuracy of multimodal and single-modality models. Interestingly, multimodal systems do not always achieve the best performance compared to their single-modality counterparts. Furthermore, diferent types of modal information contribute diferently depending on the recommendation scenario. Specifically, in e-commerce settings, textual features often play a more important role, whereas visual information tends to be marginally more useful for short-video recommendations.
While many more interesting findings are presented in [
Before addressing the question of multimodality specifically, it is helpful to view recommender systems from a broader perspective. ACM RecSys is a conference under SIGCHI, emphasizing human-computer interaction. Yet, many academic papers in the field focus primarily on algorithmic or methodological advances, often overlooking the HCI dimension.
Recently, we conducted a survey of real-world recommender systems, including only those studies
that reported online A/B testing results on production systems [
In particular, we categorize commonly observed RecSys tasks into two broad types: TransactionOriented RecSys and Content-Oriented RecSys. The goal of the former is to drive transactional actions — optimizing for conversion rates, revenue, or purchase likelihood. E-commerce platforms are typical examples. In contrast, the latter focuses on promoting user consumption and engagement, optimizing for metrics such as dwell time, clicks, or user satisfaction to encourage continued interaction, such as watching videos, listening to music, or reading news articles.
Multimodal RecSys must be situated within the larger ecosystem of recommendation. Research often narrows to algorithmic accuracy, but real-world systems involve broader considerations: interfaces, feedback loops, optimization objectives, and operational constraints like cost and latency. Evaluation diferences between ofline metrics and online A/B testing further complicate insights.
The efectiveness of diferent modalities depends heavily on context, visibility, and the extent to
which these modalities influence user decision-making [
Mixed findings suggest no universal superiority of multimodality. Domain specificity, task dependence, evaluation design, and user visibility all shape outcomes. While multimodality enriches representation, its value is inherently contextual.
During the preparation of this work, the author used ChatGPT-5 for spelling check and sentence polishing. After using this tool, the author reviewed and edited the content as needed and takes full responsibility for the publication’s content.