Usability is a key factor in the efectiveness of recommender systems. However, the analysis of user interfaces is a time-consuming process that requires expertise. Recent advances in multimodal large language models (LLMs) ofer promising opportunities to automate such evaluations. In this work, we explore the potential of multimodal LLMs to assess the usability of recommender system interfaces by considering a variety of publicly available systems as examples. We take user interface screenshots from multiple of these recommender platforms to cover both preference elicitation and recommendation presentation scenarios. An LLM is instructed to analyze these interfaces with regard to diferent usability criteria and provide explanatory feedback. Our evaluation demonstrates how LLMs can support heuristic-style usability assessments at scale to support the improvement of user experience.
Recommender systems are a central component of many digital platforms, where they provide
personalized item suggestions to help users navigate large sets of options [
Traditional usability evaluation methods include usability testing with real users [
To reduce this efort, automated solutions such as rule-based tools and heuristic checkers have been
proposed [
LGOBE
https://ase.sai.tugraz.at/ (S. Lubos); https://ase.sai.tugraz.at/ (A. Felfernig); https://ase.sai.tugraz.at/ (D. Garber);
CEUR
ceur-ws.org
While these approaches address general usability, they have not yet been applied to the specific challenges of recommender system interfaces, such as explainability, feedback mechanisms, and preference elicitation workflows, which are central to the user experience in this context. In this work, we explore how a multimodal LLM can help to analyze the usability of ten publicly available recommender interfaces based on explicitly defined criteria. We review the analysis results to highlight the feasibility and benefits of automated usability analysis for recommender interfaces and outline directions for future research.
The paper is organized as follows: Section 2 describes the experimental setup and implementation details. Section 3 presents the results. Section 4 discusses implications and future work. Finally, the paper is concluded in Section 5.
A lot of focus in recommender system research is put on the accuracy of algorithms and personalization
strategies [
The following sections outline our considered recommender scenarios, usability criteria, and describe the LLM-based analysis in detail.
To explore the LLM-based usability analysis across a diverse set of recommender interfaces, we selected ten publicly accessible platforms from various item domains, which are summarized in Table 1. These systems vary in layout complexity, interaction mechanisms, and types of recommendations. This way, the automated usability analysis could be reviewed for varying contexts to get an impression about its generalizability. Each platform was assessed in two typical usage scenarios: (i) preference elicitation, and (ii) recommendation presentation.
Amazon Goodreads Google News KaptnCook Last.fm Netflix Pinterest Spotify Steam YouTube
E-commerce Books News Articles Recipes Music Movies & TV Shows Visual Content Music Video Games Videos
URL https://www.amazon.com https://www.goodreads.com https://news.google.com https://www.kaptncook.com https://www.last.fm https://www.netflix.com https://www.pinterest.com https://open.spotify.com https://store.steampowered.com https://www.youtube.com
To have a comparable situation for each platform, we considered a new user scenario of a user interacting with the recommender for the first time. For this purpose, we used a desktop browser in incognito mode to avoid personalization efects and simulate a first-time user experience. 1 For each platform, we captured a representative screenshot2 for the usage scenarios. Depending on the platform, the preference elicitation either showed the default onboarding screens or initial filter options. The 1A new user account was registered if needed to use the application. 2Screenshots were taken at full resolution and included the visible viewport with relevant UI context (e.g., navigation bars, iflters, recommendation labels). recommendation presentation showed either the main homepage (dashboard) or a detail page including recommended items.
To ensure comparability across platforms while accounting for platform-specific interaction patterns, we defined explicit user tasks for each platform. This allowed us to maintain a consistent evaluation structure while respecting the nuances of individual interfaces. The tasks for both scenarios and all platforms are presented in Table 2. They were used to select the screenshots and provide contextual information to the LLM during the usability analysis.
Amazon Goodreads Google News KaptnCook Last.fm Netflix Pinterest Spotify Steam YouTube
Search for “Bluetooth headphones” Review the related product recommenand interact with product listings to dations shown on the product or search express shopping intent. results page.
Rate previously read books as part Review the initial book recommendations of the onboarding process to express generated based on ratings. reading preferences.
Select preferred news topics or regions Review the personalized news feed on the during setup to tailor content delivery. dashboard.
Indicate disliked ingredients during Review the list of recommended recipes the initial setup to personalize meal based on stated preferences. suggestions.
Choose a trending artist to indicate Review the list of recommended tracks music preferences. or artists based on the selected input. Select at least three preferred titles Review the personalized dashboard with during the onboarding setup to ex- recommended movies and shows. press content preferences.
Select inspirational images reflecting Review the personalized feed with recompersonal interests during the onboard- mended visual content. ing process.
Add songs to playlists based on initial Review the homepage or dashboard with recommendations to express musical recommended tracks. preferences.
Apply filters (e.g., ”Indie” genre) while browsing to express game preferences.
Review the recommended games displayed on the store homepage or Discovery Queue.
Review the recommended videos shown after watching the selected content.
Search for and watch a video on tennis serve drills to signal viewing preferences.
To analyze the usability of the recommender system interfaces in a structured way, we defined a set
of criteria that are shown in Table 3. These are based on general established usability principles [
We used the gemini-2.5-flash model by Google for the LLM-based usability analysis [18]. This model was designed to process textual and visual inputs, which makes it suitable for our experiments. We instructed the model in Python using the Gemini Developer API.3 To improve the reproducibility of
General (Both Scenarios) Preference Elicitation Recommendation Presentation
G1. Is the layout clear and visually structured? G2. Are interactive elements (e.g., buttons, icons) clearly recognizable? G3. Is the amount of information per item appropriate and helpful? G4. Are interface elements used consistently (e.g., icons, labels, colors)? P1. Can users explicitly express preferences (e.g., ratings, likes, categories)? P2. Is there transparency about how input afects recommendations? P3. Do users have control and flexibility (e.g., skip, edit, undo inputs)? R1. Are recommendations clearly labeled as such? R2. Are diferent types of recommendations distinguishable (e.g., ”Because you liked...”)? R3. Are there explanations for why items are recommended?
R4. Can users interact with recommendations (e.g., rate, hide, save)? results, we set the temperature to 0.0 for more deterministic LLM output.
For the analysis task, the recommender interface screenshot was provided as context, together with a high-level description of the platform, the considered usage scenario (preference elicitation or recommendation presentation), and an explicit user task (see Table 2). To define the role and boundaries of the LLM in this scenario, we used the system prompt shown in Figure 1.
We analyzed 10 platforms across 2 usage scenarios each, and considered 11 diferent usability criteria. This resulted in 150 individual assessments (respecting partly scenario-specific criteria, see Table 3). The LLM completed the analysis in approximately 216 seconds.
Figure 3 summarizes the fulfillment rates for each usability criterion across all evaluated platforms. The results show that general interface design aspects, such as clear layout (G1), recognizable interactive elements (G2), appropriate information density (G3), and consistent visual styling (G4), are wellsupported on most platforms. This suggests that these systems follow established design conventions and ofer a solid baseline for usability. This outcome is expected, given that the evaluated platforms are widely used and likely to have invested greatly in interface design and user experience.
In contrast, the recommender-specific criteria show a more diverse picture. Particularly, the presence of explanations (R3) and interactive feedback options (R4) was less frequently fulfilled. The same holds for transparency of underlying algorithms (R2/P2), explicit feedback options (P1), and flexibility of interaction (P3). This suggests that while basic UI design is generally strong, many platforms lack mechanisms to help users understand and influence recommendation behavior. These findings point to a gap in explainability and user control in many “black-box ” recommender settings. 4To avoid potential legal issues with publishing proprietary platform screenshots, we redrew the relevant parts as simplified UI sketches for presentation in this paper. Importantly, all analyses were conducted using the original screenshots. The sketches only serve as substitutes for publication and preserve the necessary information for understanding the reported aspects. This procedure was applied to all examples shown. For the original screen design, we refer to the respective platform web pages (see Table 1). illustrates example results related to criterion P1, which concerns the ability for users to explicitly express preferences. In the playlist creation scenario on Spotify, the LLM judged this criterion as unfulfilled, since users can only add recommended songs without any direct feedback mechanism, such as liking or disliking. In contrast, Amazon was evaluated as fulfilling the criterion. Users could actively apply clickable filters to indicate item type preferences.
The explanation provided for Spotify is particularly nuanced and insightful. While the LLM acknowledges that adding songs to a playlist can be seen as a form of preference expression, it argues that this action alone may not suficiently satisfy the criterion. This reasoning is persuasive as users might skip a recommendation for various reasons, such as disliking the artist or simply finding the song unsuitable for the current playlist, which is not explicitly communicated to the system. The suggested improvement, to provide more fine-grained feedback options, is both reasonable and actionable.
In the recommendation presentation scenario, several criteria were considered unfulfilled. Figure 7 presents example results for criterion R4, which concerns the ability of users to interact with recommended items. For Google News, the LLM noted the absence of visible interaction options and suggested improvements. In contrast, YouTube was evaluated more positively, as its recommendation cards include an accessible interaction menu (“three-dot” menu).
These examples highlight both the usefulness and limitations of our approach. While the LLM was able to identify relevant usability gaps, it also operated solely on static screenshots. As a result, interactive elements that appear only on hover or during user interaction, such as the hidden menu in Google News, may remain unrecognized. Nevertheless, the LLM’s suggestion remains valid, as for a new or inexperienced user, the lack of visible afordances can be a barrier to efective interaction and may justify more prominent cues. (b) Fulfilled P1 criterion (Amazon)
(b) Fulfilled R4 criterion (YouTube).
Another notable observation is that the LLM judged none of the platforms as fulfilling criterion R3, which concerns providing explanations for why items are recommended. However, a closer review of the platforms and LLM explanations reveals a more nuanced picture. While many platforms indeed lack explicit explanations, some ofer at least high-level contextual hints. For instance, Spotify displays recommended playlists with labels such as “Brand new music from artists you love”, which is a highlevel explanation for the recommendation. This suggests that the criterion could benefit from further refinement to distinguish between vague contextual hints and explicit, personalized explanations.
Our results suggest that LLM-based usability analysis can provide useful, low-efort insights into the strengths and weaknesses of recommender interfaces. The generated explanations and improvement suggestions were generally accurate, understandable, and relevant, which indicates the potential for such tools to support development, especially in early-stage prototyping and iterative design. While LLMs cannot replace expert evaluations, they could act as assistive tools in a human-in-the-loop setting, reducing manual efort and accelerating the identification of usability issues.
Building on these findings, several research challenges emerge: Prioritization of Issues. The number of identified usability issues of a recommender system can be large and trigger significant efort for developers to evaluate them and prioritize their fixes. The currently used binary fulfillment decisions do not capture the severity of issues, which limits the possibilities for ranking. Using more nuanced assessments using severity ratings could support issue prioritization and make the analysis more actionable by highlighting the most critical usability gaps. Prompting Design and Context. Prompt design can influence the analysis results. Our current template evaluates multiple criteria simultaneously (see Figure 2), which reduces inference costs but sometimes may overlook issues. Using separate prompts for each criterion could improve efectiveness.
Another opportunity for improvement is the refinement of the context description and clarifying
the intended meaning of each criterion. In some cases, the LLM struggled to interpret the criteria
consistently, for example, determining what level of detail qualifies as an explanation (R3). Making this
clearer by adding more detailed descriptions or examples could lead to more accurate and robust results.
Dynamic UI Behavior. A boundary of our current approach is the reliance on static screenshots,
which prevents the model from recognizing dynamic interface elements that appear on mouse hover 5 or
click. Providing a video of recorded interactions as context could show the complete interface behavior
and overcome this limitation. Beyond that, LLM-based agents that directly interact with interfaces
could be an even more elaborate approach to also automate the data collection aspect.
Validation Against Expert Assessments. Systematic comparison with expert evaluations is needed
to assess the reliability and practical value of LLM-based usability analysis. While early studies report
promising alignment with expert judgments on general usability criteria [
In this paper, we explored an LLM-based approach to usability analysis of recommender user interfaces. We applied the method to ten publicly available platforms to assess whether the identified issues were plausible, clearly explained, and accompanied by meaningful improvement suggestions. Our ifndings demonstrate the potential of multimodal LLMs to support low-efort usability evaluation of recommender interfaces, particularly during early-stage design. Also, the findings highlight diferent areas for improvement, particularly in handling dynamic interface elements and generating more nuanced, context-aware judgments. Future work will focus on improving prompt strategies and contextual understanding, and validating LLM-generated assessments against expert evaluations.
During the preparation of this work, the authors used ChatGPT and Grammarly in order to: Grammar and spelling check, Paraphrase and reword. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. 5see https://www.w3schools.com/howto/howto_css_display_element_hover.asp for an on mouse hover example. [17] A. Gunawardana, G. Shani, S. Yogev, Evaluating Recommender Systems, Springer US, New York, NY, 2022, pp. 547–601. URL: https://doi.org/10.1007/978-1-0716-2197-4_15. doi:10.1007/ 978-1-0716-2197-4_15. [18] G. Team, O. Authors, Gemini: A family of highly capable multimodal models, 2024. URL: https: //arxiv.org/abs/2312.11805.