Visual cues and attractiveness play a crucial role in everyday food choices [1]. Even when only presented with a food image, humans tend to instantly assess a food's energy density, expected taste and other characteristics [2]. As such, images are one of the key determinants of food preferences [2,3], tapping into emotional and hedonic processes of an individual [4].
The importance of visual attractiveness also applies to digital choice context, including food recommender systems [5]. Our previous research has shown the capability of recommender systems to influence food behaviors via visual features, including the promotion of either high-fat or low-fat food choices [6], as well as encouraging the search for healthier options [3]. Additionally, our earlier work has established that visual attractiveness significantly contributes to predicting the online popularity of food items [7], and these visual features can also be leveraged to infer cultural backgrounds [8].
What is currently missing is an in-depth examination of image feature modeling. Although previous studies have extracted image features and examined the relation between those features, visual attractiveness and user preferences [6,3], these models have not been optimized. Moreover, to date, image features have not been related to user characteristics (e.g., demographics, food knowledge), which are also important determinants of food preferences [9].
We present the results of a mixed-method study that explores the determinants of visual attractiveness in digital recipe images more comprehensively. Our approach builds upon previous work by modeling perceived visual attractiveness based on low-level image features [10,11,3]. Additionally, we seek to optimize this model by integrating user characteristics that have been employed in knowledge-based food recommender systems to promote healthier recipe choices [12,13,14].
Finally, we inquire more qualitatively on user justifications for provided visual attractiveness ratings, asking to motivate their quantitative judgment. We formulate the following research questions:
Compared to our extensive previous work in the field of visual attractiveness and food recommender systems [6,7,8,13], this study offers novel insights into several key aspects:
• Previous work mostly relied on low-level image attractiveness features, while this study shows how new deep-learning models compare to these old features. • This work, compared to any before, also shows as to what extent demographic features play a role in predicting visual food attractiveness. To our knowledge, no other work has shown this before. • Finally, this study tries to go beyond traditional quantitative black box approaches and reveals why images are rated less or more attractive.
To perform our study, we employed a dataset sourced from the well-known recipes website All-Recipes.com, with the addition of new recipe photos [3,14]. The dataset comprised various recipe features, including image URL, ingredients, amount of fats and sugar, and instructions and ingredients. To generate a diverse set of images, we randomly selected 200 recipes with relatively from the dataset of 58,000. As most images in this dataset were relatively unattractive [3], we used the recipe's title in search engines and image websites (e.g., Unsplash) to look for more attractive images for 100 of these recipes. To validate this process, three computational food researchers, including a co-author, voted on which of the two photos was the most attractive to ensure a diverse set of recipe images in terms of expected attractiveness.
The study involved a survey design, as depicted in Figure 1. Participants first provided demographic information, as well as responded to items that measured their subjective food knowledge (4 items) and cooking skills (6 items), using 5-point Likert scales based on earlier work [15,16,17]. We also used questions from earlier work on a knowledge-based food recommender [14], to inquire on other user characteristics, including recipe website usage and home cooking frequency, cooking experience and dietary goals. Afterwards, users were invited to rate the visual attractiveness of 12 semi-randomly selected recipe images, on 7-point attractiveness scales. In addition, to address [RQ3], they were asked to write at least one sentence about why they had given this rating. Finally, to support our examination of [RQ3], we used 5-point Likert scales on food image dimensions [18], to ask to what extent a recipe's appearance, expected taste, healthiness, and familiarity affected their attractiveness ratings.
We employed the Prolific crowdsourcing platform to recruit 192 users (65% male; 𝑀 𝑎𝑔𝑒 = 33.54) to participate in our study. The study took approximately 11 min to complete and participants were reimbursed with GBP 1.65 1 .
To address our research questions, we primarily employed linear regression models. This helped to understand the principal impacts of image attributes and user characteristics on image attractiveness, the latter derived from user ratings. For our thematic analysis, the images were split into attractive and unattractive based on the mid-point of the rating scale (4) (𝑀 = 4.33, 𝑆𝐷 = 1.80). Details of used materials and conducted analyses can be accessed through the following URL [19].
We first modeled perceived visual attractiveness based on the underlying image features. We extracted diverse low-level visual features using the OpenIMAJ Java Framework (cf. [7]). Subsequently, we conducted a linear regression analysis to predict attractiveness based on these extracted visual features.
The results are outlined in Table (1.A), revealing that several image features significantly affected the attractiveness of a recipe image: 𝐹 (8, 2100) = 32.66, 𝑝 < 0.001. Specifically, Colourfulness, Brightness, Naturalness, and Entropy demonstrated a positive association with image attractiveness. In contrast, Saturation, Sharpness, and RgbContrast negatively affected image attractiveness. In line with [3], these results suggested that users perceived colorful, bright, and naturalistic food images as more attractive. 1 Our study complied with the ethical guidelines of the Research Council of Norway and the guidelines of University of Bergen for scientific research. It was judged to pass without further extensive review.
Going beyond low-level visual image features, we used deep learning architecture models. Our toolkit included established models, such as VGG16 [20] and ResNet [21], along with the well-known transformer [22] architecture for visual feature extraction, CLIP2 [23]. Table (1.B) outlines the performance of these different models, outperforming our regression model in terms of 𝑅 2 and RMSE. This aligns with previous research where deep learning embeddings also outperformed low-level visual features within the context of food [7,24].
We further examined whether user factors affected the perceived visual attractiveness of images. Accordingly, we divided user characteristics into different categories: User demographics, User profile, which represented the backbone of a food knowledge-based recommender system, and User knowledge, which measures the user's food knowledge and cooking skills. A confirmatory factor analysis, reported in Table 2, showed that both subjective food knowledge and cooking skills adhered to internal consistency guidelines (𝛼 > 0.70) while they also met the guidelines for convergent validity (𝐴𝑉 𝐸 > 0.5).
Results of the principal component factor analysis across different subjective food knowledge and cooking skills. Items were measured on 5-point Likert scales. Cronbach's Alpha is denoted by 𝛼, 𝐴𝑉 𝐸 is the average variance explained. Items in grey and without loading were omitted.
Subjective Food Knowledge 𝛼 = 0.866 𝐴𝑉 𝐸 = 0.858
Compared with an average person, I know a lot about healthy eating.
A) presents the outcomes of the linear regression model aimed at forecasting the attractiveness of image recipes: 𝐹 (9, 2090) = 3.60. Among the various user factors examined, only two significantly affected recipe attractiveness: cooking skills (𝛽 = 0.34, p-value= 0.00021) and recipe website usage (𝛽 = 0.18, p-value= 0.020). However, none of the other user aspects affected user ratings for a given image recipe. Additionally, we also analyzed a combined model of image features and user factors, but this lead to results similar to the separate models reported in Tables (1 and 3.A). This suggested that low-level visual features had a more significant impact on food image attractiveness than user features, largely in line with preliminary findings in previous research [3,18].
To assess the influence of different food image dimensions on user ratings for food images, we modeled visual attractiveness based on the reported importance of food image dimensions. Table 3 outlines the results of the regression model: 𝐹 (4, 21) = 2.41.
Two factors significantly impacted attractiveness. First, appearance significantly impacted user ratings (𝛽 = 0.12, 𝑝 = 0.03). Second, the expected healthiness from the images also demonstrated a significant impact (𝛽 = 0.07, 𝑝 = 0.03). However, perceived taste and familiarity did not impact user ratings.
To gain insights into the reasons behind the visual attractiveness ratings, we collected qualitative justifications from users. Natural Language Processing (NLP) techniques, such as punctuation removal, repeated character elimination, and stop word filtering, were applied to analyze 2,019 user justifications for both attractive and unattractive images. From these responses, we generated two word clouds to highlight the most prominent terms.
Figure 4 shows a few examples of user textual justifications. Several participants, including user (U 𝑎 ), expressed the term 'crispy' in their assessments of attractive images, mainly referring to appearance. The word 'simple' is frequently used by users, such as user (U 𝑏 ), to convey the simplicity of recipe content. In contrast, 'mess' was more commonly associated with judgments of unattractive food images, indicating their unappealing appearance. Moreover, the repeated use of the term 'fat' suggested that fatty foods were generally perceived as unattractive, as in judgments by users (U𝑐 − 𝑑).
Judgments related to health frequently appeared in connection with the food's appearance, such as by user (U 𝑒 ) in Figure ??. The term 'restaurant' was employed in various user judgments, often associated with presentation and healthiness, as described by the user (U 𝑓 ). Conversely, the concept of unhealthiness was linked to fatty foods and messy representation, as evident in the judgments of users (U 𝑔−ℎ ) in Figure ??.
This work has explored different aspects of the relationship between the user and food images. Through an online user study, we have found that various visual features can predict the attractiveness of a given image (i.e. colorfulness, brightness, naturalness). This prediction accuracy could be slightly improved using image features extracted using deep learning techniques (RQ1). In line with earlier work [11,3,18], this suggests that the visual attractiveness of food images can be enhanced by increasing their colorfulness, brightness, and naturalness, while decreasing other features, such as saturating and sharpness. Obviously, there may be tradeoffs between these features when altering them.
Regarding user characteristics, none of the user demographics are related to food image attractiveness. In contrast, using online recipe websites and cooking skills are positively associated with the attractiveness of food images (RQ2). More novel is our contribution on the user justifications, for which we have found image appearance and perceived healthiness to be important dimensions of visual attractiveness ratings (RQ3). It seems that attractiveness are related to the expect taste or hedonic food goals (e.g., 'crispy'), while unattractive images focused on poor presentation and disliked ingredients.
Our study offers valuable insights into techniques for image attractiveness selection for various goals and domains. In particular, these techniques can be leveraged to persuade or nudge users towards specific eating goals, such as health [3,25]. We believe that leveraging the visual appeal of attractive images can address this issue. Our future studies will focus on designing image selection pipelines for the application of food recommender systems tailored to guide people toward healthy food choices without compromising the benefits of personalization. We aim to analyze and categorize the collected textual judgment through thematic analysis to build word dictionaries related to image dimensions. These dictionaries can then be used to train learning models, enabling the evaluation of food image attractiveness based on user textual inputs.
This work was supported by industry partners and the Research Council of Norway with funding to MediaFutures: Research Centre for Responsible Media Technology and Innovation, through the centers for Research-based Innovation scheme, project number 309339.
The authors acknowledge the use of ChatGPT [26] for checking and correcting the grammar of this article. No new content was generated this way; only existing text was checked and, if needed, corrected.