AI-powered technologies are becoming pervasive, and ofer endless possibilities for novel consumer applications. This paper describes the comprehensive process of designing, implementing, and evaluating SnapChef, an innovative recipe recommendation application. The results of the System Usability Scale (SUS) study conducted over a software prototype using the Android platform yielded overwhelmingly positive user feedback, indicating a high level of satisfaction with the application's usability. Despite noted imperfections in image recognition accuracy, we anticipate significant enhancements in this aspect as AI technology continues to evolve and mature over time. These findings underscore the potential for continual improvement and optimization of our application to meet user needs and expectations in the dynamic landscape of AI-driven technologies.
Food waste is a major global issue [
Over the last decade we have witnessed a boom of artificial intelligence (AI) technologies. Computer
vision has advanced significantly leveraged by deep learning approaches, in particular convolutional
neural networks [
In this work, we leverage recent trends in AI technologies to address the issue of food wastage that occurs as a result of consumer behavior. We introduce SnapChef, a mobile application that allows users to photograph food ingredients in their kitchen cupboard or fridge and receive a list of recipe recommendations. This process involves two steps: first, computer vision techniques identify the set of food ingredients; second, a Large Language Model generates a list of recipe suggestions based on the identified ingredients. The main contribution of this work is the integrates both image recognition and LLM technologies. Unlike most previous works, as discussed in Section 2, we combine food image recognition with recipe recommendations.
The structure of this paper is as follows. Section 2 presents related work. The design of the mobile application is outlined in Section 3. Subsequently, we describe the implementation (Section 4). We evaluate the application by means of a usability study describe in Section 5. Discussions are presented in Section 6, and Section 7 concludes.
others cases similar approaches but using servers based on Raspberry Pi [
Generative AI technology has only been accessible in recent years, which explains why it was not a
trend for it to address this issue in the past. However, the use of image recognition for food has proven
to be very efective. This is exemplified by the application of a convolutional neural network, that
was used to determine how much had been eaten from a plate with an accuracy greater than 98% [
In relation to the issue of food waste and its integration with smart refrigerators, this work can be diferentiated from other similar approaches by implementing novel two diferent types of AI tools.
To tackle the issue of consumer-driven food wastage, aimed at facilitating user interaction, we introduce a mobile application serving as a visual interface. The primary functions of this software entail food detection and recipe generation. To ensure a modular design, these tasks are executed through the utilization of APIs. This section delineates the proposed architectural framework.
Figure 1 presents the architecture of the application, which is composed of the following components: • User: Represents the entity that interacts with the application. • Interface: Contains the application logic. • Image Recognition API: Specifically designed to identify food items in images.
• Large Language Model (LLM) API: Provides advanced natural language processing capabilities.
The flow of control proceeds as follows: Initially, the user chooses an image through the application interface, either by capturing a photograph using the mobile device or selecting a file from the filesystem. Subsequently, the image is forwarded to the initial API for image recognition, which furnishes a string containing the identified ingredients. Upon receipt of this information, a request is dispatched to the LLM API. This module yields a compilation of recommended recipes, which is then presented to the user.
Figure 2 presents the wireframes for the proposed application. Figure 2a shows the home screen where a canvas and three buttons are displayed. The camera button which enables a photograph to be taken using the mobile device. The gallery button enables an image to be selected from the gallery. The canvas is initially blank; however, upon selection of an image from either the camera or the file system, it is displayed within the canvas. The continue button permits user navigation to the subsequent application screen.
Figure 2b shows the second screen of the application, which consists of a modal window that indicates the previously found ingredients. Finally, Figure 2c shows the generated recipes. These are displayed in the form of cards, and on each card we can find a diferent recipe. By clicking on a card, the instructions to follow for that particular recipe are displayed.
In this section, we firstly justify the rationale behind the selection of the technologies used (see Section 4.1). We then describe the implementation of the food image recognition phase in Section 4.2 and the recipe generation phase in Section 4.3. Finally, we show an example run of the application in Section 4.4.
Framework The Flutter1 open source framework developed by Google was selected for the development of the application. While for the purposes of our prototype we produce code for the Android platform, we selected Flutter as it ofers numerous advantages for mobile application development, including the ability to create cross-platform apps with a single codebase for both Android and iOS, which streamlines development and maintenance. Its hot reload feature speeds up the development process, while its compilation to native ARM code ensures high performance. Flutter provides a rich set of customizable widgets for consistent UI design across platforms and is supported by a strong community and Google, ofering extensive resources and plugins. Additionally, it allows access to native features and ensures consistent behavior across platforms, making it a robust and eficient framework for modern mobile app development.
Food Image Recognition Technology In the early stages of project implementation, image recognition models specifically for food recognition were studied. Logmeal 2, a company that specializes in “artificial intelligence for food recognition and nutritional analysis”, has pre-trained image recognition models for many diferent types of food. The Logmeal API 3 was tested and found to correctly identify diferent types of food. This API sends images in JPEG format, with a size not exceeding 10.48MB. It then returns a JSON type object containing the results of the found elements. This is then used to return food found within the body of the object.
Recipe Generation Technology We also examined Large Language Models capable of assembling recipes based on previously identified ingredients provided by prompts. ChatGPT was one of the candidates considered for this task and was deemed to be the best choice due to its ease of integration and simplicity in returning responses since it can be modeled as a JSON object instead of a simple text string. Choosing ChatGPT over other existing large language models ofers the advantage of advanced conversational capabilities and contextual understanding, thanks to its extensive training on diverse datasets. Additionally, it benefits from continuous updates and improvements by OpenAI, ensuring state-of-the-art performance and reliability in generating human-like text.
As shown previously in Figure 2a, the initial home screen provides the user with three buttons for using the camera or selecting an image from the file system. After pressing the Continue button, the image is selected and verified that it is no larger than the originally specified dimensions. If the image is too large, a function is executed to reduce the size of the image. Then an HTTP REST request of type POST is made to the Logmeal API. The parameters required and used by this request are as follows: URL: https://api.logmeal.es/v2/image/segmentation/complete/v1.0 Parameters: language = spa Authorization Bearer Token Token = <Token of the LogMeal application> Body: FormData Image= <file of the image to be sent by the application>
This Logmeal API returns a body in JSON format with the identified ingredients within its elements. If the prediction is deemed to be appropiate by the user, the application will navigate to the second screen where the ingredients are shown within a modal dialogue box. If the request generated an error, an appropriate message is displayed. The sequence of events is shown in the sequence diagram in Figure 3.
Once in the second screen, the second API that calls ChatGPT will be automatically triggered. Recall the a JSON object is obtained from ChatGPT. This is accomplished by sending two prompts. The first prompt provides contextual information at the system level. In particular, the requirement is to respond in JSON format only, making it easier to extract the response once it is received. The second prompt is a request to return a list of recipes to be displayed to the user based on previously found ingredients, and write descriptions for the fields that must contain that information when printed. Figure 4 presents the prompts used by the application. 2https://logmeal.com/ 3https://api.logmeal.es/docs/
As with the Logmeal API, a Bearer token is sent to authorize the sending of the requests. Additionally, a JSON document with the following fields will be sent in the body of the request:
URL: https://api.openai.com/v1/chat/completions model= gpt-3.5-turbo messages= <List of key-value pairs as follows> role= <system, user> content= <prompts indicated previously, one for the system that tells ChatGPT how to return the information, and the role of the user that is the prompt where the ingredients found above are indicated, and the structure of the response is specified and what value each field should have.>
Usability testing plays a pivotal role in the development and refinement of products and systems, ensuring they meet the needs and expectations of users. By systematically evaluating the user experience, usability testing helps identify and address usability issues early in the design process, ultimately enhancing the product’s efectiveness, eficiency, and user satisfaction. Through direct observation and feedback from representative users, usability testing provides valuable insights into how users interact with the interface, highlighting areas for improvement and informing iterative design iterations. Moreover, by uncovering usability barriers and usability bottlenecks, usability testing mitigates the risk of costly redesigns and product failures post-launch, contributing to overall product success and market competitiveness.
(a) Home screen of the app.
(b) Modal showing the ingredients found.
(c) Recipes suggested according to those ingredients.
Experiment Setup On May 6, 2024, a usability assessment was conducted involving 25 participants enrolled in the Data Structures undergraduate course at Jorge Tadeo Lozano University (see Figure 7). The evaluation entailed individualized interactions with an image sourced on the Internet by each participant featuring various ingredients. This was done so that each participant would select a diferent image. Participants utilized the application’s photo-capture feature to process the image and subsequently reviewed the returned recipe recommendations. Following this task, participants completed a usability survey adapted from the System Usability Scale (SUS) questionnaire4, containing Likert scale responses ranging from 1 (strongly disagree) to 5 (strongly agree). Responses were collected using Microsoft Forms. Table 2 presents the questions given to each participant, and the mean score awarded to each question. 4https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website/ I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
Mean Score 4.30 2.00 4.38 1.23 4.42 2.09 4.90 1.47 4.76 1.19
Response Validation Following the collection of participant responses, a data cleaning or quality control procedure was implemented to ensure response consistency and identify potentially random responses. The System Usability Scale (SUS) questionnaire facilitated this process, as each of its 10 questions is presented both positively and negatively, resulting in 5 generated questions. During the validation process, responses to positively and negatively framed questions were compared, with a permissible diference value set at one. Responses showing a discrepancy of more than one were deemed invalid. For instance, it would be incongruous for a participant to evaluate the statement “I found the system unnecessarily complex” similarly to “I thought the system was easy to use.” Consequently, out of the initial 26 responses provided by participants, 5 were deemed inconsistent and omitted from analysis, leaving 21 valid responses for evaluation.
Results obtained Figure 8 presents the results obtained for questions 1–6. Figure 8a presents the results to the statement “I think that I would like to use this system frequently”. The majority of responses to this question are 4 (agree) or 5 (strongly agree). Only two participants respond with a 3 (neither agree nor disagree). The mean score is 4.30, indicating that overall most participants strongly agree with the statement.
Figure 8b shows the results for the statement “I found the system unnecessarily complex”, where the majority of the responses disagree. We observe a similar trend in Figure 8c, with the statement “I thought the system was easy to use”. In both cases, we can identify that few people consider the application to be dificult or complex only four people have this opinion.
Figure 8d shows that, according to the participants, the various functions of this system were well integrated. In contrast, Figure 8e reveals that most of the responses to the statement “I thought there was too much inconsistency in this system.” are positive, although at this point there is a slight increase in disagreement compared to the other statements. The reason for this is due to some particular cases where the application had small failures in identifying ingredients. However, the average results for this statement show a mean score of 2.09. Although not negative, improvements can always be made to increase this score, as mentioned in the 6 section.
Figure 9 presents the results obtained for questions 7–10. Figure 9a we find the most positive statement of those asked above: “I think that I would like to use this system frequently”. Only two votes separate this one from achieving a score of 5, with an mean score of 4.90. It is encouraging to know that the participants believe that other people would learn to use the system quickly, which relates to the statement in Figure 9c where most users felt confident in using the application. Finally, we can deduce from the statements in Figures 9b and 9d respectively, most users do not think that the system is dificult to use nor that much expertise is needed to manipulate it. The last statement is the highest rated statement of disagreement with an average rating of 1.19.
The total usability score computed for the system was 87.14 points (out of a total 100 points), which indicates that the system is excellent in terms of usability. However, there are some areas for improvement, and in addition to ten usability questions, a free-text suggestion or comment box was provided. In this space some positive comments and suggestions for improvement were received, such as that the interface should be improved. It was also commented that image recognition does not always work correctly. Some of the comments received included: • The application is very clear and recognizes the various ingredients. It presents a clearer vision when cooking and learning new gastronomy recipes. • It needs to slightly improve the ingredient recognition functionality • It has flaws like any other AI, like hallucinations • It is necessary to improve the identification system • I think the accuracy of scanning or processing the images should be improved.
We note that, according to the user comments, the flaws in the application primarily stem from external dependencies, such as the accuracy of data sources and the reliability of third-party APIs. As AI technologies mature and become more sophisticated, performance issues related to these dependencies are expected to diminish, resulting in a more robust and reliable application.
As such, the application demonstrated, overall, a favorable reception among users. However, several areas for improvement have been identified, particularly the visual interface, which could be enhanced to be more intuitive and visually appealing. Additionally, incorporating new functionalities could significantly improve the user experience. Another crucial aspect for improvement is the accuracy of food predictions. Although the identification of most ingredients was satisfactory, integrating a newer and more advanced API could further enhance the quality of the recipes provided. The ChatGPT API efectively fulfilled its role in delivering recipes based on specific ingredients. While the accuracy of some ingredient predictions was occasionally suboptimal, none of the experiments revealed errors that adversely afected the user experience, indicating that this component may not require immediate adjustments.
Recent advances are driving the proliferation of AI-powered applications across various industries. This rapid development is set to make AI an integral part of everyday technology, enhancing eficiency (a) “I think that I would like to use this system frequently”
(b) “I found the system unnecessarily complex” (c) “I thought the system was easy to use.” (d) “I think that I would need the support of a technical
person to be able to use this system” (e) “I found the various functions in this system were well(f) “I thought there was too much inconsistency in this integrated” system.” and innovation in numerous fields. AI has the potential to significantly reduce food wastage in the food industry by optimizing supply chain management, improving inventory tracking, and predicting demand more accurately. Additionally, AI-driven applications can help consumers by suggesting recipes based on available ingredients, thereby minimizing household food waste. SnapChef underwent usability testing with a diverse group of users and received positive feedback for its intuitive interface and accurate recipe suggestions. The application’s ability to enhance meal planning and reduce food waste was particularly well-received, highlighting its practical benefits for everyday use.
As future work, SnapChef could be significantly enhanced by integrating IoT sensor technology to enable real-time inventory tracking and suggest recipes based on available ingredients. Additionally, taking dietary preferences or restrictions into consideration would personalize the user experience, ensuring that recommended recipes align with individual needs. Moreover, implementing a feature to generate a shopping list of missing ingredients would streamline the cooking process and minimize waste. Lastly, using multi-modal generative AI to generate on-the-fly video tutorials with the recipe instructions would enhance user engagement and facilitate easier comprehension of cooking techniques. (a) “I would imagine that most people would learn to use (b) “I found the system very cumbersome to use” this system very quickly” (c) “I felt very confident using the system” (d) “I needed to learn a lot of things before I could get going with this system”