Proper nutrition is essential for the health and well-being of older adults, particularly those with cognitive impairments. This population is especially susceptible to ”nutritional frailty,” a condition that compromises the nutritional status of elderly individuals and can lead to serious health issues such as malnutrition. Technological aids can be used to promote independent eating habits among elderly individuals, in nursing homes or at home, to address their needs during mealtimes. Socially Assistive Robotics (SAR) has emerged as a promising approach, with applications in social companionship, supporting independent living, and engaging older adults in cognitively and physically stimulating activities. In this paper, we present a prototype of a Social Robot behaving as a companion for older adults to provide assistance, explanations, stimulation, and social interaction during meals. The goal is to enhance self-care and food awareness. To this aim we endowed the robot with three modules: (i) recognition of eating and drinking activities, (ii) detection and recognition of food during meals, (iii) the use of an explainable ontology-based foodAI service for providing personalized nutrition advice and explanations.
Malnutrition occurs when an individual lacks the necessary nutrients to function properly, often due
to an imbalance of protein, calories, and essential vitamins. This condition has particularly severe
consequences for older adults, increasing their risk of falls, recovery times, hospitalizations, and
even death. Several factors contribute to malnutrition in this population, including loss of appetite,
dificulties with chewing and swallowing, and increased use of prescription medications [
In this realm, it is crucial to explore technological aids that promote independent eating habits among
elderly individuals in nursing homes or at home, addressing their nutritional needs during mealtimes,
while providing a positive social experience. Recently, Socially Assistive Robotics (SAR) has been
proposed as an efective technology to assist elderly people in several contexts, such as providing social
companionship [
In the context of the SISTER (SocIal robotS to support biopsychosocial frailTy of sEnioRs at home for promotion of active aging) project, we propose using a human-like social robot, specifically the Alpha
CEUR
ceur-ws.org
Mini robot, to provide cognitive assistance, stimulation and social interaction during meal-eating. We designed and implemented three modules to empower the capabilities of the social robot: i) recognition of the specific activities of eating and drinking, ii) recognition of the food during eating and drinking, iii) the use of an explainable ontology-based food-AI service for providing personalized nutrition advice and explanations.
The first two modules ofer recognition capabilities learned through Deep learning models, which
are trained on specific datasets. The first one implements human activity recognition in a living
environment and works extracting keypoints of the body joint starting from the video-camera frames
[
The second module provides computer vision and dialog capabilities to enable the robot to detect and recognize food, establishing engaging and motivating one-on-one interactions. The food recognition model relies on the YOLOv8 technology [11] and it is trained on images of (mostly Apulian) food, organized by categories. We emphasise that this model is a specialization tuned for apulian food, given the numerosity of models already available for food in general. The third module, NutriWell, leverages AI and the GraphBRAIN [12, 13] technology for intelligent knowledge retrieval in nutrition and health management. NutriWell informs users about meal suitability based on their nutritional requirements, utilizing explanations that combine feature data and user preferences. This is done using an API that retrieves graph-based information integrated with an ontology specifying relational constraints. The ontology design, derived from existing frameworks and enhanced to integrate food impacts on disorders, allows for the calculation of meal impact scores tailored to user needs and preferences [14]. These modules have been integrated into the FoodCoach application that allows the social robot to monitor the meal-time of a user, collect data on the categories of food consumed, monitor a user’s meal habits, and engage him in eating and drinking through a workflow-based dialog. As a robotic platform, we use Alpha Mini situated on the table facing the adult. The dialog uses three levels of information. First, speech based on cues, encouragement, positive statements, personal communication (e.g., calling by name) aiming to stimulate the consumption of food, and non-verbal social cues in which the robot uses gestures, the shape of the eyes, and images displayed on its eyes. Nutriwell API is then used to provide food-related information and explanations or to answer specific questions on a specific meal.
The robot receives the user’s responses over the dialog and acquires images of food and activities in real time. While it can carry on verbal communication and social interaction autonomously, the image data are transmitted to a computational unit on which the algorithms of human Activity Recognition (AR) and Food Detection and Recognition (FDR) operate. Nutriwell is used for evaluating food ingredients, calories and other food-related information, appropriateness of a meal for the user, taking into account his intolerances, diseases, preferences (e.g. vegetarianism), and so on. The robot and computational unit are connected locally according to the EDGE computing approach. Although this may limit the scenarios in which the robot may work, it avoids many critical cyber-security issues, such as data leakage or data privacy violations [15].
In the following sections, we illustrate in more detail how the Food Coach works.
To recognize meal-related actions (eating and drinking) during the times of the interaction between the older adult and the robot, we based our approach on the one proposed in [16] which is shown to be eficient in real-time. Additionally, it has been extended to identify hand-held objects involved in the eating or drinking action (Table 1).
The approach relies on supervised solutions of Graph-convolutional networks which straightforwardly model the structure of the human skeleton by representing the joints as the nodes of a graph and the bones as the edges. To train the model, we use the dataset NTU RGB+D [17] which includes the data of 60 daily activities. It has been originally arranged to conduct training and validation sessions, indeed it has 37,920 activity videos for training and 18,960 for validation. However, after preliminary experiments resulting in low performances, we have performed a step of feature augmentation on these videos, by adding the data on the human skeleton joints. To extract this kind of information we implement a two-step procedure, that is, identification of the human figures and extraction of the joints. The first step uses the technology YOLOv5, while for the second one, we apply the toolbox MMDetection [18] to the dataset AVA 1 and considered 17 types of skeleton joints. To reduce computation time and cluttering in the detected objects, each extracted frame was cropped according to the detected skeleton box, with slight padding, to include objects kept or used away from the body.
For the training phase, following the steps described in [16], we trained four diferent models on four diferent feature sets: i) Joint, the 3D coordinates of the extracted joints; ii) Joint motion, the diference between the coordinates of the joint at time + 1 and the coordinates of the joint at time ; iii) Bone, the diference between the coordinates of linked joints based on the structure of the human skeleton extracted; iv) Bone motion: the diference between the bone at time + 1 and the bone at time . Moreover, we used as an ensemble of the four models by calculating the weighted average of the outputs. The results on the validation set are shown in Table ??.
Even if the ensemble of the models has the highest accuracy, we decided to use the model trained on the joint features. This choice represented a trade-of since it allows a faster computation in real-time with slightly lower performance than the ones achieved by the ensemble of the models. However, when analyzing the performance of the model in recognizing the eating and drinking activities on the validation set of the considered NTU RGB+D dataset, we noticed that some ambiguities depended on the similarity of the body pose and movements of some activities that could be easily disambiguated by considering the object involved in the action [19, 20].
Therefore, to allow the robot to better recognize predefined actions, we extended the model with object detection. For this purpose we used the YOLO v5x6 model and picked the highest confidence detected object. Regarding object filtering, both to ignore irrelevant objects and solve the issue of background objects, which are uninvolved in the analyzed actions, we grouped object types into categories, which also allows us to use relevant data to surpass ambiguity
To complete data preparation, a centroid corresponding to the center of the body was calculated by triangulating the coordinates of the shoulder and hip joints. This also allows to simulate the zaxis, absent from the selected parameters. Finally, to provide a common format of the data passed to the tensors, a subtraction between the centroid and each joint was performed. To demonstrate our hypothesis, two models have been trained and tested. The base model uses two channels, corresponding to the x and y skeleton joint coordinates. The extended model uses four channels, corresponding to the x and y skeleton joint coordinates, the distances from the object center and each skeleton joint, and the used object class. The dataset containing 856 instances for each class (A1 and A2) was split for training and testing. The results of the test show that there was an improvement of accuracy of 3% on average, showing that adding hand-held object detection and identification to activity recognition allows a better understanding of what the user is doing and in particular to correctly recognize eating and drinking. In 1https://github.com/MVIG-SJTU/AlphAction/blob/master/DATA.md particular, the eating activity accuracy went from 0.92 to 0.94 and the drinking one from 0.93 to 0.97.
Food recognition is based on object detection and recognition using Deep Learning techniques. In this work, since we are going to test the application with users of our region, we created and trained the food recognition model on Apulian food and, in particular, food usually eaten by seniors. To create the dataset, the Web Scraping technique has been used to collect the images. Web scraping is a technique used to collect data from Web sites, including images. In this way we automated the image collection process, significantly reducing the time and manual work required. It also made it possible to extract a large number of images eficiently and systematically. To clean the collected set of images a manual sorting task was performed to select only those images relevant to the purpose. Table 3 shows some of the considered food categories:
In total, about 2,300 images were downloaded, selected and assigned to the 31 categories. Then, using Roboflow [ 21], a web platform dedicated to managing, organizing, and preparing image data for training machine learning models, collected images were stored and organized, simplifying data management and making the model training process more eficient. The images were therefore divided into training (70%), validation (20%) and test (10%) sets. This division ensured an adequate level of generalization of the model, avoiding the risk of overfitting. To prepare the images for training the machine learning model, each image was labeled with relevant information about the objects of interest within the image using Roboflow’s annotation feature.
Since in some images there was more than one instance of food, at the end a total of objects were labeled corresponding to the diferent categories of food in the dataset. Data augmentation was performed after the Auto-Adjust Contrast as a preprocessing step. As for the augmentation, the following techniques have been used: - Random Shear : Applying random shear values to the image. - Random Rotation: Applying random rotation angles to the image. - Vertical Flip: Flipping the image vertically.
After the data augmentation process, the training set contained 14,757 examples. The training of the model has been performed using Ultralytics YOLOv8, a cutting-edge, state-of-the-art model that builds upon the success of previous YOLO versions and introduces new features and improvements to further boost performance and flexibility [ 11]. As a performance measure, we used the mAP (mean Average Precision) which is commonly used as an evaluation metric in object detection tasks. In particular, the mAP@50 was calculated, which refers to the mean Average Precision at a specific IoU (Intersection over Union) threshold of 0.5. IoU is a measure of overlap between the predicted bounding boxes and the ground truth bounding boxes of the objects. Our model reached a mAP@50 of 66.5%.
Nutriwell API has been developed to evaluate how appropriate is a meal for an end-user, taking into account user intolerances, diseases, preferences (e.g. vegetarianism), food ingredients, calories and more. The dataset is constructed by combining information available in some of the most comprehensive food websites, such as GialloZaferano and AlimentiNUTrizione. Taking everything into account, many works aimed to recommend personalized food based on diseases/allergies, some of them are not explainable or limited in their interpretation since they do not make use of interpretable data structures or ontologies. The conceptualization of food barely takes into account intolerances and the impact of each ingredient on them. Datasets vary in nationality and there have been attempts to merge them but, as far as we are concerned, nobody took into account two of the most popular Italian websites.
The first step was to realize a new dataset, that is the result of the combination of existing datasets available in the form of websites. As far as we are concerned, it is the first time GialloZaferano and AlimentiNUTrizione have been employed for developing an AI-based diet assistant. The main limitation of this dataset is that it is in Italian but it may represent a relevant resource in the field for the community. One of the contributions consists of the collection of these web-based data (through scraping), translation, and dissemination. The selected diseases, allergies and intolerances have been manually selected considering their relation with food and manually listed from experts who examined the dataset. More information available here [22]. Afterwards, foods, beverages and ingredients have been provided with an impact on the diseases. The impact can be positive, neutral, or negative of the food/ingredient with the diseases. This assumption holds if the quantities respect more or less the traditional recipe. For instance, an exceeding amount of salt harms every person regardless of his/her physical state. The complete set of the ingredients contained in the recipes has been taken into account to label each ingredient with a category (e.g., fruit, meat, sauce). For this purpose, a set of categories has been defined based on the classes in the HeLiS ontology [ 23]. In particular, this ontology provides a representation of the food and physical activity domains and has been used in a real-world system for promoting healthy lifestyles in workplaces. The total classes selected are 35 and the resulting hierarchical structure is shown in Figure 1.
After defining the categories to use, the ingredients have been manually assigned to the most specific category. Labelling ingredients allows assigning diet and health labels to foods and beverages. More specifically, as in Edaman 2, it has been chosen to supplement foods and beverages with a set of diet and health labels to provide information on nutrient-level and ingredient-level aspects of the foods. In particular, 12 labels have been defined, 5 diet labels and 7 health labels. The combined dataset has been translated into a graph formalism (specifically Labelled Property Graph [ 24]) which can be accessed through an API. The graph is empowered with an ontology defining entity and relationship constraints. The ontology also acts as a scheme for the graph and has been designed starting from existing schemes and connecting information about foods with their impact on disorders that is, to the best of our knowledge, an underrepresented way of formalizing schemes in the food domain. The graph and the ontology follow the GraphBRAIN technology.
Resources exposed by the API have been identified based on the concepts represented as classes and relationships in the ontology. Specifically, the three concepts defined as top-level classes in the ontology, like Aliment, Ingredient and Disease, give rise to two separate types of resources: collection resources and singleton resources. The former represents groups of homogeneous items, whereas the latter represents specific items within a collection. For example, the Aliment concept results in a collection resource, including all the foods and beverages, and multiple singleton resources, one for each specific food or beverage. The same applies to the Ingredient and Disease concepts, where a collection groups together the items, which can also be accessed individually.
Relationships among the concepts are represented as nested sub-collection resources. In particular, the partOf relationship specifying the ingredients contained in a food is represented as a sub-collection nested within the corresponding singleton food resource. In the opposite direction, the set of foods containing an ingredient is modeled as a sub-collection within the singleton ingredient resource. Similarly, the impact scores of a specific food (ingredient) on diseases are nested within the singleton food (ingredient) resources and, in the opposite direction, the impacts of foods (ingredients) on a specific disease are nested within the singleton disease resource.
Each singleton resource is described with a set of attributes that match the attributes of the respective classes in the ontology. Furthermore, two additional attributes are included in each singleton resource: id (the unique numeric identifier of the item) and type (the most specific ontology class assigned to the item). More information about the ontology are available here [22].
The API also implements filtering features allowing clients to obtain subsets of large collection resources. In particular, one or more filters can be specified to retrieve a subset of items satisfying some constraints. For instance, the name filter parameter can be used to retrieve foods and beverages matching a string according to a similarity measure (namely, the Sørensen-Dice similarity measure).
The idea of supporting seniors with personalized recommendations while making the eating experience more enjoyable and convivial, is the focus of this paper. In [25] it is shown that most individuals involved in the study are comfortable with the idea of eating with a companion robot even if the study could not show that the robot had a significant impact on the food intake. To investigate then the role of the Social Robot in this context, as a first step, we develop the Food Coach application for the Alpha Mini robot.
Figure2 (left) shows the main capabilities of the system, which is made up of several modules. Some of them, due to computational complexity, are on a local Edge. The modules are: • Dialog Management: the dialog with the user is handled using a JSON specification of the workflow representing the dialog. The workflow includes conditions that trigger the robot’s behaviors. There is a Workflow for each of the robot’s communicative goals (i.e. greet, encourage, etc.). The dialog combines natural language and non-verbal communicative signals, such as the eyes and gestures (see figure 2 - right) • Face Detection and Person Recognition (PR): this functionality is already present in the robot SDK and will be used for detecting a person and recognizing him. This information is kept on a local database on-board the robot and it is stored according to EU GDPR rules. • User Profile : Recognition accesses the user profile. It stores personal (user’s picture, user’s age, gender, name, family, friends and physicians contacts) and medical features (chronic diseases, allergies, therapies, etc.) and preferences (food, music, TV programs, etc.). • Food Diary Manager : this module allows inserting eaten food, nutrition facts, and time of eating activity in a database present on the local Edge through dialog. • Food Coach Local Edge: allowing access to services for: – Activity Recognition (AR): a module for activity recognition described in Section 2.1 has been developed to recognize user activities in real-time and it has been fine-tuned on the recognition of eating-related activities. – Food Detection and Recognition (FDR): a module for food detection and recognition described in Section 3 has been developed to recognize what the user is eating in real-time. – NutriWell: to retrieve food ingredients, nutrients, health and diet labels starting from the name of the recognized food. It is used also to generate information and explanations about the food during the dialog. – Food Diary Database (FDD): the daily food diary is stored in a database developed for the application purpose.
The Mini Local Edge allows the robot to handle CV tasks and access the NutriWell API. Frames captured from the robot’s camera are sent continuously to the Edge and stored in a bufer. When the edge has collected a suficient amount of frames, CV modules are applied and the results are returned.
They aim at monitoring and motivating a user to eat the meal and drink while also promoting the social dimensions of eating. The user can ask for explanations about what the food he/she is eating and the robot uses known features to motivate the user. Eating-related information are stored in the food diary. Some dialog examples are shown in Table 4.
Scenario Mario is a retired 75-year-old man, widowed and lives alone. He loves sweets and his fondness for sweets has led to health issues in recent years. Indeed, he has been diagnosed with irritable bowel syndrome and high cholesterol. The Alpha Mini Food assists and engages Mario during mealtimes during which he asks for information and chat about several topics.
After the meal, Mario asks Alpha Mini whether he can eat a little box of chocolates as a dessert. Mini, using the Nutriwell API, will observe that chocolates hurt irritable bowel syndrome. Then the robot will also explain to Mario why he cannot eat it and which are the harmful ingredients in chocolates.
Then, he asks for suggestions on more appropriate desserts for his condition. Mini looks for desserts having a much lower negative impact on his health.
Using a friendly social robot to recognize users’ activities in a domestic context [26, 27] is not new and also food recognition tasks are developed in several application domains. However, the idea of supporting the elderly with Food Coach that acts as a companion while making the eating experience more enjoyable and convivial is a quite new topic to be investigated. Moreover, it is quite natural to imagine the social robot also as a recommender where score are introduced in the list of available meals, guided by user-based preferences. In this paper, we presented the prototype of an application that shows how a Social Robot, Alpha Mini, integrated with AI functionality for activity recognition, food recognition, and ontology-based reasoning can be used to assist older adults in enhancing their mealtime experience and, in the meantime, monitor and log their eating behavior by providing personalized recommendations with explanations. The approach is promising, but needs to be evaluated from two points of view: i) for accuracy in recognition when the CV tasks are made in the wild and ii) from the technology acceptance point of view, which needs to be tested with seniors during meal-time. This is part of our future work in which the Food Coach will be tested with twenty seniors living at home alone in the context of a multi component intervention for frail seniors. This work may give rise to several extensions, from recommendations about food recipes and alternatives to the evaluation of food remaining on the plate to prevent malnutrition.
This work was co-funded by the SISTER PRIN PNRR project, CUP: P2022K5WFC. [9] L. M. Dang, K. Min, H. Wang, M. J. Piran, C. H. Lee, H. Moon, Sensor-based and vision-based human activity recognition: A comprehensive survey, Pattern Recognition 108 (2020) 107561. [10] H. Ghayvat, M. Awais, S. Pandya, H. Ren, S. Akbarzadeh, S. Chandra Mukhopadhyay, C. Chen, P. Gope, A. Chouhan, W. Chen, Smart aging system: Uncovering the hidden wellness parameter for well-being monitoring and anomaly detection, Sensors 19 (2019). doi:10.3390/s19040766. [11] G. Jocher, A. Chaurasia, J. Qiu, Yolov8, YOLO by Ultralytics. https://github.com/ultralytics/ultralytics, 2023. Accessed: February 30, 2023, 2020. [12] S. Ferilli, D. Redavid, The graphbrain system for knowledge graph management and advanced fruition, in: Foundations of Intelligent Systems: 25th International Symposium, ISMIS 2020, Graz, Austria, September 23–25, 2020, Proceedings, Springer, 2020, pp. 308–317. [13] S. Ferilli, D. Redavid, D. Di Pierro, et al., Lpg-based ontologies as schemas for graph dbs., in: SEBD, 2022, pp. 256–267. [14] D. Di Pierro, S. Ferilli, An api for ontology-driven lpg graph db management, in: SEBD, 2023, pp.
303–316. [15] J. Marchang, A. Di Nuovo, Assistive multimodal robotic system (amrsys): Security and privacy issues, challenges, and possible solutions, Applied Sciences 12 (2022). doi:10.3390/app12042174. [16] Y. Chen, Z. Zhang, C. Yuan, B. Li, Y. Deng, W. Hu, Channel-wise topology refinement graph convolution for skeleton-based action recognition, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13359–13368. [17] A. Shahroudy, J. Liu, T.-T. Ng, G. Wang, Ntu rgb+d: A large scale dataset for 3d human activity analysis, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1010–1019. [18] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, et al., Mmdetection: Open mmlab detection toolbox and benchmark, ArXiv abs/1906.07155 (2019). [19] T. S. Motwani, R. J. Mooney, Improving video activity recognition using object recognition and text mining, in: ECAI 2012, IOS Press, 2012, pp. 600–605. [20] C. Liu, X. Li, Q. Li, Y. Xue, H. Liu, Y. Gao, Robot recognizing humans intention and interacting with humans based on a multi-task model combining st-gcn-lstm model and yolo model, Neurocomputing 430 (2021) 174–184. [21] B. Dwyer, J. Nelson, J. Solawetz, et al., Roboflow (version 1.0)[software], 2022. [22] B. De Carolis, D. Lofrese, D. Di Pierro, S. Ferilli, Nutriwell: an explainable ontology-based foodai service for nutrition and health management, in: International Conference of the Italian Association for Artificial Intelligence, Springer, 2024. To appear. [23] M. Dragoni, T. Bailoni, R. Maimone, C. Eccher, Helis: An ontology for supporting healthy lifestyles, in: The Semantic Web–ISWC 2018: 17th International Semantic Web Conference, Monterey, CA, USA, October 8–12, 2018, Proceedings, Part II 17, Springer, 2018, pp. 53–69. [24] M. Y. Kpiebaareh, W.-P. Wu, S. Bayitaa, C. R. Haruna, L. Tandoh, User-connection behaviour analysis in service management using bipartite labelled property graph, in: Proceedings of the 16th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, 2019, pp. 318–327. [25] M. Mancini, R. Niewiadomski, G. Huisman, M. Bruijnes, C. P. Gallagher, Room for one more?introducing artificial commensal companions, in: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, pp. 1–8. [26] A. Rossi, K. Dautenhahn, K. L. Koay, M. L. Walters, Human perceptions of the severity of domestic robot errors, in: Social Robotics: 9th International Conference, ICSR 2017, Tsukuba, Japan, November 22-24, 2017, Proceedings 9, Springer, 2017, pp. 647–656. [27] S. Rossi, A. Rossi, K. Dautenhahn, The secret life of robots: perspectives and challenges for robot’s behaviours during non-interactive tasks, International Journal of Social Robotics 12 (2020) 1265–1278.