Robots allow humans to ofload repetitive tasks to be executed with high precision. However, when we enter the space of collaboration with robots, this opens up the great potential to directly support humans. In the future, we envision that cooperative robots are integrated into kitchens as every other cooking appliance. In this work, we present several scenarios where the robot supports the human in the kitchen. We then outline the opportunities of cooperative robots in kitchens and the challenges that such a setup brings.
For decades, robots have been used in industrial settings to assemble products with high precision
and speed. However, while doing so, they are typically enclosed in steel cages. We have recently
witnessed developments in robotics where humans can now safely and cooperatively work
together with robots in the same space. With this paradigm, we can now support humans in
high-precision tasks exactly where they need it (cf. Hentout et al. [
We have already come a long way since the first steps of automation in homes. For example, roombas that vacuum our floors, robots that cut our grass or clean the pool have become commonplace. Appliances in our households are interconnected and intelligently adapt to our needs and schedules. In the advancement of robot technology, several attempts have also been made to automate food preparation [5, 6, 7]. For instance, Sugiura et al. [8] uses small mobile robots, QR codes, and a graphical user interface for the robots efectively help the user. In domestic environments, cobots are regularity envisioned to have humanlike manner (cf. de Graaf et al. [9]). In the context of cooking, prior work mainly focused on autonomous cooking. So far we have not seen attempts to incorporate telepresence robots in the kitchen, although this is a common use of robots in other contexts [10, 11].
Previous work implies that it is technically feasible for humans to work side-by-side with cobots in a small space like a kitchen. Thus, in this work, we explore a range of scenarios in which robots can be used to support humans in cooking-related tasks. However, there is still a long way to go until aspects of our vision can become a reality, both from a technical and a human-robot interaction (HRI) design perspective. Consequently, we discuss the opportunities in more depth and, more importantly, the challenges that must be solved. Most critically, these encompass establishing trust in technology, adequate interaction patterns for human-robot collaboration, and learning and adjusting to human behavior patterns for seamless collaboration.
One or multiple robotic arms mounted in the kitchen can support the human in many ways while cooking. On the one hand, this allows direct support in the room for the human, e.g., taking on cutting tasks or stirring the stew. On the other hand, the augmentation of cobots with telepresence capabilities creates new possibilities for cooperation over distance. Instead of being limited to consuming or producing only digital content (e.g., watching a video), users could engage in activities that require the physical manipulation of objects. In the following, we present several scenarios of a cobot enhanced smart kitchen.
In the most basic setting, a kitchen cobot supports the user while cooking, either helping to achieve the task faster or taking over parts of the cooking completely (cf. Sugiura et al. [8]). Here, we envision that the cobot can take on simple and disliked tasks, cutting onions or stirring in a pot every other minute. This will support people with a busy work schedule to ofload time-consuming tasks. Thus, simple cooking support will promote healthy, home-cooked food over dining out and ordering food.
Similar to existing AR applications, which allow in-field operators to receive expert advice [12], we envision that users can take cooking classes supervised remotely by chefs or nutrition experts who can intervene or demonstrate practices. With this scenario, we follow the teachingby-demonstration approach (cf. Funk et al. [13]), which could improve learning compared to seeing a video and motivate users actually to perform actions instead of just watching [14, 13]. The actions performed remotely by a chef or instructor are mapped to the local kitchen robot in real-time. In such a scenario, we envision that the remote person also has a cobot that records the chef’s actions, and the local robot performs them to demonstrate the actions to the user.
Using cameras that track user movements during cooking can recreate the exact movements by a kitchen cobot elsewhere. Not only the user’s movements are important, but also ingredients selection and quantity. However, the kitchen cobot can understand the recipe and re-create it elsewhere with additional sensors. This will enable long-distance cooperation, e.g., between two friends or a couple, allowing them to cook together despite the long distance.
We all know how hard it can be to cook traditional family recipes, such as a Thanksgiving or a casserole recipe; often, this takes years of practice, and thus, over time, traditions get lost. However, a cobots’ vision system can track the chefs’ movements and actions, allowing for recipes to be physically “recorded”, and thus, they can be reenacted at any time. This way, everyone can prepare a dish the exact same way it was prepared the day of recording. This opens a potential for diferent scenarios in which we want to prepare food exactly like grandma, a celebrity chef, a restaurant, or a friend or loved one. Finally, we can envision an authentic recipe platform to share and preserve dishes exactly how they were prepared and meant to be by their creators.
Building on pre-recorded instructions, the level of automation can be varied. That is, in an early stage the robot does most of the work to show the “student” how it is done (cf. Nakauchi et al. [15]). Over time, each time the recipe is cooked, the robot leaves more parts to the user, until they have internalized the process.
After collaborative and cooperative scenarios in the kitchen, the next step is to enable fully autonomous cooking using the cobot (e.g., baking robot [16]). In this case, instructions to the cobot chef are only made verbally. Here, we envision a waiter-style conversation to prepare a meal in the desired way.
While robotic arms are a relatively new addition to households, we have presented a set of scenarios with a lot of opportunities for society. However, we argue that for them to become reality, a range of challenges have to be addressed in the future. For users to acquire and accept cooperative robots in their household, not only the cost has to decrease further, but predictability and safety have to be ensured [17, 18].
Safety is of the essence in a cobot-equipped kitchen, as kitchen work includes potentially dangerous procedures. Especially, hot food, boiling water, and sharp objects have to be handled with care. For the robot to cut or slice ingredients, procurements have to be made to get the user out of harm’s way. The simple solution would be to have a designated “cutting area” where the robot can take ingredients to chop that the user cannot access; however, this is contradicting the cooperative nature of cobots. Thus, we argue that we need better forms of communication with humans to convey the notion of safety and algorithmic safety built into the cobot.
As future systems need to provide safety, they need to recognize people and objects in the environment, the changing environment (e.g., open doors, moved chairs), and the status of food (cf. Salekin et al. [19], Ciocca et al. [20]). For recording user movements, a number of computer vision based pose detection libraries already exist (cf. Cao et al. [21]), in addition, these can be re-trained and adapt to a specific user. In most kitchens, many ingredients and tools have more or less designated places. These can be learned by the robot control software, but it will never be the exact same place and often things will be out of place. We envision a hybrid approach, where the user can point the cobot coarsely in the right direction on an “unlocateable” object using mid-air pointing (cf. Schweigert et al. [22], Mayer et al. [23]), so that the cobot can apply an automated object detection and gripping procedure to acquire the object.
To make sure hot food or kitchenware cannot be dropped, the robot has to be equipped with adequate grippers. For instance, eggs have to be treated diferently than a cutting board. While specialized grippers and solutions to automatically attach them to a robot already exist, we argue that a cheaper more feasible solution would be separate adapters that can be attached to standard kitchenware and expose a counterpart that can be tightly gripped by a multi-use end efector.
The look and feel, aka the behavior, of a robot is still a heavily discussed topic. Today, most systems are designed to accomplish a task; however, are missing the human-centered approach. Here, we see a number of areas that need to be addressed. While a number of cobots are designed with a humanoid robot character, most are technological driven (cf. R2-D2 vs. C-3PO). Another aspect is to operate within the human interpersonal distances [24] which also can be an starting point for interaction patters (cf. Toure et al. [25]). In summary, these characteristics can be described as anthropomorphism, which will have an influence on the human-robot interaction and collaboration (cf. Paetzel-Prüsmann et al. [26]). Thus, we argue that these challenges need to be addressed to allow automation of domestic cobots ease people’s lives but also improve their connectedness, engagement and general well-being.
On the one hand, cobots as cooking appliances can support people with needs providing users with the ability to perform tasks that they cannot be done alone (cf. Ma et al. [27]). On the other hand, smart kitchens have accessibility issues [28]. Thus, while cobots can help, they can also be inaccessible. We argue that in the future, it will be essential to provide a multi-modal interaction concept to unlock the full potential of cobot cooking appliances.
Already today, we see a transformation for robots from merely taking over dangerous or repetitive tasks from humans to cooperating with them and enabling new forms of human-robot but also human-human collaboration. In this work, we outline the vision of cooperative robots as cooking appliances. First, we highlight the great opportunities by describing six scenarios in which a cobot can support the user in various ways, such as promoting a healthy diet. Then, we discuss the challenges they lay ahead to let these scenarios become a reality.
While we argue that the presented challenges will take an interdisciplinary efort to overcome, in the next step, we aim to build a testing environment to investigate the interaction in a smart kitchen with cobots. When the human-robot interaction is designed and carried out eficiently, we believe the automation of domestic cobots will ease people’s lives and improve their connectedness, engagement, and general well-being.
This project is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 425412993 and is part of Priority Program SPP2199 Scalable Interaction Paradigms for Pervasive Computing Environments. 1835–1874. URL: http://link.springer.com/10.1007/978-3-319-32552-1_69. doi:10.1007/ 978-3-319-32552-1_69. [5] H. Zhao, X. Hao, A. Wang, C. Li, Design and Implementation of an Intelligent Cooking Robot Based on Internet of Things, in: Proceedings of the 2015 Chinese Intelligent Automation Conference, volume 338, Springer Berlin Heidelberg, Berlin, Heidelberg, 2015, pp. 423–430. doi:10.1007/978-3-662-46466-3_42. [6] C. Dong, L. Yu, M. Takizawa, S. Kudoh, T. Suehiro, Food Peeling Method for Dual-arm Cooking Robot, in: 2021 IEEE/SICE International Symposium on System Integration, SII ’21, IEEE, Iwaki, Fukushima, Japan, 2021, pp. 801–806. doi:10.1109/IEEECONF49454. 2021.9382700. [7] K. Junge, J. Hughes, T. G. Thuruthel, F. Iida, Improving robotic cooking using batch bayesian optimization, IEEE Robotics and Automation Letters 5 (2020) 760–765. doi:10. 1109/LRA.2020.2965418. [8] Y. Sugiura, D. Sakamoto, A. Withana, M. Inami, T. Igarashi, Cooking with robots: Designing a household system working in open environments, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, Association for Computing Machinery, New York, NY, USA, 2010, p. 2427–2430. doi:10.1145/1753326.1753693. [9] M. M. A. de Graaf, S. Ben Allouch, J. A. G. M. Van Dijk, What makes robots social?: A user’s perspective on characteristics for social human-robot interaction, in: Social Robotics, Springer International Publishing, Cham, 2015, pp. 184–193. doi:10.1007/ 978-3-319-25554-5_19. [10] M. D’Souza, J. Gendreau, A. Feng, L. H. Kim, A. L. Ho, A. Veeravagu, Robotic-Assisted Spine Surgery: History, Eficacy, Cost, And Future Trends, Robotic Surgery: Research and Reviews Volume 6 (2019) 9–23. doi:10.2147/RSRR.S190720. [11] S. S. Kim, M. Dohler, P. Dasgupta, The Internet of Skills: use of fifth-generation telecommunications, haptics and artificial intelligence in robotic surgery, BJU International 122 (2018) 356–358. doi:10.1111/bju.14388. [12] D. Calandra, A. Cannavò, F. Lamberti, Improving AR-powered remote assistance: a new approach aimed to foster operator’s autonomy and optimize the use of skilled resources, The International Journal of Advanced Manufacturing Technology 114 (2021) 3147–3164. doi:10.1007/s00170-021-06871-4. [13] M. Funk, L. Lischke, S. Mayer, A. S. Shirazi, A. Schmidt, Teach Me How! Interactive Assembly Instructions Using Demonstration and In-Situ Projection, Springer Singapore, Singapore, 2018, pp. 49–73. doi:10.1007/978-981-10-6404-3_4. [14] M. Kardas, E. O’Brien, Easier Seen Than Done: Merely Watching Others Perform Can Foster an Illusion of Skill Acquisition, Psychological Science 29 (2018) 521–536. doi:10. 1177/0956797617740646. [15] Y. Nakauchi, T. Fukuda, K. Noguchi, T. Matsubara, Intelligent kitchen: cooking support by lcd and mobile robot with ic-labeled objects, in: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2005, pp. 1911–1916. doi:10.1109/IROS.2005. 1545346. [16] M. Bollini, S. Tellex, T. Thompson, N. Roy, D. Rus, Interpreting and Executing Recipes with a Cooking Robot, Springer International Publishing, Heidelberg, 2013, pp. 481–495. doi:10.1007/978-3-319-00065-7_33. [17] T. E. Kaonain, M. A. A. Rahman, M. H. M. Arif, W. J. Yahya, K. Mondal, Collaborative robot safety for human-robot interaction in domestic simulated environments, IOP Conference Series: Materials Science and Engineering 1096 (2021) 012029. doi:10.1088/1757-899X/ 1096/1/012029. [18] M. M.A., G.-P. J, A. Khalid., G.-L. J.M., G.-H. S., Risks management and cobots.
Identifying critical variables., in: Proceedings of the 29th European Safety and Reliability Conference, ESREL ’19, Research Publishing Services, 2019, pp. 1834–1841. doi:10.3850/978-981-11-2724-3_0791-cd. [19] M. S. Salekin, A. Babaeian Jelodar, R. Kushol, Cooking state recognition from images using inception architecture, in: International Conference on Robotics,Electrical and Signal Processing Techniques, ICREST ’19, IEEE, 2019, pp. 163–168. doi:10.1109/ICREST.2019. 8644262. [20] G. Ciocca, G. Micali, P. Napoletano, State recognition of food images using deep features,
IEEE Access 8 (2020) 32003–32017. doi:10.1109/ACCESS.2020.2973704. [21] Z. Cao, T. Simon, S.-E. Wei, Y. Sheikh, Realtime multi-person 2d pose estimation using part afinity fields, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’17, IEEE, 2017. doi:10.1109/CVPR.2017.143. [22] R. Schweigert, V. Schwind, S. Mayer, Eyepointing: A gaze-based selection technique, in: Proceedings of Mensch Und Computer 2019, MuC’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 719–723. doi:10.1145/3340764.3344897. [23] S. Mayer, V. Schwind, R. Schweigert, N. Henze, The efect of ofset correction and cursor on mid-air pointing in real and virtual environments, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–13. doi:10.1145/3173574.3174227. [24] R. Mead, M. J. Matarić, Perceptual Models of Human-Robot Proxemics, volume 109 of Springer Tracts in Advanced Robotics, Springer International Publishing, 2016, p. 261–276. doi:10.1007/978-3-319-23778-7_18. [25] D. Toure, R. Welsch, S. Mayer, The future of proxemic interaction in smart factories, in: Proceedings of the Automation Experience at the Workplace, 2021. URL: https://sven-mayer. com/wp-content/uploads/2021/05/toure2021future.pdf. [26] M. Paetzel-Prüsmann, G. Perugia, G. Castellano, The influence of robot personality on the development of uncanny feelings, Computers in Human Behavior 120 (2021) 106756. doi:10.1016/j.chb.2021.106756. [27] W.-T. Ma, W.-X. Yan, Z. Fu, Y.-Z. Zhao, A chinese cooking robot for elderly and disabled people, Robotica 29 (2011) 843–852. doi:10.1017/S0263574711000051. [28] T. Kopp, M. Baumgartner, S. Kinkel, Success factors for introducing industrial humanrobot interaction in practice: an empirically driven framework, The International Journal of Advanced Manufacturing Technology 112 (2021) 685–704. doi:10.1007/ s00170-020-06398-0.