Prerequisite Knowledge of Learning Environments in Human- Robot Collaboration for dyadic teams

Abstract

Human-robot teams are organizational units consisting of a human and a robot working on the same task at the same time in a common workspace. They work flexibly with each other and the assignment of subtasks occurs during direct collaboration. It is known from human-human interaction that these collaborative activities require certain mental models that can also be applied to human-robot interactions (HRI). Inadequate mental models of the robot can lead to misinterpretations and thus to errors. In this work, it is proposed to use the construction of three essential mental models as prerequisite knowledge for flexible human-robot interaction. The further elaboration of these models in an industrial assembly scenario serves as a concept for the development of a prototype of a learning application in further research.

Keywords

• 1 Paper human-robot interaction
• human-robot collaboration
• mental models

1. Introduction

Collaborative robots have become an integral part of many industrial companies in processes that are difficult to automate. They are intended to support humans and keep them as an active part of production [ 1 ]. It is required that the cooperation in such socio-technical systems is designed humancentered [ 3 ]. This can mean, for example, giving part of the process control back to the human being and implementing human-robot teams that work flexibly with each other. Such a dyadic team in humanrobot collaboration can be seen as an organizational unit consisting of a human and a robot [ 3 ] working on the same task at the same time in a common workspace [ 1 ]. The assignment of subtasks occurs during the direct collaboration, where both agents start a subtask on their own initiative [ 3 ] and takes place in joint coordination depending on the situation [ 5 ]. To ensure smooth and secure production processes, it is highly important that the interaction between the robot and the human feels natural and pleasant. An actual realization of such a natural interaction is not yet possible with the current state of the art [ 2 ]. Although the interaction between humans and robots shows a rise in similarities to interpersonal communication, there are still fundamental differences. For example, especially in industrial environments, robots usually lack communication channels like verbal communication or the communication of attention, for example, through the direction of gaze. These can be partially substituted by other technologies, such as displays, projections and AR-HMDs. In this process, the sometimes subtle implicit information that humans effortlessly understand in other humans is transferred into explicitly presented information. Subconscious processes with lower amounts of information to be processed now require conscious attention [ 3 ]. For humans, this means that these differences in human-robot interaction compared to human-human interaction must first be learned for successful cooperation. Inadequate human mental models of the robot and the situation can lead to misinterpretations and thus to errors. Therefore, the question arises which mental models are necessary for human-robot collaboration and which aspects should be considered for the respective models. In this work, a selection of prerequisite knowledge is proposed, which is necessary for a smooth cooperation in flexible human-robot teams.

2. Related Work

One of the prerequisite knowledges of human-robot team collaboration is a mental model of the robot. One approach to transfer a mental model to the human is, to use a humanoid robot which could be interpreted with human-like capabilities and constraints [ 4 ]. Another way to infer human-like capabilities and constraints is, to use human-captured motions and show these captures alongside the experiment [ 8 ]. These human-human interaction captures with interaction selection and temporal alignment could create a foundation for dynamic collaboration of human-robot interaction. For this, the motion data from a human-human demonstration is captured, selected, and temporally aligned with a potential similar human-robot interaction. Tausch describes this dynamic collaboration as a selforganizing sociotechnical system [ 7 ]. In this system, a human and a robot coordinate task dynamically between each other. Each set of actions could be started from either side through self initiation. Additionally, this dynamic task interaction could be different depending on the individual in order to create natural HRI [ 6 ].

Häusler and Sträter [ 3 ] write that mistakes can happen when the necessary knowledge is not present or the mental model is inadequate. This might be true for the mental model in HRI as well. Constructing the correct mental model for the robot is also agreeing with Wischmann [ 9 ] who writes that humans in HRI need an extended understanding of the full task and capabilities of the robot. Especially due to the high demand in human control over the processes in HRI compared to human-human interaction, the understanding of the mental model and full knowledge of the task are mandatory for this new type of collaboration.

According to Buxbaum and Häusler [ 2 ] the ability for human collaboration requires three types of mental models. First, the situation model includes the necessary information of the situation and possible goals, second, the person model includes the individual properties of the collaboration partner and, third, the self model includes the information about the capabilities of the self, for example own motives, goals, abilities, and limitations. In the following chapter, we specify the prerequisite knowledge of these types of mental models for human- robot teams.

3. Prerequisite Knowledge of Human-Robot-Teams

The basis for collaboration in human-robot teams is provided by the mental models for the collaboration ability of humans presented in the previous chapter. Fig. 1 demonstrates the three mental models in an collaborative assembly scenario.

The model of the collaboration partner in this case refers to the robot, even if it is not a person. Nevertheless, it makes sense to speak of a person model in this scenario. Since an intention must also be attributed to the robot as a collaboration partner so that the mirror neurons are activated in humans [ 3 ] and the next steps of the robot can be anticipated. The situation model contains the information about the working environment and the common task that is to be processed with the robot.

In order to get a better understanding of the exact aspects of the mental models involved, literaturebased core aspects of collaborative assembly were collected, grouped and assigned to the three mental models mentioned above. The result is shown in Fig 2. For the purpose of clarity, only the essential aspects for interaction between humans and robots were considered. In respect to the robot’s model, people who have little or no experience working with robots are likely to start the collaboration with a nearly empty mental model. However, memories from the media (movies, books, games) can also have an influence on this model that do not apply to a particular use case. Therefore, it is important for the human in the collaboration to have an adequate mental model of the robot that will actually be used. One of the most important aspects of correctly anticipating the robot’s actions is to first have an understanding of what capabilities the robot has. These capabilities can be further divided into physical, perceptual and cognitive capabilities. In contrast to a static preprogrammed process, in a collaborative scenario with an autonomously acting robot it is also important for the human to understand what and how the robot perceives the human and its environment and which decisions the robot can make on which information basis. Thus, the mental models are also connected to each other, e.g. only when the task and the capabilities of the robot are known, a meaningful decision can be made about assigning a certain subtask to the robot.

The concept serves to make the abstract idea of mental models in human robot collaboration more concrete by means of an example and to identify the core aspects. Thus, it can be used to define the requirements for a learning application in flexible collaborative assembly.

4. Conclusion and Future Work

In order to learn a mental model of the robot we have proposed different types of prerequisite knowledge. These include the possible interaction types of the robot and restrictions of movement. In order to collaborate, human and robot have to find a common ground with shared information and shared attention.

One limitation for sharing information might be the limited communication channels of the robot. Especially in an industrial assembly cell, the robot might not be designed to have sufficient communication channels for the current task. This could restrict the capabilities of learning the prerequisite knowledge efficiently. Therefore, a possible future approach could be, to use immersive technologies in tandem with the robot, to create an environment which multimodal information and interaction possibilities.

In this paper we show the prerequisite knowledge of learning environments in human-robot collaboration for human-robot teams. We present a detailed look at three necessary mental models, which will serve as the basis for the development of an immersive learning environment in further work.

Acknowledgments

This project was funded by the BMBF, the German Federal Ministry of Education and Research, under the MILKI-PSY (ger. abb. for Multimodal immersive learning with artificial intelligence for psychomotor training; https://www.milki-psy.de) name and the grant code: 16DHB4013.

References

[1] Buxbaum , H.J. , Häusler , R.: Ladenburger Thesen zur zukünftigen Gestaltung der MenschRoboter-Kollaboration , pp. 293 - 317 . Springer Fachmedien Wies- baden, Wiesbaden ( 2020 ). https://doi.org/10.1007/978-3- 658 -28307-019, https://doi.org/10.1007/978−3− 658 −28307-019

[2] Buxbaum , H.J. , Häusler , R.: Ladenburger Thesen zur zukünftigen Gestaltung der MenschRoboter-Kollaboration , pp. 293 - 317 . Springer Fachmedien Wiesbaden, Wiesbaden ( 2020 ). https://doi.org/10.1007/978-3- 658 -28307-0-19, https://doi.org/10.1007/978-3- 658 -28307-0-19

[3] Häusler , R. , Sträter , O. : Arbeitswissenschaftliche Aspekte der Mensch-Roboter- Kollaboration , pp. 35 - 54 . Springer Fachmedien Wiesbaden, Wiesbaden ( 2020 ). https://doi.org/10.1007/978-3- 658 - 28307-03, https://doi.org/10.1007/978-3- 658 - 28307-03

[4] Ikemoto , S. , Amor , H.B. , Minato , T. , Jung , B. , Ishiguro , H.: Physical human- robot interaction: Mutual learning and adaptation . IEEE Robotics & Automation Magazine 19 ( 4 ), 24 - 35 ( 2012 )

[5] Koehler , M.J. , Mishra , P. , Kereluik , K. , Shin , T.S. , Graham , C.R.: The technological pedagogical content knowledge framework . In: Handbook of research on educational communications and technology , pp. 101 - 111 . Springer ( 2014 )

[6] van Merriënboer , J.: The four-component instructional design model: An overview of its main design principles . 4cid. org ( 2019 )

[7] Onnasch , L. : Mensch-roboter-interaktion-eine Taxonomie für alle Anwendungsfälle . baua: Fokus 1 , 1 - 12 (07 2016 ). https://doi.org/10.21934/baua:fokus20160630

[8] Salter , T. , Dautenhahn , K. , te Boekhorst , R.: Learning about natural human-robot interaction styles . Robotics and Autonomous Systems 54 ( 2 ), 127 - 134 ( 2006 ). https://doi.org/https://doi.org/10.1016/j.robot. 2005 . 09 .022, https://www.sciencedirect.com/science/article/pii/S092188900500151X, intelligent Autonomous Systems

[9] Tausch , A. : Aufgabenallokation in der mensch-roboter-interaktion. 4 . In: Workshop MenschRoboter-Zusammenarbeit. Bundesanstalt für Arbeitsschutz und Arbeitsmedizin , Posterpräsentation, Dortmund ( 2018 )

[10] Vogt , D. , Stepputtis , S. , Grehl , S. , Jung , B. , Amor , H.B.: A system for learning continuous humanrobot interactions from human-human demonstrations . In: 2017 IEEE International Conference on Robotics and Automation (ICRA) . pp. 2882 - 2889 . IEEE ( 2017 )

[11] Wischmann , S. : Arbeitssystemgestaltung im Spannungsfeld zwischen Organisation und menschtechnik interaktion - das beispiel robotik . In: Zukunft der Arbeit in Industrie 4.0 , pp. 149 - 160 . Springer Vieweg, Berlin, Heidelberg ( 2015 )