Examining the impact of data augmentation for psychomotor skills training in human-robot interaction

Abstract

Training psychomotor skills for human-robot interaction is generally done with a human trainer educating the human on how to handle the robot effectively and interact with it safely and efficiently. The dynamic interaction between a robot and a human requires complex machine learning algorithms to be modeled, and these algorithms rely on a large amount of data to be trained. Such data are collected by sensors when a human interacts with a robot. Consequently, the data must be annotated by an expert. Finally, with the annotated data, a psychomotor skills training model can be created to assist the training process. This is a time intensive and costly process. To ease the costs and cut down collection time, we propose the use of data augmentation.

Keywords

• 1 Human-robot interaction
• data augmentation
• machine learning

1. Introduction

Psychomotor skills constitute an essential element of human-robot interaction. The development of psychomotor skills requires hands-on practice. In most cases, the practiced skills need to be executed repetitively by the learner in order to, for example, build muscle memory and support further skill development. Moreover, structured instructions and feedback facilitate the learning process and allow safe performance of the practiced skills. Thus, an educational model for psychomotor skill training needs to support the timely communication of the instructions and feedback and must define how these instructions and feedback are presented to the learner. The educational model also supports the evaluation of the learning outcome. Currently, doing this in a remote manner makes the learning process ineffective and inefficient, usually hindering the beginner's learning progress. The project MILKI-PSY aims at improving the remote learning process of psychomotor skills. In this study, the data will be collected using the multimodal pipeline framework [ 1 ] which is used to handle multimodal data specifically.

Human-robot interaction describes the process of a human interacting with a robot in a shared physical environment. In certain cases, the human and the robot operate in a cooperative manner. To ensure a safe, efficient, and effective interaction requires training of both the robot and the human counterpart. Training humans in the handling of industrial robots is usually done by a human trainer. To assist the training of psychomotor skills, we propose a pedagogical approach. The goal of our pedagogical model is to facilitate a safe, effective, and efficient learning environment for the learner. Particularly, in this study, we focus on a collaborative assembly task between a human operator and an industrial robot in which they cooperate as depicted later in figure 2.

In this paper, we first go over the related work in the field of human-robot interaction. Then, we present three research questions that we aim to address in this Ph.D. research. Next, we will discuss how we are going to achieve this in the methodology part. Finally, we conclude with the expected impact and a discussion at the end of this paper.

2. Related work

Human-robot interaction is a field dedicated to understanding, designing, and evaluating robotic systems that interact with humans. Interaction, by definition, requires communication between the interacting parties, i.e., robots and humans. The communication between robots and humans may take completely different forms depending on the distance between them. Goodrich and Schultz [ 2 ] categorizes human-robot communication into two; proximate communication in which the communicating parties share the same space (physical or virtual) and remote communication where those parties are apart. In this study, we focus on an assembly task where the human operator shares the physical space with the robot. In our remote learning scenarios, we rely on immersive technologies where there is no physical robot but the operator and the virtual robot still shares the same virtual environment. Thus, the method of communication between the parties is proximate.

Training in human-robot interaction gradually becomes more important for the next generation of robot systems. Particularly, in cases of remote training or additional training outside of the conventional teaching methods, a user-friendly interface should be used [ 3 ] to improve the learning process. This user interface should be designed differently depending on the communication category.

Immersive user interfaces such as AR and VR model the behavior of the human and robot agents. The development of such behavioral models heavily uses machine learning techniques that rely on a large amount of data. Such data can be collected via environmental sensors and cameras that capture the activities of human and the robot while they interact. When expensive machines like industrial robots are used, data collection becomes costly and effort-intensive. In such environments the number of robots available for data collection purposes is also a limitation. Having a limit of one or two robots is not uncommon in data collection. On the other side, for most data collection, the limiting factor might be a machine-operating human which naturally is limited in the amount of data they can collect in a full day. To address the issue of data collection we propose data augmentation which is a family of techniques that allows us to synthesize realistic data.

Formally, data augmentation can be defined as techniques aiming at the creation of synthetic data [ 4 ] for the expansion of the size and/or the diversity of the dataset [ 5 ]. A sub-field of data augmentation is domain randomization [ 6 ]. Domain randomization can expose the machine learning model to many different variants of the same problem [ 7 ][ 8 ] and therefore, train the model more robustly. Another sub-field of data augmentation is domain adaptation, which aims to mitigate the covariate shift problem given that training and evaluation sets derive from the same distribution. Studies exist in the literature that indicate using domain adaptation can positively impact the performance of the machine learning model, especially in the domain of human pose detection for activities [ 5 ].

Using machine learning to categorize complex psychomotor activity data for educational purposes has been done before. For example, Spikol et al. [ 9 ] used multimodal learning analytics to collect and provide different data about the interaction between the learner and the system. In cases where the number of potential activity categories is significantly limited, such as the CPR tutor from Di Mitri et al. [ 10 ] and the table tennis tutor by Mat Sanusi et al. [ 11 ], using data augmentation might only marginally increase the results and therefore might not be feasible due to the initial workload that those algorithms take. On the other hand, human-robot interaction is a complex task and therefore might greatly benefit from data augmentation.

3. Research questions

The following research questions are the focus of this Ph.D. research. First of all, it is important to know what the current status in teaching human-robot interaction is and how humans are trained to handle industrial robots. This requires an extensive review of the literature that focuses on teaching humans how to use and operate robots.

1. What are the common practices and

mistakes in human-robot interaction when handling industrial robots?

In order to allocate common practices in teaching human-robot interaction, it is crucial to know what kind of instructions are given by the trainer and what kind of feedback is received by the trainee. Moreover, we also aim to examine the effect of various training approaches (i.e., static, variable, and dynamic) on the trainees learning progress. Secondly, this study addresses current technologies that can support the training of psychomotor skills and facilitate the teaching of human-robot interaction.

2. What technological support is

achievable in educational human-robot interaction?

When looking at technologies, the focus of this research relies on what existing technologies cover both robots and humans and also what kind of machine learning technologies are available. In this study, we will utilize immersive technologies that vary in terms of level of intrusion. For example, the use of a head-mounted display to provide feedback to the learner in an augmented reality setting has a lower level of intrusion than a completely simulated learning environment. Thirdly, it is important for this research to focus on the identification and classification of common mistakes made during psychomotor skills training with robots.

3. How can data augmentation assist the successful replication of common mistakes and how can we measure this impact?

To address this research question, we will consult experts in training humans on how to interact with industrial robots and classify the common mistakes that can take place. Then we will explore how we can augment data in a meaningful manner to replicate common mistakes.

4. Methodology

In this study, first we conduct a systematic literature review in the following fields of research: which are used in the domain of educational technologies.

1 Educational human-robot interaction 2 Technologies in human-robot interaction 3 Semi-supervised learning models 4 Data augmentation – Domain randomization

The first research field is educational human-robot interaction and it refers to the education of humans in handling industrial robots. This includes the common training practices as well as common mistakes the trainee makes during the interaction. In this study, we will use an industrial robot to assemble a box in cooperation with a human as seen in figure 2. First, the human learner will be trained on how to interact with the robot appropriately. Then, the learner will be instructed on the specifics of the assembly steps.

The second research field is the technologies used in human-robot interaction. This addresses data augmentation and domain randomization which are both active fields of development, and research papers about these topics in the domains of 3D pose detection [ 5 ] and object detection [ 4 ] are released frequently in recent years.

After the systematic literature review, the next step will be to design a theoretical framework. This framework includes the design for an immersive training environment specifically for psychomotor skills training. We will use the four-component instruction design (4C/ID) [ 12 ] method to create our framework. In 4C/ID, the design is split up into four different components. The first component is the learning tasks which aim at integrating skills and show a high variability of practice. The second component is the supportive information which focuses on the performance of non-routine aspects of learning tasks. It is specified per task class and always available throughout the whole learning process. The third component is the part-task practice. This aims to provide additional practice for individual routines selected by either the trainer or trainee. The last component is procedural information. Procedural information specifies how to perform routine aspects of a task, for example by giving step-by-step instructions. This procedural information is presented just in time during training and is gradually less present with the increasing expertise of the trainee. In the case of designing a system for human-robot interaction, we will use 4C/ID to teach the human learner different aspects of handling industrial robots. 4C/ID provides a framework to handle non-repetitive tasks which include task non-specific repetitive elements. The psychomotor skills training in human-robot interaction has similar non-repetitive tasks which we are focusing on in this research.

The overall process of this research methodology is design-based research as illustrated in figure 1. The steps of design-based research include analysis, design, development, implementation, evaluation, and reflection. In contrast to predictive research, design-based research uses an iterative process. This means after evaluating the results, the entire process can be iterated based on the ADDIE (i.e., analysis, design, development, implementation, evaluation) model [ 13 ]. While a generic ADDIE model jumps from the evaluation step directly to the solution of the problem [ 14 ], this approach includes a reflective phase whereby all previous steps are examined and refined for the next iteration. This reflection and refinement of problems, solutions, methods, and design principles systemically tries to accommodate for innovative solutions for reallife problems [ 15 ].

In this study, we use machine learning to extrapolate from collected data and model the dynamic human-robot interaction environment. The process of data collection can take many forms. Using the physical environment for data collection has positive and negative aspects. On the positive side, collected data naturally captures and expresses the task that the robot has to perform. On the negative side, the data collection task has certain limitations such as, the availability and the speed of the robot and human, the limited amount of human-robot data collection stations (in most cases one or two robots), and the expected tiredness of the human. In order to counter these problems of data collection, we will be exploring data augmentation.

We are planning to develop multiple prototypes over the course of this research. The first prototype will be designed specifically for the human-robot interaction where both sides have to cooperate in order to assemble a box together. This prototype will use a virtual robot. In the prototype, the human learner can interact with the virtual robot which is a simulated 3D model visible through a camera or headmounted display.

5. Expected Impact

By using data augmentation and generating synthetic data for modeling human-robot interaction, we expect the machine learning model to perform equally or better than a machine learning model trained on physical data alone. We also expect the data collection process to be faster and reusable in future applications. Examining the impact of data augmentation for psychomotor skills training in human-robot interaction, we hope to find a reliable and safe approach for training humans how to handle industrial robots.

6. Conclusion

The training of psychomotor skills is imperative for an effective, efficient, and safe human-robot interaction. In this paper, we propose an educational approach towards the psychomotor skills training of humans in handling industrial robots. Our educational approach includes timely instructions and feedback as well as supporting immersive technologies. The development of such technologies requires machine learning techniques that rely on a large amount of data. However, it is costly and effort intensive to collect data in such settings where the availability of the robots is limited. To address this challenge, we propose the use of data augmentation.

We are going to investigate the impact of data augmentation on the performance of the machine learning models that represent the interaction between the human and the robot in a physical environment. In this study, we are going to conduct an extensive literature review in the domains of education for human-robot interaction, technologies used in human-robot interaction, and data augmentation. Then, a theoretical framework will be designed and an immersive training prototype will be developed. This prototype will be implemented and evaluated in the training environment.

7. Acknowledgements

This project was funded by the BMBF, the German Federal Ministry of Education and Research, under the MILKI-PSY (ger. abb. for Multimodal immersive learning with artificial intelligence for psychomotor training; https://www.milki-psy.de) Name and the grant code: 16DHB4013.

8. References

References

[1] D. Di Mitri , J. Schneider , M. Specht , H. Drachsler , Multimodal pipeline: A generic approach for handling multimodal data for supporting learning , 2019 .

[2] M. A. Goodrich , A. C. Schultz , Humanrobot interaction: a survey , Now Publishers Inc, 2008 .

[3] M. Ishii , A robot teaching method using hyper card system , in: [1992] Proceedings IEEE International Workshop on Robot and Human Communication , 1992 , pp. 410 - 412 .

[4] S. Borkman , A. Crespi , S. Dhakad , S. Ganguly , J. Hogins , Y.-C. Jhang , M. Kamalzadeh , B. Li , S. Leal , P. Parisi , C. Romero , W. Smith , A. Thaman , S. Warren , N. Yadav, Unity perception: Generate synthetic data for computer vision , 2021 .

[5] E. Spyrou , E. Mathe, G. Pikramenos, K. Kechagias , P. Mylonas , Data augmentation vs. Domain adaptation-a case study in human activity recognition , Technologies 8 ( 2020 ).

[6] L. Weng , Domain randomization for sim2real transfer, lilianweng .github.io/lillog ( 2019 ). URL: http://lilianweng.github.io/lillog/2019/05/04/domainrandomization.html.

[7] J. Tobin , R. Fong , A. Ray , J. Schneider , W. Zaremba , P. Abbeel , Domain randomization for transferring deep neural networks from simulation to the real world , 2017 .

[8] OpenAI , I. Akkaya, M. Andrychowicz , M. Chociej , M. Litwin , B. McGrew , A. Petron , A. Paino , M. Plappert , G. Powell, R. Ribas , J. Schneider , N. Tezak , J. Tworek , P. Welinder , L. Weng , Q. Yuan , W. Zaremba , L. Zhang, Solving rubik's cube with a robot hand , 2019 .

[9] D. Spikol , E. Ruffaldi, G. Dabisias, M. Cukurova, Supervised machine learning in multimodal learning analytics for estimating success in project-based learning , Journal of Computer Assisted Learning 34 ( 2018 ) 366 - 377 .

[10] D. Di Mitri , J. Schneider , K. Trebing , S. Sopka , M. Specht , H. Drachsler , Real-time multimodal feedback with the cpr tutor , in: I. I. Bittencourt , M. Cukurova , K. Muldner , R. Luckin , E. Millán (Eds.), Artificial Intelligence in Education , Springer International Publishing, Cham, 2020 , pp. 141 - 152 .

[11] K. A. Mat Sanusi , D. D. Mitri , B. Limbu , R. Klemke , Table tennis tutor: Forehand strokes classification based on multimodal data and neural networks , Sensors 21 ( 2021 ) 3121 .

[12] J. van Merriënboer , The four-component instructional design model: An overview of its main design principles . 4cid. org , 2019 .

[13] M. Molenda , In search of the elusive addie model , Performance improvement 42 ( 2003 ) 34 - 37 .

[14] M. De Villiers , P. Harpur , Design-based research-the educational technology variant of design research: illustrated by the design of an m-learning environment, in: proceedings of the South African institute for computer scientists and information technologists conference, 2013 , pp. 252 - 261 .

[15] T. Amiel , T. C. Reeves , Design-based research and educational technology: Rethinking technology and the research agenda , Journal of educational technology & society 11 ( 2008 ) 29 - 40 .