Augmented reality is a promising technology for vocational training of assembly processes in industry. A field that has received little attention is the application in collaborative assembly, where humans and robots work closely together. Working with a robot increases the complexity of the assembly process and therefore requires effective training. The MILKY-PSY project will investigate, among other aspects, how such immersive training can be implemented for training in collaborative assembly. The development follows the design-based research approach in order to involve the potential user in the early stage. In this work the results of the analysis phase, from three interdisciplinary workshops, are presented. This includes the identification of a specific assembly process, the definition of a training procedure, and the technological requirements for developing such an application. Another important point is the collection of information that should be provided to the trainee for motivated and successful learning. This information can be divided into two categories. Firstly, information that is necessary to complete the specific task and secondly, information that is not directly related to the task but is useful for understanding the task or increasing motivation. These results serve as the basis for a proposed design framework for AR-based training in collaborative assembly presented in this work.
Good vocational and further training is essential for companies, but it is also costly
and time-consuming. In order to meet the increasing demands on well-trained personnel
regarding changes due to the digital transformation, Industry 4.0 and the shift towards
more sustainability, learning and training in industrial settings need to shift towards an
Education 4.0, too [
The MILKY-PSY project aims to develop an innovative, immersive learning
environment for independent learning of psychomotor skills using AI. In the course of the
project, applications for two different fields where the training of psychomotor skills is
necessary will be developed and tested. One is based in the field of sports and the other
in an industrial production environment. This work in progress focuses largely on the
application in the production environment, more precisely, in collaborative assembly.
The aim is to investigate how the application of an AR training system can support the
learning of collaborative assembly processes. Although there are a number of
documented examples of AR systems for assembly training [
As a work in progress, this paper presents the results of the initial phase of the development of a design framework for the provision of multimodal content in an ARbased training system for the acquisition of psychomotor skills. In a first step, the authors give an insight into the current state of the art regarding AR based training systems and AR applications in robotics. In chapter 3, the authors describe the underlying methodology and present the approach to answering the following research question: How can a framework for an AR-based training system for the acquisition of psychomotor skills in a collaborative assembly scenario be designed in order to increase efficiency and effectiveness in training processes? In chapter 4, the authors present and analyze the results of the first iterations that directly feed into the development of a preliminary framework (chapter 5) for collaborative assembly training. The final chapter contains a summary of the results to date and provides an outlook on future research. 2
AR applications in the field of manufacturing have been the subject of research for a several years and there is now a range of work in the field of AR assisted training. This chapter summarizes the current state of AR training for assembly purposes in general, before looking at those specifically for collaborative assembly.
AR applications can add visual and auditory content to the assembly workplace. They
allow to provide the learner with useful information at the right moment during an
assembly procedure. These in situ instructions allow to reduce the cognitive load of the
learner, thus reducing the error rates [
AR technology has become increasingly interesting for human-robot interaction due to
its visualization possibilities. In their review, Makhataeva & Varol [
Although there is already some related work in the field of AR assisted assembly training, AR assisted training for collaborative assembly is still rather rare. Much of the research focuses on safety aspects of human-machine interaction, visualizations and worker support. However, as collaborative assembly is becoming increasingly important in industry, further research is needed.
The development of technology-assisted training purposes is often accompanied by
challenges regarding acceptance and the general use. The need for early involvement
of relevant social groups in development processes is now widely recognized. Methods
like design based research can be applied in development contexts in order to minimize
acceptance issues and improve user experience. “[C]onducting research in context,
rather than in a controlled laboratory setting, and iteratively designing interventions” are
the underlying principles of the design based research approach in order to gather
“authentic and useful knowledge” [
As previously described, the main purpose of this paper is to present the results of
the analysis phase and the specification of the framework. Therefore, the analysis of
the following important aspects of learning with media namely technology, pedagogy
and content as well as their interdependencies is necessary [
The starting point for the user-centric analysis is the identification and specification of a suitable application case. As a result, the collaborative assembly of a sensor case was identified as an application case for the development of the immersive learning environment. Based on the context of use and the requirements analysis an initial theoretical design concept has been developed. In the next section the results of the analysis phase, consisting of the three aspects technology, pedagogy and content are presented. 4
This section presents the results of the analysis phase. First, the outcomes of the three interdisciplinary workshops are presented, which serve as the basis for a design framework for the development of a first prototype.
Content: For the development of the first prototype, a collaborative assembly task was selected in which the trainee assembles a sensor case in collaboration with a robot. Figure 3 illustrates the singular steps of the collaborative assembly task and demonstrates which tasks are to be performed by a robot.
Afterwards a decomposition of the assembly procedure into subtasks and elementary activities based on observations of the process was carried out. In addition to the analysis of singular tasks, individual actions and associated failure modes were identified.
Pedagogy: The target group is novices with no or little previous experience with the
assembly task and collaborative assembly in general. At the end of the training, the
trainee should be able to perform the assembly task independently, without assistance,
to understand and anticipate the robot's movements, and to interact with the robot. To
do this, the learner must know and be able to perform the elements and procedures
required for the task. Training in this case is a process of learning psychomotor skills,
which according to Fitts and Posner [
Technology: A number of hardware and software components are required for the
development of the prototype. The selection of these components is partly based on the
experience of previous projects, as well as further research from the consortium. In
relation to the selected application case, the collaborative assembly, a technology is to
be preferred which allows for the operation to be carried out hands-free. Therefore, with
regard to the delivery of the multimodal content, the industry-ready HMD Microsoft
Hololens has been chosen. In addition, sensors and cameras are required to record the
movements and vital parameters of the learners. This data must then be merged and
processed before it can be used as the basis for generating the most individualized
insitu guidance and feedback. To meet these demands, Słupczyński & Klamma [
Based on the results of this analysis, the authors developed a first draft of a theoretical framework for their first prototype.
The design framework proposed in the following aims to summarize and structure all relevant content for the development of an AR-based training system for collaborative assembly. With the Cobot, another component enters the training system that needs to be considered during development. As previously mentioned, the user plays an active part and important role within the entire development process. Based on the results of the three consecutive workshops, a preliminary training procedure has been developed for learning the assembly procedure as shown in Figure 4.
Learning is always linked to a certain prior knowledge. Therefore, basic skills are
acquired at the beginning and serve as a basis for the actual process and can be built
upon [
Once the training process itself has been defined, the next step is to specify the content that may be necessary or helpful to acquire the skill. Throughout the process of collecting the necessary information provided to the learner, it became apparent that there are two different types of information. More precisely, there is information that is necessary or helpful for completing the task and information that does not relate to individual subtasks but is intended to increase overall understanding and motivation. The task-related content is needed for the acquisition of the procedural skills and the required knowledge. The components required for this are, on the one hand, a step-bystep in-situ instruction, which can be derived from the individual subtasks. According to the fading approach, these initially very detailed instructions should be gradually reduced throughout the training process. An important part of learning applications is feedback, for which three types have been identified: On the one hand in the form of a correction for incorrectly executed steps and on the other hand as reinforcement for correctly executed steps. As a conclusion, the trainee can be given summary feedback on the whole process execution.
In addition, information provided, such as approaches from the field of gamification, are intended to increase motivation or serve better contextual understanding, such as the information provided by the robot. 6
In this paper, an approach has been presented on how to design a framework for an AR-based training system for the acquisition of psychomotor skills in a collaborative assembly scenario can be designed. For this purpose, a context of use and requirements analysis was first carried out in three workshops. The resulting findings form the foundation for the design framework presented, which in turn serves as the basis for the first design cycle in the development of the prototype in the further course of the MILKYPSY project. In the course of the development process, the framework should be evaluated and adapted if necessary. In addition, one objective of further research should be how artificial intelligence can be used to develop context-aware AR training systems that come close to the quality of a personal trainer.