In the last few years, head-mounted displays (HMDs) received a growing amount of attention by the scientific community, especially in the industrial domain. Due to its possibility to work hands-free while providing the user with necessary augmented information, HMDs can enhance the quality and efficiency of assembly and maintenance tasks. Offering tailored information requires knowledge about how to design and present augmented reality (AR) content. However, design guidelines especially for assembly training tasks as well as usability evaluations are very limited. In this paper, we want to overcome this limitation by introducing an application as well as 10 design recommendations for HMD-based assembly training tasks. Furthermore, we execute a user study with15 participants using an engine assembly training task to evaluate the software usability and present results from the system usability scale (SUS) questionnaire, the AttrakDiff as well as the NASA task load index (NASA-TLX) questionnaire.
Augmented Reality (AR) becomes a part of our daily lives. Several applications for smartphones and tablets are already being used by millions of people.
Augmented information are designed to improve communication, enhance human skills and some of them are just for fun. Hand-held devices and projectors are typically used to display superimposed information [1]. In the last years, head-mounted displays (HMDs) received growing interest by researchers in the industrial domain because they offer a hands-free usage and help to increase the quality and efficiency of assembly and maintenance tasks [2; 3]. In order to design a suitable AR application for manual procedural tasks, researchers have to know the optimal information visualization for different devices. Our research is focusing on assembly training tasks because they are very important for the automotive industry. Well executed training must be designed efficiently to ensure a good knowledge transfer whereby optimal process and product quality is guaranteed. However, design guidelines for HMD-based applications as well as comprehensive usability evaluations are still missing [4]. We want to close this gap by providing the following contributions. The second section aims to give a brief overview of the related work. Our patented application is introduced and described in section 3. We aim to set this application as the optimal standard for information visualization using HMDs for assembly training tasks. We execute a user study with 15 participants to assess the usability of our application. Detailed information about the experiment are given in section 4 and section 5. Due to our gained knowledge during the application development and assessment, we extrapolate and present 10 design recommendations for HMD-based assembly training in section 6. A brief discussion and summary follows at the end of this paper in section 7.
Design Guidelines are helpful advices for developers and designers. They provide instructions on how to adopt specific design principles such as controllability, learnability or customizability. Software design recommendations such as the DIN EN ISO 9241-110 [5], Shneidermans 8 Golden Rules of Interface Design [6] and the 10 usability heuristics for user interface design by Jakob Nielsen [7] are often used for general software development. Specific guidelines for projection-based AR are presented by Funk [8]. Eight principles, i.e. hands-free usage and personalized feedback, were gained during a four year project using assistive systems for impaired workers. Further specific recommendations, especially for assembly and maintenance training tasks were published by Webel [9]. Principles such as mental model building, haptic hints, visual aids and passive learning were introduced and focused on acquiring assembly and maintenance skills which is our focus as well. However, until now it is still uncertain how to visualize augmented information efficiently using head-mounted displays (HMDs). Scientific contributions in that field are very rare and limited to just a few [10]. We want to overcome this limitation by giving a first suggestion in the next chapter.
This section provides a brief overview of our patented application. We describe the relevant functionalities and show our user interface design. The application was created using Photoshop for the interface design and Unity3D for the front-end programming. This multimodal application consists of six features with intuitive icons (Figure 1).
The trainee can choose between six modalities for each assembly step. A sound feature provides clear auditory instructions about the current task. Another feature visualizes superimposed static 3D data of the corresponding part (Figure 2). This feature may help to learn the position and orientation of the related part. When selecting a feature, the icon-color changes to green and a click sound occurs which gives the user an immediate feedback of his action. Every feature can be activated and deactivated by either clicking or using the voice command ‘select’.
The text feature provides annotations about the current task showing the relevant parts, the activity (e.g. assembly or plug), the associated materials such as screws and the needed tools. We also implemented a Bezier-curve (Figure 3) which was found to be a good solution for picking guidance in previous studies [11; 12]. We added an animated arrow to visualize the end of the tunnel. The user can use this augmented tunnel to find the correct parts in the shelf. This solution avoids picking mistakes and improves the training performance due to part search no longer being required. Another feature provides superimposed animated 3D information using an outline shader (Figure 4). The outline visualization is sufficient to recognize the part’s geometry and position. Additional arrows show the screw positions. We highly recommend this visualization technique because it allows to assemble the relevant part without any superimposition problems. A visualization such as in Figure 2 may affect the assembly process because the real part is hard to recognize due to the strong color rendering.
The last feature is a video (Figure 5) whereby the user receives detailed information about the current task. Watching a video with an HMD observing someone performing a task facilitates task transfer. We designed the video feature similar to a regular video player. A play and pause button enables the user to have complete control over the feature. The progress bar supports the user in building a mental representation of the task. Additional context information such as a progress bar in the middle of the interface as well as the task overview when selecting the brand icon (Figure 6) supports mental model building and strengthens the training transfer. The user receives information about the finished, the upcoming and the current task. Users are further able to switch between the assembly steps either by clicking the left or right arrow as well as using the task overview or using the voice commands ‘next’ and ‘back’. This concept offers user control and avoids
According to the concept of learning introduced by Fitts and Posner [13], we structured our application using different learning stages. Information are gradually reduced during the training. All features are available in the tutorial level. The first level is made for exploring the task, the application as well as familiarizing with using a HMD. Two features with the strongest guidance were blocked in the beginner level. The augmented tunnel and the outline features were equipped with a bolt sign to visualize the restriction (Figure 7). When clicking on one of the restricted feature, the avatar (we named Embly) loses one of its seven lives (Figure 8). This game-based learning approach aims to motivate the user finishing the task autonomous using the available features without killing Embly. Two more functions, the 3D feature as well as the video function are additionally blocked in the intermediate level. Every feature is restricted in the expert level. Only a default audio with information about the underlying task is provided for each step.
Additionally, we used a single backward fading learning approach which was found to be effective for learning by Renkl [14]. This means, the last step is faded out in the tutorial level, the last two steps in the beginner level, the last three in the intermediate level and the last four steps in the expert level. The user is asked to select the correct part before receiving information about the task (Figure 9). At this time, all six features are blocked. Participants receive a visual (green color) and an auditory feedback as soon as they select the right part. The part is marked red in color if the user selects the wrong part. Afterwards, all features become active and the user is able to continue the assembly procedure. We used this approach because it offers several advantages. Backward fading can decrease the cognitive workload, enhance the learning transfer and improves the initial performance of manual procedural task [15].
We conducted a user study with 15 participants to evaluate the usability of our HMD-based software for assembly training tasks. This section describes the study design, explains the procedure, introduces the hardware setup, gives a detailed information about the participants and reports the results of our measurements.
To evaluate the usability of our training software for assembly training tasks, we designed an experiment with three groups and different knowledge backgrounds in AR and assembly processes (independent variables) following a between-subject design recommended by Nielsen [16]. Measuring the usability of a software includes the assessment of effectivity, efficiency and user satisfaction variables [17]. To gather the effectivity of our training software we measured the dependent variables assembly (AM) and picking mistakes (PM), self-corrected assembly (CAM) and picking mistakes (CPM) as well as correction by help for the assembly (CBHA) and picking (CBHP). We further verify the backward fading questions (BWF) allocating either one point (correct answer) or zero points (wrong answer). As dependent variable for the efficiency, we measured the tutorial level completion time (TLCT), the beginner level completion time (BLCT), the intermediate level completion time (ILCT) as well as the expert level completion time (ELCT). Additionally, we collect user satisfaction data using the extended system usability score (SUS) according to Bangor [18] as well as the AttrakDiff questionnaire [19]. We also used the NASA Task Load Index (NASA-TLX) to rate the perceived workload during the experiment [20]. We measured six dependent variables, the mental workload (MWL), the physical workload (PWL), the temporal workload (TWL), the user performance (UP), the user effort (UE) as well as the user frustration (UF). A high cognitive workload may harm the learning process because fewer cognitive resources are available which are needed to store relevant information in the procedural memory.
For our experiment, we used a Microsoft HoloLens HMD to display our training software, providing all assembly instructions. In contrast to other researchers who used low complex Lego Duplo assembly task to evaluate their solutions [21], we used a real engine assembly task. The test environment was build referring to the production workplace (Figure 10). The workplace consists of three areas. A shelf area providing all the parts and screws necessary for the assembly process. All tools can be found in the tool area. The assembly area includes a driverless transport system (DTS) mounted with a six-cylinder engine. We used an assembly training task with 15 steps following production specification. The training contains low complexity tasks such as screwing a lifting eyebolt but also high complexity tasks such as installing, screwing and plugging a harness.
Through a public invitation, we acquired five office employees, five assembly employees as well as five ARexperts in preparation of our study. All participants were informed in advance to bring safety boots and safety gloves. We initially made all participants familiar with the environment since the test environment and the assembly task was new for every participant. At first, we explained the assistive system and informed every participant that their participation is voluntary. We further told them to inform us whenever they feel uncomfortable so we can abort the experiment immediately. Afterwards, we explained the purpose of the usability study. After explaining the ambition of the experiment, we measured and adjusted the user’s interpupillary distance (IPD) which is important for the visual quality. Holograms may appear unstable or at an incorrect distance when using an incorrect IPD. We showed how to adjust the HoloLens and started with the Microsoft Learn Gestures Application to familiarize our participants with the interaction modalities. Once the participants felt confident using the HMD, we kindly asked our users to start our training application. All participants were asked to complete the tutorial level at first, continuing with the beginner, intermediate and expert levels. Users had to work through 15 assembly steps in each level. The assembly sequence between the levels was not modified. Only the provided information were reduced. Between each level, we disassembled the engine back to its initial state. During that time, participants were asked to have a 10 minutes break. We offered various sweets and softdrinks to generate a pleasant break. During the study we measured the time for each level, assembly and picking mistakes, self-corrected mistakes, corrections by help as well as the backward fading questions. To measure the training time between each assembly we paired our application with a database using a WiFi internet connection. This approach guarantees a reliable data collection. After finishing the fourth and last level, participants were asked to rate the training software using three established questionnaires. We used an extended SUS to evaluate the usability of our software. Participants had to finish a 10 item questionnaire using a five options Likert scale ranging from strongly agree to strongly disagree. The second questionnaire (AttrakDiff) aims to determine the pragmatic and hedonic quality. The questionnaire was finished with the NASA-TLX to assess the cognitive workload during the training.
We invited 15 participants (13 male, 2 female) for our user study following Nielsen who recommends performing a usability study using three groups with five users each [16]. The participants were aged from 21 to 42 (M = 30.06; SD = 6.20). Five of them were office employees, five were assembly employees working in the BMW Group production and five were AR-experts with at least 5 years background in AR. We asked each group for their AR and assembly background using a five item Likert scale ranging from much experience to few experience. Much experience were scored with 4 points, few experience with 0 points. Office employees stated to have no background in AR (M= 0.20; SD = 0.40) and medium experience with assembly processes (M= 1.60; SD = 1.35). The assembly workers had a strong background in assembly processes since it’s their daily routine (M = 3.80; SD= 0.40) but their knowledge about AR was limited (M = 0.40; SD = 0.49). In contrast to that, all AR-experts stated to have a strong background in AR (M = 4.00; SD = 0.00) and medium experience with assembly processes (M= 2.00; SD = 0.63). All participants were capable to understand, read and write the German language since the entire auditory instructions provided by our software as well as the questionnaires were in German.
There was no significant difference between the
assembly training times (Table 3). The Shapiro Wilk
Test showed a normal distribution for TLCT, ILCT and
ELCT and non normal distribution for BLCT (p=.02). We
used a one way ANOVA which showed no a statistically
significant difference for the TLCT (F(
During the study, all three groups made a few errors
(Table 1; 2) but there was so significant difference
between the groups. The Shapiro Wilk Test did show a
non-normal distribution for all variables (AM, PM, CAM,
CPM, CBHA, CBHP). We used the Kruskall Wallis Test to
find difference between the variables but the test
showed no significant difference for AM (χ² (
We used the NASA-TLX to measure the mental
workload during the experiment but there was no
significant difference for the six subscales between the
groups. The Shapiro Wilk Test did show a non-normal
distribution for PW (p=.029) and TWL (p=.038). The
Kruskall Wallis Test showed no significant difference for
PW (χ²(
The results from the SUS also showed no significant
difference (F(
Based on the experiences we gained from using our HMD-based application with different user groups, we propose 10 recommendations for designing HMD-based assembly training tasks. We also believe that these guidelines are generic and easily transferable to other procedural tasks and different domains (e.g. learning a surgery procedure) which could help designers and researchers to create more meaningful applications.
We highly recommend to use a simple, clear understandable, consistent application design. Low complexity designs and uniform colors help to reduce the cognitive workload which improves the training transfer. However, visual complexity increases the brain activity and therefore the cognitive workload which harms the procedural memory.
Guiding a user step-by-step through a procedural task improves the initial performance but may have an adverse effect on the training transfer. Users might not be able to repeat a task without having the support from an assistive system. Instead of just guiding a user through a task, they should have full control of the application. Users are able to activate and deactivate different features, switching between next and previous assembly steps as well as different levels of difficulty at any time, when using our software. This allows users to work self-paced without feeling like a robot.
Humans are used to multimodal feedback since it is provided by a lot of technical systems in our daily lives (e.g. visual and sound feedback is provided when pressing a button in an elevator). Adapting familiar feedback approaches to new technologies such as HMDs helps to familiarize new users in a shorter amount of time. Furthermore, the combination of different modalities such as visual, auditory or tactile feedback can improve the learning transfer through stimulating various human information channels.
Providing a wide range of information at the beginning of a learning process helps novice users gather essential information of the task. Our evaluation revealed that once users completed the tutorial level, they become much more familiar in using the software modalities as well as performing basic movements which indicates the time improvement in Table 3. At this point, information should be gradually reduced by not frustrating users with too much unnecessary information. We recommend offering different user modes ranging from a tutorial to an expert mode using various amounts of information. This concept allows completely novice employees as well as experienced users to use a software product in accordance to their skill level.
Executing manual procedural task requires hands-free usage which is usually realized when using a HMD. Additionally, developers should take into account that extra effort as well as limitations in performing a task should be avoided when designing interaction concepts. Therefore, we suggest using voice interaction due to two facts. Most of the time, users carry parts and tools when performing assembly tasks. They should be able to interact with the HMD without using their hands which can be realized using voice interaction. Through our study, we also learned that gesture interaction, especially the HoloLens Airtap was hard for many participants since it’s an unnatural movement. More natural and intuitive gesture interaction concepts may help to overcome this limitation in future [22].
Adding context information can help users to build and strengthen a global picture of a task. Having a strong mental model of a procedure allows someone to perform a task efficiently without requiring support from an assistive system (e.g. HMD). Due to our user study, we recommend using progress bars for visualizing step-by-step progresses as well as when implementing video players to present the length of a video. A complete task overview was also found to be helpful by our participants for building a global picture of the assembly training task.
Assembly processes are often boring and monotonous for many employees. When it comes to learning new procedures, the majority are willing to acquire new knowledge but they wish more fun during the training since it takes a lot of time and is very serious. Gamebased learning approaches which are already widespread in schools for teaching kids might help to improve assembly trainings. Previous studies already stated that providing self-quantified information such as errors and time, combined with gamification elements can enhance work processes [23]. Our participants revealed that Embly is cute and motivates them to finish the assembly training successfully. Designing our application according to a game was also found to be very enjoyable by our participants. We also believe that the attractiveness of assembly jobs can be increased for younger people when providing innovative technologies such as HMDs in combination with gamebased learning applications.
When analyzing the click rates of our application, we found differences between users and specific user groups (Table 4). Most of our participants tend to use traditional, familiar media such as watching a video or reading a text. On the other side, some participants also preferred using three-dimensional content. Therefore, providing the opportunity to choose between different types of information will help many users finish an assembly task successfully. Additionally, offering multimodal information can enhance the training transfer.
People are familiar working with multilayered mobile phone applications. Each layer contains different information and functionalities. Taking this into account when developing applications for HMDs can help novice users to become adjusted to a new software and technology very fast. We adapted this approach and designed a clean multilayered software for assembly training tasks. Only the UI (Figure 1) is visualized permanently, all other functionalities and information are hidden under sublayers, selectable on demand. We recommend using this concept since all of our participants liked it.
Superimposed three-dimensional content supports trainees in learning the position and orientation of a specific part as well as improves the spatial perception. Providing additional animations not only helps to understand what and where to assemble, it also shows how to assemble a specific part. Therefore, we recommend using different 3D augmented reality content. We further suggest using an outline shader when presenting animated 3D parts. While watching a looped animation of the assembly process, users are able to install the part simultaneously. Rendering only full-color shader parts while wearing a HMD might bother users executing the assembly process because real parts are difficult to recognize. Users might tend to watch underneath the HMD to accomplish the assembly process.
In this paper, we introduced a novel concept for assembly training task using HMDs. According to the requirements for industrial killer applications introduced by Navab [24], we build a reliable, scalable, user friendly killer application for real engine assembly training tasks and described every feature in detail. An experiment with 15 participants, divided into three groups with different skill levels, were executed to evaluate the usability of the software. Results regarding effectivity, efficiency and user satisfaction variables showed no significant differences between the three groups. One reason for that might be the low number of participants. Therefore, we argue that everyone can use this application, no matter which skill level or knowledge background someone has. Due to that fact, we have created a standard tool for assembly training task which ensures a consistent educational result. All participant enjoyed using the HMD-based application proven by the high SUS of 90.5 and the result from the AttrakkDiff. Based on the experiences during the evaluation, we proposed 10 design recommendations for HMD-based assembly training. These principles can be adopted by other researches to create a successful training application. Considering future work, we want to evaluate our application with a larger amount of participants by measuring the trainer transfer.
We thank all participants for their unaffordable input and BMW, for helping and providing with the tools, spaces and funding to realize and complete this project successfully.
In: Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06), S. 22c-22c. DOI: 10.1109/HICSS.2006.476.