In Industry 5.0, the interaction between humans and collaborative robots (cobots) is becoming more and more important for manufacturing processes [1]. Cobots represent a shift in robotic technology. Traditional robots typically operate in confined work cells or dedicated spaces having predefined and automated tasks. Unlike traditional robots, cobots operate in environments where they can interact directly with human workers to solve tasks that require a combination of human cognition and robot strength and repeatability.
In manufacturing processes, human action recognition and segmentation are crucial for many reasons: to promote human-robot cooperation [2]; to assist operators [3]; to support employee training [4,5]; to increase productivity and safety [6]; or to promote workers' good mental health [7]. In particular, the accurate recognition and segmentation of the actions, including the timing of when the actions commence and conclude, is essential for the cobot to understand and interpret the intended actions of the human collaborator, to synchronize its actions, respond in real-time, and ensure smooth cooperation with the human collaborator [8] [9].
Recently, the research has notably focused on using multimodal data, which can contribute to developing more sophisticated and adaptive action recognition systems. In particular, the information derived from skeletal joints enables researchers to capture temporal variations in body movements. It offers flexibility in focusing on the entire body or specific body parts, allowing for a comprehensive representation of the action recognition and bypassing eventual privacy concerns [10] [11].
To the best of the authors' knowledge, few visionbased datasets exist on human-cobot cooperation for object assembly in industrial manufacturing. For this reason, in the last few years, our research has been focused on the task of generating real datasets for practical applications of action recognition in the manufacturing context. The datasets for Human Action Multi-Modal Monitoring in Manufacturing (HA4M) and the Human-cobot collaboration for Action Recognition in Manufacturing Assembly (HARMA), consist of multimodal information acquired during the assembly of an Epicyclic Gear Train (EGT), depicted in Figure 1, without and with the collaboration of a cobot, respectively. The HA4M dataset was recorded using one single depth camera, while the HARMA dataset was recorded using two depth cameras. The Microsoft ® Azure Kinects have been selected as depth cameras in both cases.
The two proposed datasets present various main contributions compared to the existing ones [13,14] in the context of object assembly in industrial manufacturing:
• The datasets provide untrimmed sequences of several types of data: RGB frames, Depth maps, RGBto-depth-Aligned (RGB-A) frames, and Skeleton data. The availability of a variety of multi-modal data represents an added value for the scientific community to test different machine learning approaches in action segmentation as well as action recognition tasks, by using one or more data modalities. • The datasets present a variety in action execution due to the different order followed by the subjects to perform the actions and the interchangeable use of both hands. • The actions have a high granularity as the components to be assembled and the actions themselves appear visually similar. As a result, recognizing different actions is very challenging and requires a high level of context understanding and objecttracking skills. • Both datasets provide a good base for developing, validating, and testing techniques and methodologies for the recognition and segmentation of assembly actions.
Preliminary experiments have been conducted to test state-of-the-art temporal action segmentation methods, the ASFormer [15] and MS-TCN++ [16], on RGB and skeletal data achieving considerable accuracy rates in action segmentation.
The remainder of this paper is organized as follows: Section 2 presents the datasets and describes the assembly task, reporting details on the acquisition setup, study participants, and data annotation. Section 3 reports some experimental results on action segmentation. Finally, Section 4 delineates conclusive remarks.
The task involves the assembly of an Epicyclic Gear Train (EGT) (see Figure 1), which involves three phases: the assembly of Block 1, the assembly of Block 2, and then the completion of the EGT that makes up both blocks. The HA4M dataset contains videos of different operators that assemble the complete EGT. The HARMA dataset, instead, contains videos of different operators that assemble the EGT in collaboration with a cobot. All the subjects participated voluntarily in the experiments. They were asked to execute the task several times as preferred (e.g. with both hands), independently of their dominant hand. Furthermore, the subjects performed the task at their comfortable self-selected speed so that high time variance could be noticed among the different subjects. The subsequent sections give more details on both datasets.
The HA4M dataset contains 217 videos of the assembly task performed by 41 subjects. The acquisition setup is composed of a Microsoft Azure Kinect ® camera placed on a tripod in front of the operator as pictured in Fig. 2.
The camera is at a height of 1.54 𝑚 above the floor, at a horizontal distance of 1.78 𝑚 from the far border of the table, and is tilted down to an angle of 17°. As shown in Figure 2, the individual components to be assembled are spread on the table in front of the operator and are placed according to the order of assembly. The operator can pick up one component at a time to perform the assembly task standing in front of the table. The experiments took place in two laboratories: one in Italy and one in Spain. Two typical RGB frames captured by the camera in both laboratories are shown in Figure 3. The Figure also depicts the two supports fixed on the table to facilitate the assembly of Block 1 and Block 2. Tables 1 and 2 list the components and the actions necessary for assembling Block 1, Block 2, and the whole EGT, respectively. Notice that the final action (ID=12) involves additional tools, such as two screws and an Allen key to secure the EGT.
As listed in Table 2, the total number of actions is 12, divided as follows: four actions for building Block 1, four for building Block 2, and four for assembling the two blocks and completing the EGT. Some actions are performed more times as there are more components of the same type to be assembled: actions 2 and 3 are executed three times, while action 11 is repeated two times. Finally, a "don't care" action (ID=0) has been added to manage pauses between action transitions or unexpected events such as the loss of a component during the assembly.
The HARMA dataset comprises 160 videos (80 videos per camera) capturing the assembly task performed by 27 subjects in collaboration with a cobot (Fanuc CRX10ia/L robotic arm). Each subject performed the task multiple times, resulting in 240 task executions in the dataset. The acquisition setup is pictured in Fig. 4. The two Microsoft ® Azure Kinect cameras are placed on a tripod in Frontal and Lateral positions to the Operator Workplace. The Frontal Camera is at a height of 1.72 𝑚 above the floor and down tilted by an angle of 6 𝑑𝑒𝑔𝑟𝑒𝑒𝑠, while the Lateral Camera is at a height of 2.07 𝑚 and 19 𝑑𝑒𝑔𝑟𝑒𝑒𝑠 down tilted. Two typical RGB frames captured by both cameras are shown in Fig. 5. As shown in Fig. 5, the EGT components are spread over the Operator Workplace, so the operator can pick up one component at a time to perform the assembly task in seven pickand-place actions [14]. The operator assembles Block 1, whereas the cobot assembles Block 2. The assembly of Block 2 done by the cobot is not considered in the HARMA dataset, as our goal is to recognize the actions performed by the operator to trigger the cobot when it has to approach the operator to perform the collaboration action. So, the HARMA dataset comprises videos of only the assembly task performed by the subjects, including the collaborative action needed to join Block 1 and Block 2 (action 5 in Tab. 3). Table 3 lists the seven actions included in the HARMA dataset. As can be noticed in Table 3, unlike the HA4M dataset, the Cover is secured with two hooks (see Figure 6). 3.
This section presents preliminary experiments and results on temporal action segmentation by applying state- ASFormer [15] and MS-TCN++ [16] models have been applied to test action segmentation performance. The AS-Former (risp. the MSTCN++) models were fed using RGB and Skeletal data extracted from both datasets, performing the training over 120 (risp. 100) epochs, collecting losses for each iteration. The best model is chosen as the one with the lower loss within the total number of iterations and is used in the test phase.
Tab. 4 lists the performance rates in terms of Accuracy, Edit Score, and F1-score. Accuracy is a frame-wise metric that measures the proportion of correctly classified frames in the entire video sequence without capturing the temporal dependencies between action segments. The Edit Score, instead, measures how well the model predicts the ordering of action segmentation without requiring exact frame-level alignment. Finally, F1-score with a threshold 𝜏 , often denoted as F1@𝜏 , accounts for the degree of overlap between the Intersection over Union (IoU) of each predicted segment and ground truth segments [17]. In the experiments, the threshold 𝜏 has been set to 60%, 70% and 80%. Focusing on these metrics, it can be noticed that all the considered models succeeded in correctly segmenting the actions for the assembly task. In particular, the Accuracy rates reached high values (over 91%) in both cases of using RGB or skeletal features.
For completeness, Figure 7 shows a qualitative representation of action segmentation obtained by applying MS-TCN++on and ASFormer models to one video from the HA4M and one from the HARMA dataset. These shows better segmentation results also for actions 2 and 3. Furthermore, in the HARMA dataset, the availability of two cameras allows us to compensate for the lack of data when one camera fails to provide skeletal data due to occlusion or out of range [18].
The present paper depicted an examination of two industrial datasets, namely the Human Action Multi-Modal Monitoring in Manufacturing (HA4M), and the Humancobot collaboration for Action Recognition in Manufacturing (HARMA). Both datasets address the high demand for human action recognition and segmentation within industrial manufacturing contexts, particularly regard-ing scenarios involving Human-Robot collaboration and interaction. The multimodal features within the datasets encompass a variety of actions and interactions in industrial assembly tasks, allowing this work to lay the foundation for the development and enhancement of intelligent systems aiming at the understanding and assisting human operators in manufacturing production lines.
To properly evaluate HA4M and HARMA, state-of-theart temporal action segmentation models were considered, namely ASFormer and MS-TCN++, which demonstrated notable success in exploiting the data provided by the datasets. The comparison between the RGB and Skeletal features underlines the potential of a multimodal approach to balance the computational efficiency with the precision required for the recognition and segmentation of complex tasks.
The conducted experiments prove that, overall, both RGB and Skeletal features performed properly. RGB data provides rich visual information about the scene but typically requires higher storage space and computational complexity compared to skeleton-based data representation. On the other hand, by using skeleton data is possible to abstract away detailed appearance information and focus solely on the spatial configuration of body joints and movements. Therefore, it's essential to carefully find a good trade-off and select the data modality that best aligns with the goals and constraints of the working context.
The presented datasets are benchmarks for further studies in novel models and algorithms that can improve the accuracy and reliability of action recognition and segmentation systems in industrial settings. HA4M and HARMA offer a valuable resource for the research community, allowing ongoing innovation and development of human-robot collaboration systems in complex, realworld scenarios.
This research has been partly funded by PNRR -M4C2 -Investimento 1.3, Partenariato Esteso PE00000013 -"FAIR -Future Artificial Intelligence Research" -Spoke 8 "Pervasive AI", funded by the European Commission under the NextGeneration EU program.