RoboCup is an International robotics initiative whose aim is to promote robotics and AI research. RoboCup's long-term goal is to create a fully autonomous humanoid robot team capable of competing and winning a soccer game against the human World champion team, in compliance with the oficial rules of FIFA, by 2050. In this paper, we describe a two-step method for action recognition. In the ifrst step, we extract the pose of the robots using a pose detector trained on a novel dataset for pose estimation called UNIBAS NAO Pose Dataset, which is a contribution of this work. In the second step, a Spatial-Temporal Graph Convolutional Network is used for modeling the gameplay, with particular regard to fall-down detection. Experimental results show the efectiveness of our approach in detecting falls for humanoid robots.
Since its foundation in 1996, one of the objectives of RoboCup was to push the boundaries of
research by ofering high-level challenges. In 2022, the RoboCup Standard Platform League
(concerning NAO robots playing soccer 5 vs 5) has proposed an Open Research Challenge [
UNIBAS Wolves and SPQR Team participated in the Open Research Challenge presenting
MARIO, an end-to-end architecture for computing visual statistics in RoboCup SPL [
The remainder of the paper is organized as follows. Section 2 contains a description of existing methods for extracting statistics from soccer matches. Section 3 presents the method for detecting the robot fall-down action, including the description of the dataset used for training. Quantitative experimental results are shown in Section 4. Finally, conclusions are drawn in Section 5.
2. Examining the team’s collective performance; 3. Calculating useful statistics designed to validate game schemes.
The goal of the analysis is to collect data to improve the quality of play so as to increase the chance of winning. At the basis of conducting a video analysis of soccer matches, it is essential 1available at https://drive.google.com/drive/folders/1wY9Xsz3O_gYc4BbGb4p_gALotynjcH-E?usp=sharing to identify the athlete, and by employing increasingly advanced technologies, we have gone as far as identifying the type of motion and detecting interactions with opponents.
Video analysis approaches can be grouped in two main categories. The first group includes commercial systems that are semi-automatic, meaning that the labeling of players is entered manually through user interaction. Semi-automatic commercial systems are used, for example, by TVs or professional teams to observe game situations and actions that occurred during matches. The second group is made of commercial and academic systems that are designed toward eliminating user interaction.
Semi-automatic systems include Piero [
The use of supervised algorithms is a common characteristic of fully-automatic approaches.
For training those systems it is crucial to collect a large dataset containing properly annotated
samples. For example, SoccerNet [
In this work, we present MARIO [
2. A graph convolutional neural network for detecting the movement.
Before describing our approach, it is convenient to introduce our dataset for NAO robot pose
estimation, called UNIBAS NAO Pose Dataset [
NAO are humanoid robots, however, there are some diferences between the NAO structure and the human one. Thus, we have created a specific dataset for detecting the pose of a NAO robot.
In particular, we collected video frames from the RoboCup Standard Platform League (SPL)
teams. Using the COCO Annotator tool [
To detect the pose of the NAOs we used a slightly optimized version of OpenPose, called
Lightweight OpenPose [
To produce a new estimation of keypoint heatmaps and Part Afinity Fields (PAFs), the refinement stage takes features from the backbone concatenated with the previous estimation of the keypoint heatmaps and PAFs. In order to share the computation between heatmaps and PAFs, a single prediction branch is used in the initial and refinement stage. All the layers are shared except for the last two, which directly produce keypoint heatmaps and PAFs.
To capture long-range spatial dependencies, each convolution with 7×7 kernel size is replaced by a convolutional block with the same receptive field. This block design allows having three consecutive convolutions with 1×1, 3×3, and 3×3 kernel size convolutional layers. To preserve the initial receptive field the last one has a dilation parameter equal to 2. Finally, a residual connection is used for speeding up the network. The final network has ∼2.5 times less complexity than convolution with 7×7 kernel.
The detection of the fall-down action is carried out by using a Spatial-Temporal Graph Convolutional Network (ST-GCN). The ST-GCN model includes 9 layers of spatial-temporal graph convolution operators, namely ST-GCN units (see Fig. 3).
The output size is 64 for the first three layers, 128 for the following three layers, and 256 for the last three layers. A dropout layer with a probability of 0.5 is present after each ST-GCN unit to prevent overfitting. The strides of the fourth and seventh temporal convolution layers are set to 2 as pooling layers. After the 9 ST-GCN units, a global pooling is performed on the resulting tensor to get a 256-dimension feature vector for each sequence. Then, the 256-feature vector is fed to a softmax classifier. Stochastic gradient descent with a learning rate of 0.01 is used as optimization method. The learning rate is programmed to decay by 0.1 every 10 epochs.
ST-GCN takes as input the output produced by the pose estimator. Since the trained ST-GCN requires 14 skeleton key points, the 18 key points coming from the NAO pose estimator are modified by removing the key points representing ears and eyes. The input for the ST-GCN model is a sequence of 30 skeletons. It is worth noticing that a skeleton is not included in the sequence if the number of key points returned by Lightweight OpenPose does not reach 80% of the expected total (see Fig. 4).
The ST-GCN model outputs the action class with the related probability score. If the
probability score is greater than 50%, the action class produced by the model is considered valid,
otherwise, it is rejected. For training the model, we used the Le2i data set proposed by Charfi et
al. [
The performance of the proposed approach has been measured by considering separately the pose estimation module and the fall detection system.
The overall performance of the system is good. However, there are some errors. The accuracy percentage of 70% in detecting the correct key points might be due either to the presence of occlusions, which means that the robots are not correctly recognized, or an unusual robot pose that is not included in the data set. In some cases, the robots are not completely facing the camera, but instead are placed to the side of the camera.
The camera position and the robot position also afect the ST-GCN system for fall-down detection. Such a problem is related to the training data set, where the annotations are taken from a diferent point of view with respect to the one used during the RoboCup matches.
We have presented an approach for detecting the fall-down action for NAO in RoboCup SPL matches. In particular, the approach is based on two steps: 1) the estimation of the NAO pose by using Lightweight OpenPose and 2) the detection of the fall-down action by using a Spatial-Temporal Graph Convolutional Network (ST-GCN).
For evaluating the pose estimation performance, we have conducted quantitative experiments using a novel data set called UNIBAS NAO Pose Dataset. For training the ST-GCN, we used publicly available data from Le2i dataset, while for testing it we considered a set of labeled videos of Robocup matches. The proposed method proves its efectiveness in detecting the fall-down action and provides a few false positives due to the camera position and the occlusions generated by the other robots or the referee.
As future work, we intend to create a new data set for NAO action recognition and we want to increase the number of actions recognized by the system.