A basketball player's ability to respond to situations is an important factor that greatly influences the outcome of a game. In this study, we propose a method to evaluate the situational ability of basketball players using a VR simulator. The VR simulator replicates a 2-on-2 game scenario, and the system captures the players' line of sight and movements. The Head-Mounted Display (HMD) records the gaze, while the movement is tracked by capturing images of the player and estimating the player's skeletal structure using OpenPose. We then propose and analyze an index, derived from the collected data, that represents the player's situational responsiveness. From the collected data, we propose and analyze indicators of situational responsiveness. The indicators include gazing objects, number of head turns, body movement, movement trend, and average reaction time. In the experiment, we showed that the proposed indexes can be used to objectively evaluate the players' ability to respond to situations by analyzing them. This study was conducted to evaluate the complexity of a basketball player using a VR simulator. This research introduces and validates an innovative method for evaluating basketball players' ability to respond to complex situations using a VR simulator. We believe that this research has the potential to make a significant contribution to the field of sports analysis.
Although the ideal scenario would be to prepare an ex- The data to be obtained are described first. The data perience equivalent to an actual game, in order to simplify to be obtained are two types of data: gaze information the factors to be considered when devising the indicators, and body movement information, respectively. For the we decided to adopt a two-on-two scenario after discus- gaze information, the player’s gaze vector, the object that sions with the coach and the manager of the University collided with the gaze vector for the first time, and the of Tsukuba basketball team. direction of the player’s face are recorded. In addition,
The eye movement of the players is measured using the the player’s joint coordinates are obtained by skeletal
position and posture measurement function of the head- estimation using OpenPose on the body video recorded
mounted display used for the VR simulator and its inter- by an external camera. Based on the above information,
nal eye-tracking function. The body movements of the we propose indicators and evaluate the players.
players are recorded by an external camera and analyzed
later; skeletal information is estimated by OpenPose[
Hughes et al.[
There have been reports of VR simulators for table ple and objects are investigated by recording the players’
tennis training[
evaluate the situational competence of basketball players.
no review with the same authors in [
OpenPose to estimate the skeleton from images recorded the speed of information processing and the time it takes by an external camera in order to analyze the motion. The to respond to complex situations by comparing the data joint coordinates are obtained from the estimation, and of each player. If the situation is relatively easy to judge, the distance between the camera and the player can be such as an opponent’s pass, this indicator corresponds calculated using the epipolar constraint method. Thus, it to the reaction time of the players. By comparing with is possible to calculate the distance between each joint of the reaction time, we can evaluate the time required for the player in a plane parallel to the camera’s image plane. the player to understand a complex situation. Details of the use of the epipolar constraint method and the calculation of the distances between joints are described in Chapter 5. In addition, visualization of the 4. VR Simulator distances between joints with respect to time and visualization of the rate of change of the distances between 4.1. Hardware joints with respect to time are provided. With these two Information on the HMD used in this research is deitems, it is possible to evaluate "body movement" and scribed below. A head-mounted display (HMD) called "intensity of movement." Body movement allows us to VIVE PRO EYE will be released by HTC in 2019. It has a evaluate when and how much a player raises his hands, high image quality and a viewing angle of 110 degrees whether he has suficient defensive intention against an and can present a clear first-person view with a 360opponent’s action, and whether he makes a quick deci- degree perspective. With a total of 2880 horizontal pixels sion against an opponent’s feint. The intensity of body and 1600 vertical pixels, users can experience an image movement can also be analyzed in terms of a player’s with a screen resolution of 1440 x 1600 viewed through instantaneous power, tfiness management, tactical ten- one eye. The built-in gyro sensor and BASE STATION2 dencies, and tendency to close of all possible penetration provide head tracking, while VIVE PRO EYE features eye routes, or to focus on key points with good timing. tracking through eye rotation.
The head positions of players are recorded by the HMD, A Go Pro Hero 5 external camera is used to capture and by visualizing the data, the "movement tendencies" the players’ movements. The horizontal, vertical, and of the players can be evaluated. Since the position of diagonal degrees of the Field of View (FOV) were 69.5 the players in a game has a great influence on tactical degrees, 118.2 degrees, and 133.6 degrees, respectively. decisions and the situation of the game, it is necessary to analyze the movement tendencies of the players for each situation. The movement trajectories of the players with 4.2. Software respect to time make it possible to analyze the tendency of players to move forward to help or to back up and defend their own marks in response to a known scenario.
The recording of the movement trajectories of the players in the diferent scenarios also allows the evaluation of how the players handle each situation and how proactive they are in their tactical decisions.
Technology, is used to create the 3D virtual space. Steam VR is used for this research to project the scenario on the HMD and to realize real-time monitoring during the experiment.
An indoor basketball court is constructed in a 3D
virtual space to simulate a game. Figure 1 shows an overall
view of the basketball court, and Figure 2 shows a
bird’s3.3. Indicators using gaze and body eye view. Figure 3 shows a view of the goal ring from
information the center of the gymnasium, and Figure 4 shows a view
of the goal ring from the side. The area of the
basketThe index "average situation analysis time" refers to the ball court is the same as the oficial competition court,
average time it takes an athlete from the moment he or 28 × 15. The court is measured from the
she sees a situation to the moment he or she gathers in- inside and all lines are 50 wide. The height of the
formation, thinks and analyzes, and starts to move. It has goal ring is set to the oficial height of 305.
been pointed out that neuroreflex training has a signifi- The NPCs used in the scenario are shown in Figure 5.
cant impact on the acquisition of skills in sports[
By setting up the behavior of the 3DCG objects in advance, it is possible to reproduce the flow of a match. The system is set up in such a way that the players can watch the game situation in the first person while we can see the situation as the players see it in real-time. In order for the players to have a match experience that is close to reality, and for their cognition and actions to be as realistic as they actually are, we believe that the immersion of the experience in the scenario we have prepared is necessary. The space should be secured so that players can move at their own discretion and can move their arms and legs
eye rotation information to estimate gaze. The builtin gyro sensor and two BASE STATION2s are used to track the head. Figure 7 shows the line of sight of an athlete in the virtual space, indicated by a red line. The white and green lines representing the players’ visual
Figure 7: Image diagram of line of sight vector, visual field moves against the opponent’s actions are captured by an range, and face orientation vector. external camera. Since the experiment participants put on HMD, the original OpenPose sometimes fails to find the head part. We have trained the original OpenPose model additionally to find the head wearing the HMD. ifeld range have an angle of 110 degrees. The blue line in We see that our refined OpenPose model can estimate the middle represents the player’s face orientation vector. the head wearing an HMD after the training in Figure 10. Figure 8 shows an image of the game from a first-person In this research, since the head can be accurately meaperspective with the players wearing HMD. sured by the HMD system, the position and posture infor
A collision detection attribute is added to all objects in mation of the head can be used to represent the OpenPose the virtual space, and the collision with the line-of-sight posture estimation results in three-dimensional space. vector is used as the criterion for determining the object The position and posture of the external camera are meato be gazed at. The collision coordinates are recorded and sured in advance. called the gazing point. With the above data, it is possi- Since the players may change their body orientation ble to reproduce the motion of the player’s head during during the scenario, the location of the external camera his/her experience. When reproducing the player’s expe- is determined in advance according to the play scenario rience, a sphere is rendered at the gazing point. When so that posture estimation is possible in any orientation. the player’s gazing vector collides with the basketball, In addition, when the player extends his hand to the a pink sphere is rendered, otherwise, a blue sphere is ofensive side of the field, the scenario design will take rendered. An image of the collision rendering is shown into consideration that the hand should be extended to in Figure 9. the left or right as seen from the external camera.
Under the above conditions, the epipolar constraint for the player’s head is shown by the red dotted line in [before training] [after training]
Figure 11(a). 6.1. Reasons for determining the scenario
Since the head position is recorded by the HMD and
the position information of the external camera is pre- The scenario used in this research was created
considermeasured, the distance between the camera and the plane ing the following factors.
perpendicular to the camera optical axis can be calcu- A monocular camera is used to record the movement
lated from the distance between the head and the camera. of the person to be experienced and collect skeletal
inforTherefore, the horizontal and vertical lengths of the im- mation. To obtain the most efective estimation results,
age plane captured by the camera can be calculated from the scenario should be determined by filming from the
the FOV of the camera at the position of the subject. front and taking into account the efect on the orientation
As shown in Figure 11(b), based on the above informa- of the experiment participants. Factors that influence the
tion, the distance between joints can be calculated using experiment participants’s judgment are minimized so
the joint information from the OpenPose estimation re- that the experiment participants’s ability to judge the
sults, making it possible to evaluate the participant’s limb situation can be correctly ascertained while maximizing
movements. the reproduction of the in-game situation[
The skeletal information and inter-articular distance reported to be likely to be beneficial for decision making in the plane perpendicular to the camera optical axis can in a sports environment with high levels of interference now be measured spatially. and suficient self-control. Therefore, the scenario used in this research simulates a two-on-two match.
In order for the experiment participants to experience a realistic match and to be able to make decisions that are closer to reality, it is considered necessary to make the experience of the scenario more immersive. For this reason, the movements of the NPCs in the scenario are animated using actual motion-captured movements, and the player. the basketball court uses assets that resemble a Japanese Scenario 5 evaluates the tendency of players to move school basketball court. and their ability to judge the situation in response to an opponent’s feint. After the game starts, Red 1 dribbles 6.2. Composition of the situation and feints to pass the ball to Red 2. Red 1 then briefly changes direction and dribbles to the right of the goal We carefully designed five scenarios in total in the game ring. Red 2 simultaneously crosses the three-point line form of 2 vs. 2. The experiment participant is a member of in the opposite direction and approaches the goal ring. the blue team at defense, and the two red team members After dribbling, Red 1 feints a shot and passes the pole to attack. The three players are NPCs. The NPC that the Red 2. Red 2 receives the ball and shoots. In this scenario, experiment participant should deal with is called Red there are two feints, and the player’s reaction to them 2, and the other player on the Red team is called Red 1. can be examined. The tactical decisions of the players The NPC on the Blue team who plays a defensive role can also be investigated in response to the actions of Red against Red 1 is called Blue 1. Red 1 and Blue 1 first stand 2. near the center of the basketball court, near the three point line. The player who experienced the game and 6.3. Acquisition of eye gaze information Red 2 stand in the center of the basketball field, near the three-point line on the left side facing the goal ring. An by VIVE EYE image is shown in Figure 7. The game in all scenarios In order to analyze the eye movements of players as they starts when Red 1 passes the ball to Blue 1 and Blue 1 respond to complex situations, we describe a method for throws the ball back to Red 1. The following sections estimating their field of view and point of gaze. Based describe the contents of each scenario. on the face direction vector from the head rotation in
Scenario 1 is created to evaluate the players’ tendency formation, the athlete’s visual field boundary is a line to gather information. After the game starts, Red 1 drib- with a 55-degree left-right opening. As shown in Figure bles and then shoots. Red 2 is within the view range of 1, the player’s visual field boundaries can be visualized the observer. In this scenario, the player pays attention to by placing the white and green boundary lines on the left Red 1, looks between Red 1 and Red 2 and pays attention and right sides of the player’s face, centered on the blue to both, or pays attention mainly to Red 2, and so on. face vector, starting from the measured position of the
Scenario 2 is to be designed so that the reaction time of player’s head. Similarly, the gaze vectors shown in red the players can be evaluated. After the game starts, Red and the points of collision with the gaze vectors can also 1 dribbles and then passes the ball to Red 2, who shoots. be visualized. The gazing points can be visualized by placThe time from the moment the player sees Red 1 pass the ing spherical objects at the coordinates of the recorded ball to the moment he starts a defensive move is used to gazing points. In Figure 2, multiple blue balls represent evaluate the reaction time of the player. the fact that many of the gazing points are located near
In Scenario 3, the setting of the situation is considered the head of the red-clad player on the right side. so that the players can evaluate the response to the situation when there are members of the opposing team 6.4. Skeletal Information Estimation with outside of their field of vision. After the game starts, Red 1 dribbles and passes the ball to Red 2, who shoots. OpenPose However, Red 2 is outside the viewer’s field of vision. In The proposed system uses OpenPose to estimate skeletal this situation, if the experiment participants do not pay information from video captured by an external camera attention to Red 2, he/she will be hindered in judging the and introduces a method to determine the position of situation, and thus the balance of attention to the two the experiment participants by combining the estimated players on the Red team can be observed. head position and position information recorded by the
Scenario 4 evaluates the movement tendencies of the HMD. players and their ability to judge the situation in response However, the original trained model of OpenPose ofto an opponent’s feint. After the game starts, Red 1 drib- ten fails to estimate the head of the person wearing the bles and feints to pass the ball to Red 2. Red 1 then briefly HMD, and if the head estimation fails, the aforemenchanges direction and dribbles to the right of the goal tioned epipolar constraint cannot be used. We used the ring. Blue 1 chases after Red 1 and Red 2 goes for help. extended OpenPose to perform additional training of Finally, Red 1 shoots. The video allows the player to OpenPose on the image dataset of the HMD wearer. The evaluate the presence or absence of defensive action in dataset was created by photographing the subjects in response to a feint pass, and the time required to deter- various backgrounds and clothing, taking into account mine that it is a feint can also be analyzed. The tendency the possibility that the accuracy of the model may be to go for help can also be evaluated by the movement of afected by diferences in lightness, darkness, and angle.
participants try it on as a test. Next, we played scenario 1 for practice in order for the participants to confirm the area in the VR virtual space in which they can engage in activities and to understand the behavior of the system. After the scenario is finished, we let the participants experience a range of movement of about three steps with the VR gymnasium displayed.
Explain the simulation experience and the number of attempts made during the simulation experience. Request the red team to defend against the red team’s ofense. During the experiment, an external camera will be used to record the experimenter playing each of the five scenarios described in section 6.2 four times, for a total of 20 scenarios. Make sure that there is no sequential efect when playing the scenarios. Explain that after each play is completed, the experiment participants should return to the initial position.
Twenty-five members of the University of Tsukuba men’s basketball team, ranging in age from 19 to 21, participated in the experiment as participants 1 through 25. All participants had previous basketball experience.
First, the participants were given an overview of the study, the eye gaze data to be acquired, and the fact that the data would be recorded by an external camera. Next, we explain that the range of movement for the participants during the experiment is a circle with a radius of 2 meters. To allow participants to confirm the range of movement, the range of movement is taped on the ground in advance. Figure 12 shows the situation during the experiment. ticipant has two choices. One is to keep shaking his head and always collect information about the two players.
The other is to retreat and keep a distance so that Red 1 and Red 2 are visible at the same time. Participants who chose to turn their heads without retreating tended to focus on the players, whereas participants who retreated and looked at the two red team players simultaneously tended to look between the two red team players. From this figure, it can be analyzed that participants 3, 5, and 8 pay particular attention to the players. Participants 1, 9, and 10 are the ones who look at the gym most often.
These participants are considered to be looking between two ofensive players.
Next, Table 1 shows the average number of times participants looked at the NPCs in Scenario 2. Here we take the data of participants 3, 5, and 8, who tend to gaze at the red team, and participants 1, 9, and 10, who look between the ofensive players. From this table, we can see that the participants collected information better from Red 1. In addition, participants 3, 5, and 8 gazed at the players more frequently than participants 1, 9, and 10. the results of subtracting the values in Table 2 from Table 1.
In scenario 3, Red 2 is outside the participant’s field of view. In this experiment, we observed three methods of gathering information from the participants. The first was to collect information by repeatedly shaking their heads, the second was to retreat so that they could see both members of the red team, and the third was to look at Red 2 only when necessary and focus mainly on Red 1, who had the ball from the beginning. Participants 3, 5 and 8 chose the method of gathering information while shaking their heads, while participants 9 and 10 chose to retreat. Participant 1 chose the third method, "Look at Red 2 only when necessary. The change from Scenario 2 to Scenario 3 revealed that Participant 5 increased the number of times he looked at Red 1 and Red 2 in particular.
Participant 8 also increased the number of times he gazed at Red 1. In contrast, participants 9 and 10 did not change significantly. Participant 1 gazed at Red 1 less frequently and paid more attention to Red 2 in contrast to Red 1.
The reason for this may be that the players increased the number of times they gazed at Red 2 in response to the longer time they spent gazing at Red 1.
Figure 15 shows the average time participants took each time they switched their visual targets in Scenarios 4 and 5. This figure shows that Participant 1 was par
Table 2 shows the average number of times partici- ticularly quick to switch gazing targets, and the results pants saw the NPC in scenario 3. There is a diference in Table 3 suggest that Participant 1 only captured the in the scenarios between Table 1 and Table 2 in terms of minimum necessary amount of information for Red 2. In whether Red 2 and Red 1 are visible at the same time, and contrast, Participant 9’s style of play was to stop lookby comparing the two tables, it is possible to analyze the ing between Red 1 and Red 2 each time he switched his players’ information gathering methods. Table 3 shows gaze to Red 1 and Red 2, and to collect more information.
Participants 8 and 10 balanced themselves, and changing attention when the opponent moves, such as passing or targets after grasping information from one person was dashing. Participants 9 and 10 adjust their own positions considered to be the reason why it took them longer to do to gather information during the game and tend to keep so. We also observed participant 5 in a replicated game both ofensive players in view, but participant 10 can be situation, and the reason why he took longer to change analyzed as a balancing type who pays more attention targets compared to the other participants was thought to the space between players than participant 9. to be due to his lower head rotation speed.
These results allow us to analyze the characteristics of the information gathering methods of participants 1, 3, 5, 8, 9 and 10. Participant 1 pays attention to the area around Red 1, who has the ball, and pays attention to obtain at least the minimum information about the location of Red 2 when necessary. Participants 3, 5, and 8 move their eyes to keep track of the surroundings and pay
Figure 16 shows an image of the distance between each participant’s joints. On the left is the experiment and on the right is a bar chart of the distances between the joints. The horizontal axis is the part to be measured and the vertical axis is the corresponding length. By continuing to record such data, the rate of change of the inter-articular distances can be determined and the intensity of the players’ movements can be evaluated.
Figure 17 shows the distance between the joints of participant 22 for each frame in Scenario 3. The vertical axis is the distance between the joints for the horizontal frame. The reason for the longer distance between the joints in the later frames compared to the first frame can be attributed to the fact that the player initially assumed a defensive posture against Red 1, but Red 1 passed the ball to Red 2, who then made a defensive move against the shot by Red 2.
Previous studies have shown the potential for players to
reproduce their route choices and analyze their tactical
decisions during a match[
Figure 20 and Figure 21 show the distance between Red 1 and Red 2 during a single play of Participant 15 in Scenarios 2 and 3. It shows the tactic of participant 15 keeping his distance when he cannot see Red 1 and Red 2 at the same time and approaching to defend as soon as his mark receives the ball, whereas Red 2 is within his ifeld of view in scenario 2.
Figure 22 shows the average situational analysis time of
the participants in Scenarios 4 and 5. Note that the
average situation analysis time refers to the time between
when a player sees an opponent’s pass or feint and when
his body begins to move. It was noted that a human needs
about 150 ms to grasp the image he sees. [
to provide the same experience as a real game, and participants’ eye gaze and body information were collected with the aim of evaluating players’ ability to respond to situations using a number of indicators. The aforementioned indicators were used to analyze the players’ willingness to defend and help, and the tendency to move and the tendency to make tactical decisions were evaluated with the information collected from their gaze. Therefore, it can be said that the method used in this study can be used to evaluate the players’ ability to respond to situations.
In this experiment to evaluate situational readiness, we did not analyze the complete three-dimensional posture of the participants, and there were cases in which decisions could not be made when the limbs were in front of or behind the head. It will be necessary to verify the efectiveness of the method of evaluating the complete three-dimensional posture with multiple cameras in order to fully understand the body information of the players.
experiment based on the proposed index.
We invented a new index and an evaluation method to evaluate players’ ability to respond to situations based on their gaze information and body movements. We constructed a system that can reproduce a realistic game situation using a VR simulator so that players can make decisions that are equivalent to those they would make in a game. We prepared scenarios to evaluate the players’ ability to respond to situations.
The evaluation experiments were conducted, data was acquired by the proposed system, and trends in player evaluations and behavior were analyzed.
Part of this work was supported by JSPS KAKENHI 21H03476/23K21685.