This paper focuses on arm swing and the development of gait evaluation technology based on arm swing sensing data as a preliminary step to generate content that visualizes gait conditions in a VR environment. We extracted the characteristics of "arm swing height" and "forward and sideways swing" as important points of arm swing and developed an automatic detection method for each. For "arm swing height," the results of the automatic detection method developed for 11 of the 14 participants in the practice data were consistent with the experts' evaluations. For the "forward or sideways swing," two automatic detection methods were tested, and the results showed that each detection method matched the experts' evaluations for the different groups of features. In the future, we will expand the sensing methods to include not only arms but also legs and other parts of the body and improve the automatic evaluation methods.
Improved gait is an important factor in enhancing
quality of life and promoting health throughout life.
Studies by [
Meanwhile, advances in virtual reality (VR) technology will enable the development of new gait improvement programs. Behavior sensing in a VR environment can be easily performed regardless of the physical environment and can be a useful tool for understanding individual gait characteristics. Therefore, this study aims to make effective use of the VR environment.
In doing so, we focus on arm swing, as this can be easily
performed using existing VR controllers for data
acquisition. The case study [
Developments in virtual reality (VR) and sensing
technology have had a significant impact on the
reproduction and evaluation of gait. Previous studies
have taken the approach of reproducing the sensation of
walking using plantar vibration, vestibular sensation,
and tactile sensation [
Since VR controllers can acquire 6DoF data, there is a possible direction for gait evaluation utilizing angular velocity. Research cases such as estimating joint motion using angular velocity as in [7] or estimating lower limb motion as in [8] are useful for expanding the scope of this research. In addition, the case study of video-based gait analysis [9] demonstrates the usefulness of feature analysis with video.
Walking distance was 10 m because the walking test performed in the study [10] using VR during treadmill walking was 10 m.
Our study develops a new automatic evaluation method for gait improvement in a VR environment, referring to these previous studies, and provide a new approach to gait improvement.
To collect the basic data necessary for the initial study of VR sensing-based gait state evaluation, we conducted a data recording experiment of a scene in which a subject walked with a controller in a VR environment.
The study used Meta Quest Pro VR headsets and corresponding controllers for data collection. In addition, the program that enables data collection was developed by our collaborative research partner, the VR company Alpha Code [11]. This program can collect gait data from subjects by remote control
The data collected is in 6 Degrees of Freedom (6DoF) format. This covers six movements including forward/backward left/right tilt and rotation (3DoF) as well as forward/backward, up/down, and left/right movement. However, in this study, the analysis will focus specifically on the 3-D coordinate system.
The subjects of data collection were 14 healthy university students (12 males and 2 females) aged 20~23 years.
Subjects were required to walk in a straight line for 10 meters while wearing a VR headset and controller. A pass-through, a 10-meter walking guideline, and a console screen positioned in an unobstructed view are displayed on the VR screen as shown in Figure 1. During this time, the teleoperation program runs and 6DoF data is collected from the controller in the subject's hand. We also referred to the fact that the walking test conducted in a previous study [10] that used VR during walking on a treadmill in a previous case study was also 10 m and determined that 10 m was appropriate. During this process, video recording was also conducted using a smartphone (POT-LX2J) from a fixed frontal angle.
This study utilized the knowledge and insights of a coresearcher, an expert in gait instruction. The expert was asked to observe walking videos of 14 subjects wearing expert's ratings were used to design the rating scale for this study. As shown in Figure 2, the expert was presented with walking images and trajectories viewed from three directions, XY (front), YZ (side), and XZ (top), based on spatial coordinate data obtained from VR sensing. The expert carefully examined the data for all subjects, evaluated their gait state, focusing on arm swing, and commented on the characteristics of their gait state. After reviewing the comments for all subjects, "arm swing height" and "forward or sideways swing" were extracted as characteristics of arm swing that affect the evaluation of gait state. We developed an automatic judgment technique for these two viewpoints.
Assuming an environment in which an expert evaluates daily walking, we had the expert view only the subject's walking video and evaluate the fourteen subjects' walking conditions at four levels (very good, good, poor, very poor) based on the two perspectives extracted in Section 4. We designed, developed, and evaluated methods of automatic evaluation from two perspectives, with the four-step evaluation by the expert as the correct answer. By setting a representative value (R) and threestep threshold values (T1, T2, and T3), a four-step automatic evaluation is generated according to the following algorithm: If R<=T1 then "very poor", else if T1<=R<T2 then "poor", else if T2<=R<T3 then "good", else "very good." We designed, developed, and evaluated a method to automatically evaluate the two perspectives, taking the four-step evaluation by the walking expert as the correct answer.
Section 4, it was inferred that physical swing size is one of the perspectives from which experts evaluate the state of gait, so we decided to use this as an indicator for automatic evaluation. Based on the representative values of arm swing magnitude, we developed a method to generate a four-level automatic evaluation using a threshold process based on arm length. 1. The maxima of arm swing for each cycle are extracted with a peak detection program implemented in python, and the average value is calculated to be the representative value. The value marked in the graph in Figure 4 is the local maximum. 2. The difference between the initial Y-coordinates of the VR goggles and the left controller is used as an approximation of the arm length. 3. A three-step threshold is established to generate a four-step automatic evaluation. Several sets of thresholds were compared and the one that could generate the closest results to the expert's correct data was selected. The three thresholds (T1, T2, and T3) are, T1 = arm length*0.1, T2 = arm length*0.2, and T3 = arm length*0.3 (in meters) were determined.
A "forward swing" means that the arm is swung parallel to the direction of gait, while a "sideways swing" means that the arm is swung in a direction not parallel to the direction of travel. The "forward swing" is considered the ideal arm swing for walking because it is easier to propel the gait. The difference is detected by using the trajectory on the XY plane as a cue. Although various detection methods are possible, two methods were considered in this study. 1.Method Based on Amplitude of Arm Swing in X-axis Direction A method was developed to generate an automatic fourstep evaluation based on representative values of leftand right-hand swing amplitudes in the X-axis direction. First, find the respective minimum and maximum values of the left- and right-hand trajectories in the X-axis using a python program, and then find the difference between them. Figure 5 visualizes the trajectory of the arm swing in the XY plane and the difference. Next, for the value of the difference obtained in step 1, a set of three thresholds was used to generate a four-level automatic evaluation. Several sets of threshold values were compared and the one that could generate the closest result to the expert's correct data was selected. Here, the three-step threshold values were T1=0.1, T2=0.14, and T3=0.17 (unit: m). 2.Method based on angle of arm swing.
We introduced a method of evaluation based on how many degrees the arm swing moves in the left-right direction touching the shoulder as a reference. The arm height is taken as the vertical axis and the arm swing width as the horizontal axis and evaluated according to the diagonal angle. A threshold was set for the angle, and a four-step evaluation method was developed. First, output the difference of Y-coordinates between the headset and the hand controller.
Next, find the angle at the vertices indicated in Figure 6 in the triangular shape composed of the difference between the maximum and minimum values in 5.2 on the horizontal axis and the value output in step 1 on the vertical axis using a python program.
Then, generate a four-step automatic evaluation using the three thresholds set for the angles obtained in step 2. As in the previous step, several sets of threshold values were compared and the one that could generate the closest results to the expert's correct data was selected. The threshold values were T1=17, T2=12, and T3=10 (degrees).
The evaluation by the expert was taken as the correct answer, and the automatic evaluation method with the two perspectives developed was evaluated. The evaluation was performed in each hand, and the average value was used as the final evaluation. For the analysis of the results, three groups were established: exact match, category match, and mismatch. For the values of the automatic evaluation, perfect match was defined when the error with the correct data was within 0.5, and category match was defined when the categories of good (3 or 4) and bad (1 or 2) were matched. A mismatch was considered when the data did not belong to any of the above categories.
The results of the automatic evaluation by the proposed comments on each case, which were extracted only from those related to arm swing, in addition to whether they when even category match was included, the automatic evaluations. This confirmed that physical swing size was included in the expert's evaluation criteria. On the other hand, the accuracy of the evaluation was low for cases with high arm swing evaluated by the expert's and for cases with variations in the arm swing itself. These cases can help to refine the criteria for high arm swing.
In terms of arm swing forward or sideways, two automatic detection methods were tested using (1) the X-axis amplitude in the frontal plane (XY plane) and (2) the angle of swing relative to the Y-axis as feature values. The results for each method are shown in Tables 1 and 2, respectively. It was found that when evaluated by swing width, accurate evaluation was possible for examples of small arm swing and sideways swing, but there was a possibility of erroneous evaluation for those with too small arm swing or those with large overall arm swing and consequently large swing to the sideways. On the other hand, when the evaluation is based on the angle of the arm, it is possible to cover a few cases where the overall swing of the arm is large and the sideways swing is also likely to be large, but it is difficult to deal with cases where the arm length is not in proportion to the arm swing. The automatic evaluation by the two methods was found to be in match with the expert's evaluation for each group of subjects with different characteristics. It was suggested that the introduction of the automatic evaluation method combining both arm swing width and arm angle features is promising.
gait evaluation from all the comments collected in this study. 1. Arm and trunk movements Emphasis is placed on the awareness of hand swing and the forward swing of the arm swing, as well as its width and height. Movement only from the elbow down and the shoulder blades are also important observation points. 2. Leg movement and body balance below-the-knee movement, hip movement, X-leg condition, pelvic tilt, and foot grounding are observed in detail. It is believed that movements that cause the hip joint to open outward may lead to a tendency to let the arms escape outward. 3. Propulsion and gait stability The propulsive force and the stability of the height of the arm swing are also important factors in the evaluation. It has been pointed out that movements in which the arms move ahead of the body, or movements in which the arm swing is forceful and unnecessary, can cause the gait to become unbalanced.
The present study reveals that the gait instruction expert evaluates the overall quality of gait by comprehensively observing the movements of each body part. These findings lead to the design of behavior sensing methods for features other than arm swing and the extension of automatic evaluation methods based on these methods.
We proposed gait sensing and evaluating methods focusing on arm swing in a VR environment. We found two evaluation scales, "arm swing height" and "forward or sideways swing," and developed automatic detection methods for each. Compared with subjective evaluation by experts, we showed that the developed method was able to automate some of the perspectives of gait evaluation held by experts. Furthermore, the results of this evaluation also revealed the characteristics that experts focus on when evaluating gait, which had been implicit until now.