MediaPipe is a leaning-based human pose detection technology that detects the position and movement of a person's body, face, fingers, etc., from videos. Nowadays, many orthopedic studies put eforts into finding a biomarker of orthopedic diseases from the correlation between gait and orthopedic, using MediaPipe. This paper presents the results of applying the LSTM(Long Short Term Memory)-Autoencoder-based anomaly detection technique for orthopedic diseases, e.g., sarcopenia disease and the capability of distinguishing the normal and abnormal gait. We compare the sensitivity of the anomaly detection based on 5 human body points in predicting sarcopenia so as to find the primary gait features of human body. In addition, we present four environmental factors afecting MediaPipe Recognition that can improve the accuracy of anomaly detection using MediaPipe. Our anomaly detection approach detects 92% (35) of sarcopenia patients from 38 patients.
In recent years, AI(Artificial Intelligence) that can help
doctors’ decisions based on accumulated data is
positioned as a research flow in the medical area[
knee joint to patients with osteoarthritis, one of the
musculoskeletal diseases[
Image tracking systems, such as marker-based VICON, resulting in abnormalities in physical function. In
addioping quantitative gait analysis systems[
In this paper is organized as follows: Section 2 explains This paper is ofered lateral gait video data of 78 the data pre-processing process and research method. sarcopenia patients and 29 normal persons from the In Section 3, we propose the experimental results. In Gyeongsang National University Hospital Orthopedic. Section 4, we describe environmental proposals for using After pre-processing, this paper uses lateral gait image MediaPipe. Finally, we conclude this paper and present data from 38 sarcopenia patients and 21 normal persons. future work in section 5. In the data pre-processing process, we remove parts of videos that can cause recognition errors when applying MediaPipe, such as parts that are not on the lateral of 2. METHODS pedestrians or that the camera frame covers the joints.
And we apply MediaPipe to extract joint points and cal
This section describes the data extraction, preprocess- culate the angles of both knees, hips, and nose-shoulder ing process, and the model used in the study. using the DMS of DEEVO Co. Ltd. And we exclude the video data with spike values of the angle data caused 2.1. MediaPipe Description and Data by the MideaPipe’s recognition error that can afect the Extraction model training. MediaPipe’s recognition errors refer to these cases: recognizing the background as a pedestrian’s
MediaPipe is an open-source gait analysis system sim- joint because of clothing in color similar to the
backilar to OpenPose developed by Google in June 2019[
is a model created to overcome the vanishing problem,
which is a drawback of RNN. An Autoencoder is a neural
network that compresses input data and restores the
compressed data to its original form. LSTM-Autoencoder
is a model that combines these two models. The
LSTMAutoencoder has a low loss value for the time series
data as learned and a relatively high loss value for the
untrained data[
than 30, and the nose-shoulder is 10. On the other hand, the right and left hips show relatively low results with 7 and 5 sarcopenia patients, respectively.
Fig. 3 shows a graph detecting outliers in the left knee Fig. 5 is a Confusion Matrix showing rates of predictof sarcopenia patient who have video ID 29. The red line ing sarcopenia in other body parts when predicted in a is the threshold obtained through training, and the red particular body part. For example, 0.29 in the first coldots are outliers. umn and second row means the ratio of the number of
Fig. 4 shows the number of sarcopenia patients de- predicted sarcopenia through the Right hip and Left hip tected by each body part. The number of sarcopenia to the number of predicted sarcopenia through the Right anomaly detected is 35 out of 38, showing an accuracy of hip. When both knees are predicted to be positive, the about 92%. The number detected in both knees is more rate of other body parts also predicted to be positive is
The most significant advantage of MediaPipe is that gait analysis is possible with a smartphone without a high-spec camera. It is easy to analyze for an inexperienced expert. Also, as shown in Table. 1, MediaPipe recognizes more joint points than OpenPose, enabling accurate gait analysis. However, there are environmental proposals when taking a video to increase the recognition accuracy of MediaPipe.
Therefore, this paper presents the four environmental factors afecting MediaPipe recognition and their solutions: frame imbalance, headless, clothing, and background. about 85% and about 28% for both hips and nose-shoulder. 1) Frame imbalance: It is a case where some body part In other words, both knees have a higher anomaly detec- of a pedestrian is out of frame. In Fig. 6(1), there is a tion ratio than both hips and nose-shoulder. And among recognition error because the pedestrian’s left foot is out both knees, the anomaly detection rate of the right knee of the frame. is higher. 2) Headless: It is a case in which only the body below the neck is shown in the video. In Fig. 6(2), a recognition error occurred because a video frame does not include 4. SUGGESTING the pedestrian’s head and shoulder.
ENVIRONMENTS TO USE 3) Clothing: It is a case where pedestrians wear clothMEDIAPIPE ing similar in color to the background, clothing too large or too loose for their bodies. In Fig. 6(3), MediaPipe recognized the knee as the angled hem of the large black shorts.
4) Background: It is a case where the video included angular objects such as drawers and doors. In Fig. 6(4), a recognition error is caused by a drawer behind the pedestrian.
analyze the angle data of the video is applying MediaPipe. More than 50% of the total video data have a MediaPipe recognition error, resulting in spike values that can afect the model training. The recognition error is caused by the four environmental factors MediaPipe recognition described in subsection 4.1: frame imbalance, headless, clothing, and background. Only video editing cannot block all recognition errors for video data, including any of the four environmental factors MediaPipe recognition. Therefore, we present four environmental proposals when taking a video.
1) Frame: MediaPipe is a real-time gait analysis system that recognizes pedestrians for each video frame. However, the frame imbalance described in subsection 4.1 can cause MediaPipe’s recognition errors in cases: where the distance between the camera and the pedestrian is not kept constant, that a body part is out of video frame due to the diference between the camera’s moving speed and the human’s gait speed. As a result of anomaly detection using angle data of normal videos with recognition errors and sarcopenia patients’ videos, they are all detected as sarcopenia patients. We suppose this result to be the efect of the spike values of the angle data caused by the recognition error. Therefore, if we use a video with frame imbalance, a normal person may be predicted as a sarcopenia patient. Also, it may be dificult to determine the severity of sarcopenia.
In this paper, we propose an ideal frame and maximum frame range that does not cause frame imbalance when taking a video, using the coordinates of the bounding box of the pedestrian center of the video through the YOLO algorithm. it shows the average of the top, bottom, left, and right margins in total sarcopenia video data. Based on this, this paper proposes an ideal and maximum frame range that can recognize pedestrians.
in subsection 2.3 and the YOLO algorithm to determine Figure 8: Frame Range
the ideal and maximum frame range. YOLO(You Only
Look Once) is an algorithm that detects people or things Fig. 8 shows the ideal and the maximum frame range
through object detection in an image or video. For each for recognizing pedestrians. We calculated the frame
video frame with a pedestrian in the center of each video ranges using the average of the top, bottom and left, right
data, YOLO v3[
This paper classified each video data as long and short taking a video to apply MediaPipe, we can improve the distance, using the bottom, left, and right margins. If it is recognition accuracy by following these proposals. Also, larger than the average of each margin, it is classified as it can reduce the recognition error caused by 2) headless. long distance, and if it is smaller, it is classified as short 3) Clothing: It is better to wear clothing that shows distance. The reason for excluding the top margin is that body shape than clothing that covers joint points, such as it could be afected by the height of the pedestrian. Then, skirts, shorts, and loose clothing. Also, clothing that coneach video data is divided into long and short distances trasts colors with the background can reduce recognition by the frequency of the previously classified distance. errors.
Finally, we divide the video data into 13 long-distance 4) Background: MediaPipe tends to recognize angled and 25 short-distance samples. objects as joints, so it is good to avoid angled objects in the background if possible. We can remove or cover Table 3 angled objects in a limited environment before taking a Minimum and Total Average video. However, there is a problem that it cannot resolve recognition errors caused by angle objects in environ
Margin Top Bottom Left Right ments where objects cannot be restricted, such as a street Minimum(%) 2.2 8.2 16.4 15 or a house.
Total Average(%) 17.1 18.5 33.5
28.1
using the LSTM-Autoencoder-based anomaly detection method. And it presents four environmental factors MediaPipe recognition and their solutions: frame imbalance, headless, clothing, and background that afect recognition error when applying MediaPipe.
As a result of the technique, outliers are detected in 35 out of 38 sarcopenia patients with 92% accuracy. From Fig. 5, both knees are the most sensitively detecting body parts. Among both knees, the right knee has a higher anomaly detection rate. We suppose that the low anomaly detection rates in parts except for knees are due to characteristics that distinguish between sarcopenia and normal being uncertain. Therefore, this paper expects a better performance of a model that can respond sensitively to data in the future.
This paper presents four environmental factors MediaPipe recognition and their solutions: frame imbalance, headless, clothing, and background that afect recognition when applying MediaPipe. Frame imbalance is a case where some body part of a pedestrian is out of frame. It can reduce recognition error by maintaining the previously proposed ideal frame range for pedestrian recognition when taking a video or by retaining the maximum frame range when impossible. This frame range can also solve recognition errors due to the headless that appears only on the body below the neck in the video. And clothing that shows the pedestrian’s body shape well and in color contrast with the background can make fewer recognition errors. However, in the background case, there is a problem in that the recognition error cannot be entirely solved for the video taken in an environment where angular objects cannot be restricted, such as a street or a house.
For future work, we intend to research how to improve the accuracy of gait analysis by solving the background problem among the four environmental factors afecting MediaPipe recognition.