This paper presents a 60 GHz millimeter wave radar technique for the classification of behaviors related to bed-leaving and bed-lying of elderly adults using real data acquired in nursing care facilities. Time-range and time-velocity distribution images corresponding to short-time data were generated based on frequency-modulated continuous-wave (FMCW) radar signal processing. Then, bed-leaving and other behaviors such as bed-lying and sitting on the bed were classified with 97 % accuracy using the convolutional neural network whose inputs were the generated time-range images. However, in other experiments, the classification of gaits during bed-approaching and bed-leaving using time-range images was failed. To overcome this, we used the time-velocity images calculated based on Doppler radar processing and achieved a 90% accuracy rate for their classification. These results suggest that millimeter wave radar can be a useful tool for monitoring the activities of elderly adults in a non-invasive way for daily monitoring in nursing care.
In the elderly society, the number of careers is insufficient for the number of people requiring care, and the burden on each career have become increasing. In nursing homes, it is necessary to make rounds even late at night to prevent care recipients from falling out of bed or wandering off, and this may be one reason for the increased workload on careers. To solve this problem, previous studies have reported the development of sensor-based monitoring systems using various devices to grasp the behaviors of the person on the bed. In particular, early detection of the bed-leaving and related behaviors is important to detect abnormal situations.
For this purpose, pressure sensors [
In addressing the aforementioned challenges, applications of radar technology to monitoring systems
for elderly persons have been recently studied [
To address the aforementioned issues in the radar-based monitoring systems, this study used millimeter-wave radar images, which fully contain range and velocity information of human body parts, to classify human behaviors related to bed-leaving and bed-lying. By applying frequency-modulated continuous-wave (FMCW) radar signal processing, time-range (time-distance distribution) and spectrogram (time-velocity distribution) images were generated as the input of deep learning-based behavior classification. Accurate classifications were demonstrated using the images corresponding to short-time data, for real-time monitoring systems for nursing homes.
Our experiments aim to classify the behaviors related to bed-leaving and -lying events of the elderly adults staying at nursing homes via the unstrained measurements using millimeter-wave radar: the assuming behaviors we need to classify were bed-leaving, bed-lying, bed-approaching, and sitting on bed. Fig. 1 shows experimental setup. A 60 GHz FMCW radar was installed at height of 40 cm from surface of bed. The radar bandwidth was B = 6.8 GHz, the range resolution was ΔR = c/2B = 2.4 cm (where c is the speed of light) and its frame rate was 80 Hz. The participants were three elderly adults who were staying the nursing care facilities. No instructions were provided for the participants during the measurements; they were staying at their room as usual. Fig. 2 outlines the states of the participants for classes "bed-lying" and "sitting on the bed".
The experimental protocol was approved by the local ethics committee of Toyama Prefectural
University and the participants were first given written and verbal instructions explaining the testing
procedures. Written consent was then obtained from each participant prior to measurement. In addition,
we did not collect any information that can identify the participants such as their names and addresses.
parts [
, = b 2 where A is the transmitting amplitude, t is the time, and f0 is the central frequency, and M = B/Tm (Tm is the time length of chirp signal). The received signal from a target whose range is R = ctr/2 is expressed as ( ) = ′ cos 2 0 + b − 2 2 where g' is the receiving amplitude and fb is the frequency differences between transmitting and receiving chirps which is called the beat frequency. As indicated in Fig. 3, the beat frequency is corresponding to the delay of chirp signal which is determined by target range. The relationship between target range and beat frequency is expressed as Eq. (3).
According to Eq. (2), the Fourier transform of ( ) extracts fb, and the target range R is calculated by (1) (2) (3)
To classify the behaviors of participants, we calculate the time-range and time-velocity distributions from the radar received signals. First, we calculate time-range map by iterating the signal transmission with the framerate of F = 80 Hz. We define the received signal for k-th signal transmission as ( , ) where τ = / is the time with respect to the signal transmission. The range profile for each time k is calculated as where FTt[ ] denotes the Fourier transform with respect to t. We obtain time-range distribution using X as ( , ) = [ ( , )], | ( , )|. (4) (5) where R is calculated from fb using Eq. (3).
Fig. 4 shows an example of an image of time-range distribution for a certain participant. In this data, a participant's state change order was: bed-lying, bed-leaving, bed-approaching, slight motions on the bed before bed-lying, and bed-lying. The components corresponding to higher amplitude are the ranges of that the participant is existing. In this example, the participant firstly lying on the bed. Then, the participant leaves from the bed and moved out of the room. The participant returned to the room about 40 seconds later and was supine again on the bed. We can confirm these behaviors from Fig. 4: Variation of range corresponding to walking from bed to outside of the room was confirmed around t = 30 s, no significant components corresponding to bed-leaving state were confirmed at approximately t = 40–70 s, and we can confirm the variation of range corresponding to walking approaching at around t = 70 s and the slight motions before lying to bed at approximately t = 80–100 s. No variations in ranges at t > 110 s corresponds to sleeping at bed without large motions. Bed-lying
Walking
Bed-Leaving bBeefhoarveiors
Walking bed-lying
Bed-lying
Then, time-velocity images are obtained from the range profiles. The velocity information is derived as frequency information (Doppler frequency fD) with respect to time . To obtain time-variation of the velocity for each R, we applied the short-time Fourier transform (STFT) to the range profile X. The time-Doppler frequency distribution is calculated as d( , D, ) =
[X( , )]. 2 1 d( , D) =
| d( , D/(2 0), )|d , Using this STFT result, we obtain the time-velocity distribution of the ranges that the participant exists as where we assume the participant exists in the ranges from R1 to R2.
Fig. 5 shows an example of the time-velocity image calculated from the time-range image of Fig. 4 We set R1 = 0.5 m, R2. = 3 m, and a length for the Hamming window function used in the STFT process was 1 s. The larger variations of velocities corresponding to the leaving from and approaching the bed were confirmed. The significant components of smaller velocities correspond to the slight motions in lying in the bed. (6) (7)
Bed-Leaving Walking
Walking 3.2.
For the classification of behaviors of the participants, we used a CNN whose inputs were time-range or time-velocity images. We cropped the time-range image every 0.5 seconds and convert it to a 224 × 224 PNG image, and these cropped images without axes are used as the dataset for the inputs of the CNN. The behavior corresponding to each image is classified using the CNN. Similarly, we perform cropping and data resizing for the time-velocity images. Fig. 6 shows the examples of the generated images for the classification.
For the CNN architecture, we used the ResNet-18 [
In this section, the classification results for representative behaviors related to bed-leaving and lying movements are presented. For each classification, the CNN was trained for 70 % of the images and the remaining 30 % of the images were used as the test data. We evaluated the classification performance via the hold-out validations iterated by randomly varying the split of training and test data. 4.1.
Firstly, the classification of bed-leaving and bed-lying. Bed-leaving is defined as the state that the participant is not in the radar measurement area, and bed-lying is defined as the participant's slight motions in the bed lying. The bed-lying behaviors include not only the states that the participant is resting in bed but also that the participant is moving in a moseying manner, including turning over behaviors. From the three participants, we generated 100 time-range images corresponding to the bedleaving and -lying each. The number of the hold-out validations was ten.
Table 1 shows the confusion matrix. The mean classification accuracy was 99.2 % and there were no misclassifications of the bed-leaving to bed-lying. Because the detection of bed-leaving is relatively important for practical use, these results indicated appropriate performance.
Because the detection of the bed-leaving is important for daily monitoring systems, this subsection considers the classification of the bed-leaving and other various behaviors not only the bed-lying; the other behaviors class includes the behaviors of the bed-lying, sitting on the bed, and walking. For the evaluation, we generated 76 time-range images corresponding to various behaviors different from the bed-leaving and bed-lying. We used these generated images and 100 time-range images used in the previous subsection. The number of the hold-out validations was ten.
Table 2 shows the results and the mean classification accuracy was 97.4 %. Although the classification accuracy was deteriorated compared with only the classification with bed-lying, sufficient accuracy was achieved.
Then, we classify the sitting and lying on the bed. This classification is efficient to accurately grasp the states of the participants. We used 69 time-range images for the sitting on the bed and 100 timerange images for the bed-lying. The classification accuracy was evaluated using 10 times hold-out validations.
Other behaviors 4.3 % 95.7%
Table 3 shows the confusion matrix and the mean classification accuracy was 92.5 %. There were larger misclassifications of the sitting to lying because both classes had some similar images due to their relatively small movements. However, appropriately better accuracy was achieved.
Finally, we classify the gaits during bed-approaching and bed-leaving, which aims to detect the symptoms of the leaving before it happens. We generated 22 time-range images corresponding to the walking motion in the bed-approaching and -leaving each. Because the size of the dataset is small, we performed 20 times of hold-out validations. Furthermore, for this classification problem, we also generated the time-velocity images with the same received signals because it is considered that the differences in the directions of the gait may be easily detected as the velocity information.
Table 4 shows the results using the time-range images and the mean classification accuracy was 65.4 %. As indicated in this result, the classification of these gaits is difficult using the time-range images because the short time images did not sufficiently include information on motion direction. Fig. 7 shows the examples of the time-range images of the assumed two classes and indicates that their differences are unclear.
Table 5 shows the results the time-velocity images. The mean classification accuracy was 90.1 % and this means that the classification accuracy of gaits was improved using the velocity information compared with the range information. As indicated in Fig. 5, there are differences between the bedleaving and approaching motion velocities. Therefore, our findings suggest that time-velocity images offer a relatively effective approach for gait classification, including the determination of their respective directions.
In this subsection, we discuss the merits and limitations of our study. Compared with the conventional studies that used the other sensors, our radar-based method can remotely recognize the behaviors related to the symptoms of the bed-leaving without any privacy concerns. Typically, although pressure sensors embedded in beds are commonly used for this purpose, they struggle to discern detailed preliminary behaviors of the bed-leaving for its early detection. Besides, widely used infrared cameras raise privacy concerns as they visually capture participants' daily activities. The radar-based method circumvents these issues by employing radio waves for remote sensing of human behaviors, whereby time-range and time-velocity distributions do not contain any private information. Additionally, the millimeter-wave radars can accurately measure velocity, enabling facile recognition of subtle movements in contrast to optical remote sensors like cameras and lasers.
Furthermore, compared with the other radar-based studies, our study achieved the classification of detailed behaviors including sitting and lying and walking motions bed-approaching and bed-leaving with shorter data of 0.5 s. By using the CNN with inputs of time-range and time-velocity images, we achieved accurate classification of motions associated with bed-leaving using a efficient methodology, devoid of additional parameters reliant on the measurement environment.
However, this study is subject to three limitations. First, the number of participants was only three
and the limited motion patterns of lying, sitting, bed-leaving, and bed-approaching behaviors were only
measured and classified. In practical applications, because these patterns may vary among individuals,
necessitating a larger dataset comprising various participants to enhance the learning capabilities of our
classification system. Secondly, significant behaviors and movements pertinent to daily monitoring
applications, such as various sitting postures and bed-leaving motions accompanied by falls, were not
considered. To address this, additional data from participants representing various profiles should be
incorporated to enable the classification of a wider range of behaviors. Another important limitation is
that we used the transfer learning using ResNet-18 only. Although the ResNet-18 is known as the
efficient network for radar-based human motion recognition, other studies have demonstrated the
effectiveness of various architectures tailored to the unique characteristics of radar images [
To develop remote monitoring systems for nursing care, this study demonstrates the classification of various behaviors related to bed-leaving and bed-lying using 60 GHz millimeter-wave FMCW radar. The CNN, utilizing time-range images generated from radar data at 0.5 s intervals, achieved the accurate classification of bed-leaving and other behaviors including sitting on the bed with over 90% accuracy. Although time-range images were unable to accurately classify gaits during bed-approaching and bedleaving, these were classified using time-velocity images generated based on Doppler effects. These results suggest the feasibility of real-time behavior classification for early detection of bed-leaving and other significant situations of care-receivers. To develop practical systems by removing the limitations discussed in the previous section, future experimental validations were required using data of a larger number of participants and other dangerous situations and behaviors, such as falling.