Tiny Machine Learning (TinyML) is a novel research area aiming at designing machine and deep learning models and algorithms able to be executed on tiny devices such as Internet-of-Things units, edge devices or embedded systems. Smart pervasive devices are rapidly becoming omnipresent in our every-day life, and TinyML and its paradigm of executing everything on-device (and thus not moving the data from where they are collected) has been crucial in designing algorithms and applications that enhance the privacy of users. From this perspective, radar sensors are currently emerging as a valid alternative to common sensors (e.g. microphones, cameras...). Given the impossibility to recognize precisely the identity of the user, they can be used in cases where it is important to recognize the presence or the behaviour of human beings while guaranteeing at the same time to preserve their privacy. UltrawideBand (UWB), in particular, is a radar technology that is particularly promising for use in pervasive systems. Indeed, its precision, low energy consumption and fastness are particularly suitable for privacy-preserving embedded applications. We introduce here, for the first time in the literature, a TinyML solution integrating pre-processing and tiny convolutional neural network for subject recognition (i.e., recognizing the age-class of the target) through the analysis of UWB-radar data. The proposed solution has been successfully tested on a real-world application of in-car subject recognition.
In recent years the difusion of tiny devices, such as Internet-of-Things (IoT) units, edge devices
and embedded systems, representing the technological asset of the “computing everywhere"
paradigm [
The role of Tiny Machine Learning (TinyML) is to design, develop and deploy MDL models and
algorithms for tiny devices. TinyML solutions present in the literature typically introduce tiny
MDL architectures and approximate-computing solutions (such as quantization [
The aim of this paper is to introduce a TinyML solution for subject recognition (i.e.,
recognizing the age-class of a person) on UWB-radar data. The proposed solution, which extends what
introduced in [
The proposed solution has been successfully tested on a real-world in-car subject recognition application. In particular, the proposed UWB-based TinyML solution for the in-car presencedetection has been successfully deployed and tested in real-world conditions on an ESP32 microcontroller unit (4 MB of flash memory, 512 KB of S-RAM memory), equipped with a UWB-radar module comprising only one pair of antennas.
The paper is organized as follows. Section 2 describes the related literature, while Section 3 introduces the proposed TinyML solution for UWB-based subject recognition. Section 5.0.3 details the problem definitions and the experimental results for the in-car presence detection scenario. Finally, Section 6 draws the conclusion and describes the future research directions in this field.
This section describes the related literature in the field of TinyML (Section 2.1) and the available
UWB-radar solutions for presence detection and activity recognition (Section 2.2). Given the
novelty of the proposed problem, no solution is present in the literature for subject recognition.
We emphasize that, the only TinyML solution able to process UWB-radar data available in the
literature is our previous work on presence detection[
The research in the field of machine learning for embedded systems and IoT units is mainly addressed from two diferent point of view: the development of custom hardware and the design of approximated MDL solutions. We here concentrate on approximated MDL solutions.
The design of approximated machine/deep learning solutions capable of addressing the strict
technological constraints of embedded and IoT units is a relevant and continuously-growing
research field. The techniques introduced in this area can generally fall within the field of TinyML
[
A diferent approach to approximate CNNs is to reduce the memory required by the solution through the use of quantization, which exploits limited-precision data types [16][17] for the CNN weights and, possibly activations. In such a direction, [18] combined both task dropping and precision scaling techniques to design approximated CNNs able to be executed in IoT units.
Other solutions focus on reducing the mean inference time of deep neural networks. Adaptive
Early Exit [
The literature about the usage of artificial intelligence with UWB-radar data mainly concentrates on tasks similar to person detection and human activity recognition (HAR). UWB-radar were used also for human sensing and vital parameter estimation. In this related literature, we focus our attention on UWB-radar solutions that rely on a single receiving antenna.
Most of the solutions for presence detection based on uwb-radar relies on thresholds or statistical approaches to distinguish between empty records and records where a human is present [19][20]. These solutions are usually heavily dependant on the dataset, and thus fails to generalize. Finally, anti-abandon systems for cars based on radar can be also found in the literature [21][22] but they do not rely on uwb-radar and they are only meant to detect the presence of a general subject in a car.
Of particular interest is the work of [23], who has used UWB radar data for wireless human sensing and personal identification. In this field, the possibility to recognize the breath and heartbeats of the targets is proven also in [24] and [25]. Nevertheless, in each of these researches the target is standing in front of the radar without the possibility of moving, making it very dificult to generalize in a general use-case scenario.
Let ∈ R× , with , ∈ N, be the output of the UWB-radar receiving-antenna installed on the device, being the collected number of radar scans and the number of “bins" characterizing the acquisitions of the antenna. In more detail, the value [, ] with = {1, . . . , } and = {1, . . . , } represents the energy acquired by the -th scan at the -th bin. We emphasize that = · , being the UWB-radar frame rate (i.e., the number of acquisitions per seconds) and the acquisition time horizon (in seconds), while represents the number of “quantized" distances in the acquisition range, i.e., from MIN_RANGE to MAX_RANGE, of the UWB-radar antenna1. An example of the acquisition of is shown in Figure 1. 1Each value represents the amount of energy of the reflected radar wave. MIN_RANGE, MAX_RANGE and are parameters depending on the specific radar device used and on its configuration.
For this work, the dataset presented in [
A high number of studies have enlighted the possibility to reliably estimate the breathing frequencies of human targets with the use of UWB-radar data [25, 23].
The standard breathing frequencies of each age category at rest have been estimated as illustrated in the table 1, where the data have been taken from [26].
From this perspective, the breath frequencies of the targets from UWB-radar data could be used to to distinguish among diferent age-classes (to recognize subjects). Since the radar module makes scans at a frequency of 5Hz, it is possible to match each frequency bin of the birth to 6 weeks 6 months 3 years
Adults Elderly ≥ 65 years old
Avg. respiratory rate at rest 30-40 breaths per minute 25-40 breaths per minute 20-30 breaths per minute 15-18 breaths per minute 12-28 breaths per minute Fast Fourier Transform (FFT) data with a frequency range, calculated as: dimension of each frequency bin = 5Hz 128 = 0.039 ≈ 0.04
For example, the Hz range of the first bins will be 0 - 0.04 Hz, while the last one will be 4.965 Hz
From these estimations, the table 1 have been updated to include the bins where it is possible to expect the breathing frequencies to be recognizable.
In order to check these hypotheses, for each class the FFT data of some scans belonging to the dataset used for the experiment in section 5.0.3 have been visualized as a heatmap, searching for peaks in energy around the expected bins.
The following visualizations represent the euclidean norm of the real and imaginary parts of the FFT data of specific scans, in which only one target is present. The first bins have been artificially set to 0 in order to better display the interesting portion of data.
Note that in order to better display the peaks, in the visualizations the scale is not fixed.
In Figure 3 the data of two scans containing only one Adult in the first seat are reported. Around bins 6-8, in which the breathing frequency of the target should reside, it is possible to observe some peaks in the data, but overall, especially in the second record, they are not easily distinguishable.
The visualizations have been repeated also for children (Figure 4). In these visualizations it is much more dificult to clearly distinguish peaks traceable to the breathing frequencies of toddlers and babies (expected bins 8-17). It’s also interesting to note that in that visualization the scale is almost an order of magnitude smaller with respect to the adults’ records: this could mean that the breathing is not absent, but that it is much more dificult to distinguish among records of people with diferent age-class it from the noise of the recording. Nevertheless, the magnitude of the signal could in principle be a relevant aspect for the classification of the record. Anyway, even in the same class of records, there are significant diferences between one record and another. There is no clear unique behaviour for the same type of data, or at least is hardly recognizable by watching the graphs.
The proposed TinyML solution for subject recognition based on UWB-radar comprises two main modules: a pre-processing module and and tiny deep convolutional neural network called TyCNN-C. These two modules, which are detailed in the sequel, have been jointly designed and developed to maximize the recognition accuracy, while satisfying the strict technological constraints of tiny devices.
The pre-processing module comprises the following three steps: Fast-Fourier Transform,
lowpass frequency selection and data normalization. This module has not been substantially
modified from the TyCNN for presence detection implementation, and thus we remand to [
The first step of pre-processing aims at computing the FFT of . The considered FFT algorithm is the Cooley-Tukey algorithm [27]. The FFT is computed on all the rows of .
The second step aims at selecting only a sub-range of the frequencies in to reduce the memory and computational demands of the next tiny convolutional neural network (see Section 4.2). Being < /2 the selected cut-of frequency, the goal of this module is to remove from the rows corresponding to the frequencies larger than . In this application setting, assuming the relevance of the breathing frequencies discussed in section 3.3, can be scaled down (with a negligible loss of accuracy) to roughly 2Hz.
Finally, the last pre-processing step comprise a log-scale transformation and a Z-score normalization (where mean and standard deviation are computed on the training set of the TyCNN-C).
We introduce here TyCNN-C, which is the TinyML CNNs for subject recognition with
UWBradar data. The proposed solution, which extends the TyCNN design proposed in [
An overview of the TyCNN-C is shown in Figure 5, while the TyCNN-C architecture and the quantization mechanism are summarized in what follows.
Being the input of the TyCNN-C, the processing layers can be summarized as follows: 4.2.1. A 2 × 2 Max Pooling layer this layer aims at reducing the size of the input . In more detail, the goal of this layer is to reduce the memory demand of intermediate activations as well as the number of operations required by the TyCNN-Cs to compute the inference. 4.2.2. A sequence of Tiny Convolutional Blocks the Tiny Convolutional Blocks (TCBs) introduced for the TyCNN, were reused for the TyCNN-Cs architecture. Each block comprises the four following steps: • two convolutional layers comprising square × dilated filters with dilation rate equal to 2; • the ReLu activation function; • a 2 × 2 Max Pooling layer.
In the considered subject-recognition application described in Section 5.0.3, the following configuration of the TBCs have been considered: = 2, = 14 and = 7.
The aim of this last layer is to provide the final classification of the TyCNN-C. In more detail, this layer is composed of a flattening layer, a dropout layer (with dropout rate equals to 0.3), and a single dense layer. Diferently from the previous TyCNN design, the dense layer is characterized by softmax activation.
For the training we considered the Categorical Crossentropy as loss function, while Adam was selected as optimizer. The learning rate was set to 0.3e-4, while the number of training epochs was set to 400. Given the fact that the distribution of the classes in the training set were really skewed, the errors on each record were weighted, using weights inversely proportional to the representation of the corresponding class in the dataset. Once the TyCNN-C has been trained, the full-integer post-training weight quantization algorithm introduced in [28] has been used to transform the 32-bit floating-point weights into 8-bit integers. The same quantization scheme has been also applied to inputs and activations.
The considered tiny device is based on an ESP32 Microcontroller unit (MCU). Following the
notation introduced in [
The input matrix is characterized by M = 53, N = 200, each acquisition is W = 20 s long, and the frame rate was fixed to = 10 Hz. has been set to 1.66 Hz, such that the dimensions of are M = 53 and L = 86.
For the experimental results the dataset has been randomly split into 75% for the training and 25% for the testing, four runs have been considered (in a cross-testing fashion) and the average classification results is reported. Furthermore, the standard deviation was computed and used to estimate confidence interval (95% confidence).
Table 3 report the detailed memory footprint and the number of operations of the network on a per-layer basis, while table 4 describes the classification abilities of the proposed solution together with the memory footprint and the computational load for the subject-recognition scenario.
As a baseline we considered a simple algorithm that assigns the most represented class in the training dataset to every test data point.
Furthermore, since the imbalances in the dataset make the accuracy not the best metrics to evaluate the performance of the algorithm, the confusion matrix is here reported: True \Pred absence (0) child (1) adult (2)
absent (0) 182 (82.72%) 31 (19.01%) 0 (0%)
child (1) 34 (15.45%) 113 (69.32%) 7 (15.22%) adult (2) 4 (1.83%) 19 (11.67%) 39 (84.78%)
The proposed solution completely matches the technological constrains with = 47.8 and = 8.576. We measured experimentally the execution time of the solution on the ESP32 board. The total execution time is 940 ms, divided in 230 ms for preprocessing data and 710 ms to perform the inference with the TyCNN-C. Preprocessing required 27136 B to be executed in memory, and thus can be executed in the same memory space of dimension ^ = 31304 B where the activations of the networks will be stored, hence not influencing the memory footprint.
The aim of this paper was to introduce, for the first time in the literature, a TinyML solution for subject-recognition based on UWB-radar. To achieve this goal we used TyCNN-C, an adapted version of the TyCNN network design used for presence detection. The efectiveness and eficiency of the proposed solution have been successfully evaluated on a real-world scenario for in-car subject recognition.
Future works will encompass comparisons with other state-of-the-art architectures, alwayson scenarios for the proposed solutions, incremental learning mechanisms to support the on-device learning and the extension of the use of UWB-radar to human activity recognition.
The authors would like to thank Ing. P. Lento, and Dr. A. Bassi from Trusense s.r.l. and Ing. G. Viscardi from Politecnco di Milano for the support in the work. [12] I. Fedorov, R. P. Adams, M. Mattina, P. Whatmough, Sparse: Sparse architecture search for cnns on resource-constrained microcontrollers, Advances in Neural Information Processing Systems 32 (2019). [13] A. Kumar, S. Goyal, M. Varma, Resource-eficient machine learning in 2 kb ram for the internet of things, in: International Conference on Machine Learning, PMLR, 2017, pp. 1935–1944. [14] S. Han, H. Mao, W. J. Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and hufman coding, arXiv preprint arXiv:1510.00149 (2015). [15] Y. He, X. Zhang, J. Sun, Channel pruning for accelerating very deep neural networks, in:
Proceedings of the IEEE international conference on computer vision, 2017, pp. 1389–1397. [16] Z. Cai, X. He, J. Sun, N. Vasconcelos, Deep learning with low precision by half-wave gaussian quantization, in: Proc. IEEE Conf. on computer vision and pattern recognition, 2017, pp. 5918–5926. [17] S. Gupta, A. Agrawal, K. Gopalakrishnan, P. Narayanan, Deep learning with limited numerical precision, in: International conference on machine learning, PMLR, 2015, pp. 1737–1746. [18] C. Alippi, S. Disabato, M. Roveri, Moving convolutional neural networks to embedded systems: the alexnet and vgg-16 case, in: 2018 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), IEEE, 2018, pp. 212–223. [19] J.-E. Kim, J.-H. Choi, K.-T. Kim, Robust detection of presence of individuals in an indoor environment using ir-uwb radar, IEEE Access 8 (2020) 108133–108147. [20] S. Chang, al., An algorithm for uwb radar-based human detection, in: 2009 IEEE Radar
Conference, IEEE, 2009, pp. 1–6. [21] A. R. Diewald, al, RF-based child occupation detection in the vehicle interior, in: 2016 17th
International Radar Symposium (IRS), 2016, pp. 1–4. [22] A. Caddemi, E. Cardillo, Automotive Anti-Abandon Systems: a Millimeter-Wave Radar Sensor for the Detection of Child Presence, 2019. doi:10.1109/TELSIKS46999.2019. 9002193, pages: 97. [23] A. Rahman, V. M. Lubecke, O. Boric-Lubecke, J. H. Prins, T. Sakamoto, Doppler Radar Techniques for Accurate Respiration Characterization and Subject Identification, IEEE Journal on Emerging and Selected Topics in Circuits and Systems 8 (2018) 350–359. doi:10.1109/JETCAS.2018.2818181, conference Name: IEEE Journal on Emerging and Selected Topics in Circuits and Systems. [24] T. Sakamoto, Personal Identification Using Ultrawideband Radar Measurement of Walking and Sitting Motions and a Convolutional Neural Network (????) 9. [25] S. Rana, M. Dey, R. Brown, H. Siddiqui, S. Dudley, Remote Vital Sign Recognition through Machine Learning augmented UWB, in: 12th European Conference on Antennas and Propagation (EuCAP 2018), Institution of Engineering and Technology, London, UK, 2018, pp. 619 (5 pp.)–619 (5 pp.). URL: https://digital-library.theiet.org/content/conferences/10. 1049/cp.2018.0978. doi:10.1049/cp.2018.0978. [26] K. E. Barrett, W. F. Ganong, Ganong’s review of medical physiology., McGraw-Hill Medical, 2012. [27] J. W. Cooley, J. W. Tukey, An algorithm for the machine calculation of complex fourier series, Mathematics of computation 19 (1965) 297–301. [28] B. Jacob, S. Kligys, al., Quantization and training of neural networks for eficient integerarithmetic-only inference, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2704–2713.