In this paper we describe an ongoing research project in which we investigate the capability of AI-systems to recognize individuals from motion capture data, e.g. using a neural network. In our previous work [1] we also showed which motion features more strongly characterize each individual. In addition, we report on the application of some techniques suggested by Explainable AI's literature. In particular we have analyzed the parsimonious linear fingerprinting (PLiF) [ 2] and a specific learning shapelets method suggested by an individual on the basis of his/her movements or ges- temporal scales) are more relevant for the neural netItal-IA 2023: 3rd National Conference on Artificial Intelligence, orgaIn particular we have investigated whether the re- Tavenard[3]. The remainder of this paper is organized
The possibility of using AI models in order to recognize tures has been studied in depth in the past years due to its significant applications in the security and medical areas. At the same time, it has become more and more important to provide an explanation for the decisions made by AI models and the European GDPR makes this point very clear (art. 13-15,22).
In [
Let us begin by providing a definition of the time series [6]. A Time Series = { 1, 2, ... }, ∈ ℝ× -dimensional real valued observations or time steps . We talk about univariate time series if = 1 , and multivariate time series when > 1 . Furthermore, a time-series classification dataset = ( , )
is a set of is a sequence of time series, =
1, 2, … , ∈ ℝ×× assigned labels (or classes) = 1, 2, … , ∈ ℕ . For dataset containing classes, can take diferent values. When = 2 , is a binary classification dataset, whereas when > 2 , is a multiclass classification dataset. In our case we have 60 classes that correspond to the 60 participants to the experiments, and so = 60 .
We can now define the problem of time-series classification
, TSC, as follows: given a TSC dataset , TSC is the task of training a function or mapping from the space of possible inputs to a probability distribution , with a vector of over the values of classes .
specific for time series classification when CNNs is used. In particular, they introduced a perturbation-based methods that directly compute the contribution of the input features by removing, masking, or altering them, running a forward pass on the new input, and measuring the diference with the original input.
an agnostic, post-hoc model in which we modified some preprocessing parameters, as presented in Section 3.3, and then evaluated how these modifications afected the accuracy of the neural network. Therefore, we apply what Rojat defines a perturbation-based methods.
lations between time series, we decided to explore other methods with the main purpose of extracting the most significant features within the time series that make classification possible.
A particular explanation method consists of extracting from the time series the sub-sequences of values that are most representative of class membership. These type of sub-sequences are called shapelets [6]. Shapelets were ifrst introduced in [ 8] and are calculated by finding the subsequences that maximize the information gain when dividing the set of all subsequences into two classes based on their distance from the candidate.
Tavenard [
which is called the shapelet-transform space in the re- described in depth in [4]. In total, 60 participants, formlated literature.
On other method we try to apply in our research was the parsimonious linear fingerprinting (PLiF) for time seing 30 pairs, performed a movement synchronization task. Participants were instructed to maintain a slow movement rhythm (full movement cycle: single flexion/extension movements) by having them practice in a preliminary phase with a reference metronome set at 0.25 Hz. As shown in Figure 1, participants performed the experiment under three diferent conditions. In the ifrst case, they performed alone (solo condition) with the only requirement being that they adhere to the instructed rhythm. The other two experiments were conducted in pairs, where they were asked to keep their right index ifngers pointed toward each other (without touching) and to perform rhythmic flexion-extension movements as synchronous as possible, either toward the same direction (in-phase) or toward opposite directions (antiphase).
Each trial had a duration of 2.5 minutes.
We decided to develop our AI program with Python. The software was essentially split in two main components. The first module assigned to the preprocessing that is common to all the diferent XAI techniques applied later on. The second executed module it depends on the XAI technique we decided to used.
The first module, described in chapter § 3.1, has a composite structure with diferent possible choices to preprocess the data and diferent parameters to set.
Once the data preprocessing has been completed, the segmented series are sent to the second module. In our previous work we have analyzed the response of a neural network. The TensorFlow Keras library was used to generate the neural network model. A Convolutional Neural Network (CNN) as been built for multiclass classification (60 classes, one for each partecipant). In this research we wanted to investigate two other techniques PLiF and learning shapelets, that are specific techniques for XAI in case of time series and already introduced in section 2.
In the preprocessing phase, it is possible to independently choose the series type, filter method, type of segmentation, and whether the series must be normalized. Depending on the choice made, it is necessary to specify diferent input parameters. Table 1 lists the diferent parameters required for each pre-processing choice.
Choice MAW Band Pass Extension-Flexion Sliding Window
Parameter Window Dimension Low and High frequency cut Resized Subseries Dimension
Subseries Dimesion and gap We considered two diferent methods of cutting the series. As a first approach, we cut the time series corresponding to the maximum finger positions on the x-axis. In this way, each subseries represents a complete movement, extension and flexion of the index finger. However, this type of cutting creates a subset of diferent lengths that cannot be used directly as input to a CNN. This means that we had to resize the subsets to a fixed default size that had to be the same for all of them.
The second option for cutting the time series is called sliding windows and decides a priori the size of the subseries to be obtained and the gap between the next two subseries. We applied this method to check whether there was hidden information in the time series that was not related to the whole motion cycle (extension-flexion). 3.3.2. Series Filtering
Moving average window (MAW) is a basic tool commonly used in time series to smooth signals. For each point in the series, a fixed number of successive points are averaged, and the result replaces the starting point. Clearly, the number of points involved in averaging corresponds to the size of the window: the larger the window, the smoother the signal will be.
We also applied to the signal standard frequency filters. Basically, we created a bandpass filter in which low and high cutof frequencies can be set. If the low frequency cutof was set to 0, it was applied as a low-pass iflter. For this purpose, we created a Butterworth filter using the default function of the scipy. signal library in Python.
We decided to apply a CNN, as suggested in the literature for multi-class classification of time series data [ 9] [10] [11].
The structure of the neural network is described in the table 2. We used RMSprop as the optimizer and performed early stopping to avoid overfitting. We compared the results obtained with this neural network with a similar network made with a pytorch instead of tensorlfow.keras. The results of these two networks are very similar.
1D Convolution Kernel: 3@64 ActFunct: ReLU
1D Convolution Kernel: 3@64 ActFunct: ReLU 1D Max Pooling
Pool Size: 2 1D Convolution Kernel: 3@64 ActFunct: ReLU
1D Convolution Kernel: 3@64 ActFunct: ReLU 1D Max Pooling
Pool Size: 2
Dense Dense
Softmax ferent parameters on recognition performance, we examined the accuracy of our CNN. First, we found that by using a low-pass filter set at 50 Hz and cut based on the maximum finger position, the accuracy of CNN was higher considering the normalized speed profile as input, than the one obtained with the position or acceleration profiles.
The experiment comparing the two types of series segmentation doesn’t show significant diferences in terms of accuracy, while the computational time was drastically Figure 3: In the figure, it is reported how the accuracy longer when the data was cut with sliding windows. changes for diferent band pass filters: each curve corresponds This convinced us to choose the cut at the maximum to a filter with a specific low-frequency cut (0 Hz, 2 Hz, 4 Hz, iffnogreoruprosseirtieios.n as the standard segmentation method r6epHozr,te8dH.z), while in the x-axis, the high-frequency cut is
As shown in Figure 2, the maximum accuracy obtained by applying MAW, occurs with a window size of zero, that is, when no filter has been applied. As the window any physiological relevance. Instead, in our experiments, size increases, the accuracy decreases until it reaches 30% we still see further increases when frequencies above 30 with a window size of 100 points. Remember that when Hz are added, meaning that the network is still learning MAW windows include 100 points, only the main shape something. Future in-depth analyses should investigate of the motion is visible. The change in accuracy as a func- what our neural network is learning in the range between tion of diferent frequency filters is shown in Figure 3. 30 Hz and 70 Hz. In any case, at 20 Hz, the accuracy is As can be clearly seen, the fundamental frequency (0.25 already 65 percent, a very good performance for classifiHz or the instructed rhythm of finger flexion-extension) cation among the 60 classes. is the most significant frequency, and if it is removed Since one of our initial goals was to investigate the from the signal, no recognition can be made. The fact role of sub-movements in defining individual motor sigthat each individual is characterized by its own preferred natures, we focused our attention on the frequencies tapping rhythm is well known in the neurophysiological of 2-4 Hz. Although we did not notice any significant literature [12]. Moreover, our experiments show that this change in the accuracy of the bandpass filter with a low frequency alone is not suficient and accuracy increases cutof at 2 Hz and a high cutof at 4 Hz, we could clearly with the addition of other frequencies. Interestingly, it observe a significant increase in the slope of the low-pass is known in neurophysiology that in motion, even with iflter around these frequencies (figure 4). As explained the diferences that may exist between the movement earlier, the fundamental frequency (0.25 Hz) probably of a finger or a leg, frequencies above 15 Hz begin to contained most of the information (less than 30% accube attenuated. From 20-30 onward, there is no longer racy). However, the performance is far from their plateau; indeed, the model improves vastly with the addition of the secondary motion interval.
As we said at the beginning of this paper, CNN is just one of the possible path that can be follow when we want to classify time series. In our current work we make some ifrst experiment using PLiF and tslearn trying to find if it’s possible to group in cluster the time series due to some intrinsic characteristics and than verify if these clusters may identify the correct class of the TS that means the owner of the movement. As a starting experiment, we use a low-pass filter set at 50 Hz and cut based on the maximum finger position and the resulting time series were use as input for both tslearn and PLif.
For PLiF we used the mathlab tool proposed by Li [13].
We tried applying the tool to a smaller csv file than the one obtained from the preprocessing python program described above. The initial goal was just to verify that the tool worked correctly with our inputs. For this purpose we created inputs with an increasing number of series, at ifrst only 5 series belonging to 2 diferent classes and then gradually increasing the number of series and classes involved. The figure 5 shows the result of applying PLiF to 5 time series belonging to 2 diferent classes. Unfortunately, it is not possible to see a clear distinction between the series belonging to the 2 classes.
Similar results we obtained with tslearn. In this case we gave as inputs to the model four series that belong to four diferent classes, and asked the model to use four shapelets to identify peculiar sub-sequences in the series.
The results are proposed in the figure 6. It’s clear that none of the shapelets can make a clear distinction among the series.
Our previous work demonstrated that it is possible to recognize subjects by starting from their index finger movements.
In addition, we found that the fundamental frequency is critical in subject recognition, but not by itself. Higher frequencies contribute significantly to increasing accuracy, but only in the presence of the fundamental frequency. It would be interesting to be able to explain the gap between the 30 percent accuracy obtained with the fundamental frequency alone and the 75 percent accuracy obtained with a low-pass filter with a bandwidth of 60-70 Hz.
Our current goal was to understand whether secondary movements play a central role in the recognition process, and although it has not been fully demonstrated, we have some clues. The slope of the accuracy curve has provided us with a cue for further investigation in this direction.
In this work we tried to answer this question by apply- of Machine Learning Research 21 (2020) 1–6. URL:
ing more structured clustering method such as tslearn http://jmlr.org/papers/v21/20-091.html.
or PLif, section 2, to determine what other signal fea- [4] A. Tomassini, J. Laroche, M. Emanuele, G.
Naztures most influence CNN classification. The experiments zaro, N. Petrone, L. Fadiga, A. D’Ausilio,
Interperdone so far to extract meaningful shapelets by applying sonal synchronization of movement intermittency,
a Python library (tslearn) implemented by Tavenard [
Also with PLif, the obtained results were not signifi- telligent Systems in Accounting, Finance and Mancant. This is easier to explain because PLif build up the agement 25 (2018) 63–72. doi:10.1002/isaf.1422. clustering based on the spectral analysis of the signals. [6] A. Theissler, F. Spinnato, U. Schlegel, R. Guidotti, In our case, the fundamental harmonic was established Explainable AI for Time Series Classification: A Reby the person conducting the experiment and a more view, Taxonomy and Research Directions, IEEE Acsophisticated analysis is needed to extract meaningful cess 10 (2022) 100700–100724. doi:10.1109/ACCESS. peculiarities. 2022.3207765.
This result is also consistent with an attempt we made [7] T. Rojat, R. Puget, D. Filliat, J. Del Ser, R. Gelin, to classify the spectra of the time series with the neural N. Díaz-Rodríguez, Explainable Artificial Intellinetwork described in Section 3.4, which was unsuccessful gence (XAI) on TimeSeries Data: A Survey, 2021. as well. arXiv:2104.00950.
A possible explanation for the unsatisfactory results [8] L. Ye, E. Keogh, Time series shapelets: A new obtained by the two methods we applied is that they were primitive for data mining, in: Proceedings of the created with macro classifications in mind (distinguish- 15th ACM SIGKDD International Conference on ing between walking and running, walking on a hard Knowledge Discovery and Data Mining - KDD ’09, (concrete) or soft (carpeted) floor), while in our case we ACM Press, Paris, France, 2009, p. 947. doi:10.1145/ wish to classify among 60 diferent classes. 1557019.1557122.
At the same time, we do not want to abandon these [9] Z. Cui, W. Chen, Y. Chen, Multi-Scale Convolutwo interesting methods of XAI for future applications tional Neural Networks for Time Series Classificain neurophysiology. In fact, one possible development tion, 2016. arXiv:1603.06995. of our research is to investigate early development of [10] Y. Zheng, Q. Liu, E. Chen, Y. Ge, J. L. Zhao, Time neurodegenerative diseases such as Parkinson’s. series classification using multi-channels deep convolutional neural networks, in: F. Li, G. Li, S.-w.
Hwang, B. Yao, Z. Zhang (Eds.), Web-Age InformaAcknowledgments tion Management, Springer International Publishing, Cham, 2014, pp. 298–310.
This work was partly supported by the University of [11] H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, Ferrara FIRD 2022 project ”Analisi di serie temporali da P.-A. Muller, Deep learning for time series clasmotion capture con tecniche di machine learning”. sification: A review, Data Mining and Knowledge Discovery 33 (2019) 917–963. doi:10.1007/ References s10618- 019- 00619- 1. arXiv:1809.04356. [12] B. H. Repp, Y.-H. Su, Sensorimotor synchronization: A review of recent research (2006–2012), Psychonomic Bulletin & Review 20 (2013) 403–452.
doi:10.3758/s13423- 012- 0371- 2. [13] L. Li, PLiF, 2010. URL: https://github.com/lileicc/ dynammo.