Brain-machine interfaces (BMIs) are alternative communication channels that have allowed healthy and disabled people to control external devices from brain signals. In the last decades, the growing attention towards neurorobotics has led to the proliferation of several BMI-based systems for controlling diferent devices including telepresence robots, powered wheelchairs, robotic arms, and upper/lower-limb exoskeletons. Despite the potentialities of these systems, it has emerged the necessity to create new forms of interaction between the human and the robot in order to increase the granularity of the user's commands which are, in turn, translated into specific robot's actions. In this preliminary work, we present how artificial intelligence can be exploited to design and tune a model able to convert the user's intention into continuous robot's movements.
Brain-Machine Interfaces (BMIs) provide an alternative interaction channel that does not depend
on the brain’s normal output pathways of peripheral nerves and muscles [
A key component in BMI systems is the control strategy that determines how to convert the
output of the BMI decoder into signals for the external device [
There are two diferent approaches in the literature [
The other approach is named continuous and is designed to increase the precision and
granularity in controlling external devices. However, such a paradigm is more dificult to
implement due to the non-stationary nature of the EEG and the uncertainty of the classifier
output. Indeed, it is less studied than the discrete case. Only a few approaches are proposed in
the literature based on: (a) mapping of the brain activity via linear/quadratic functions into a
continuous control signal for the robot as in [
In particular, in this work, we focus on the continuous approach based on dynamical systems proposed in [18], that, as already demonstrated, has allowed to increase the performance in controlling a telepresence robot via a motor imagery BMI and increase the coupling between the user and the devices than the discrete control. However, such an approach relies on multiple parameters that can be dificult to tune especially for a non-expert operator. The purpose of this preliminary paper is to investigate how AI can be exploited to detect a relation among the parameters with the aim of simplifying the control framework and facilitating their tuning.
For sake of completeness, this section aims to briefly introduce, from a technical point of view, the two state-of-the-art control strategies for BMI based on two classes motor imagery examined in this paper.
In the discrete approach, the raw probabilities are integrated over time according to an exponential smoothing [19, 18]. Considering the posterior probabilities in output from the classifier at time , the final control signal is computed as:
= · + (1 − ) · − 1 ∈ [0.0, 1.0] is a smoothing factor that determines the weights of the posterior probabilities at time t than the previous one. To translate the control signal into discrete high-level commands, it is compared with respect to a control threshold.
combination of two forces: The continuous control based on the dynamical system presented in [18] relies on the linear Δ = · [Φ · (− 1) + (1 − Φ) · ()] is associated with the previous state of the system, while is calculated on the current output of the decoder. The sum of the two components aims to reduce the oscillatory behaviour of the output from the classifier, help the user to deliver commands when intentional and conversely filter the false positives. Indeed, is in charge of applying a conservative contribution when the state is around 0.5, otherwise to push towards one of the two classes. is radial symmetrical with respect to 0.5 to handle the two classes in the same way. Formally, according to the design reported in [18], , represented in Fig. 2, is computed as follows: () = ⎨ ⎧⎪− ( 0.5− * )
− * (
* ( − 0.5)) ⎪⎩( 0.5− * ( − − 0.5)) ∈ [0.5 + , 1] ∈ [0, 0.5 − ) ∈ [0.5 − , 0.5 + ] while is equal to: Please refer to [18] for further details.
() = 6.4 · ( − 0.5)3 + 0.4 · ( − 0.5) (1) (2) (3) (4)
In this preliminary work, we are focusing on studying the force in the continuous framework described in Section 2.2 with the purpose of investigating the relation among its parameters. As highlighted in Equation 3, the shape of such a force strongly depends on two main parameters: • defines the size of the conservative zone. The bigger is , the higher is the “resistance" of the system to send a command. • influences the transition from the conservative to the pushing behaviours and vice versa by handling the “amount of resistance/help" from the system. The higher is , the more dificult is the change of state in the system.
Therefore, since both parameters adjust the conservative/pushing behaviours of the examined dynamical system, we hypothesise that there is a correlation between and . In this preliminary phase, we validate our hypothesis by fixing the other parameters with the values reported in Table 1 that are set coherently with the previous experiments in [18, 20] to avoid introducing confounding factors.
As regards and , first, we have applied on a pre-collected dataset a data-driven optimization that searches the best values of the two parameters by optimising a new cost function, introduced in Section 3.2, and assessing the resulting performance per each subject and combination. Then, we have applied a regression analysis on the best achieved values for each subject to find the relation between the two parameters.
In this work, we have exploited a pre-collected dataset (140.9170 min in total) related to the two-classes motor imagery task where the user was asked to imagine the movements of both Parameters Φ ℎ ℎ
Values The values of the examined parameters, where , Φ, , are from the Equation 2, ℎ and ℎ are related to the Equation 5 used to optimise the parameters and and are set according to the control threshold set in the previous studies [20]. hands vs. both feet and then received the feedback according to the predicted class (e.g., online). The data were previously collected using the motor imagery protocol available inside ROS Neuro1 framework [20] with a discrete control strategy. Such a dataset contains the data of eleven subjects (S1-S11), including in total 325 online trials for both hands and 325 online trials for both feet. Three subjects (S2, S6, S9) have no previous experience with BMI.
To optimise the values of and , we have proposed a new metric namely a cost function that rewards/penalizes the control signal by taking into account the following aspects: threshold.
score equal to zero. • It is necessary to maximise the times in which the control signal is repeatedly above the control threshold. In this way, we want to avoid/limit oscillations over and under the • To achieve a more stable control, we use a band of interest rather than a single control threshold. We want to force the control signal to pass the entire band without falling inside it. Thus, we penalise when the control signal belongs to the band by attributing a • Given the nature of BMI, as demonstrated in the previous studies, it is infeasible to deliver intentional command within 1 sec. However, it would be appreciated if the user will send the wanted commands as fast as possible. Considering such observations, we minimise the time required to deliver the commands by filtering the impracticable values (< 1 sec). An illustrative representation of the criteria behind such metric is shown in Fig. 3.
The cost function assigns a score for each combination of and according to the following three constraints: 1. for each time ∈ [0, ] with the duration of the entire trial, we compute the signed distance between the current control signal and the extremes of the band (see ((, ), ℎ, ℎ) = ⎧0 ⎪ ⎨
(, ) ∈ [ℎ, ℎ] (, ) − ℎ
(, ) ∈ [0, ℎ) ⎪⎩(, ) − ℎ (, ) ∈ (ℎ, 1]
3. the optimisation of the time for which the control signal is outside the band of interest.
Such a condition aims to make the user quickly deliver commands. To manage this The values of ℎ ℎ are reported in Table 1. 2. the temporal constraint related to the filtering of the unfeasible commands. With this purpose, we introduce the function that filters the commands due to the control signal overcoming the thresholds before 1 sec (see Fig. 3b): (, , ℎ , ℎ) = ⎪⎩1 ℎ
ℎ ∈ [
∧ * > 1 ⎧ ℎ ≤ (, ) ≤ ℎ ∀ ∈ [0, ] ⎪ ⎨* ∃(* (, ) ≤ ℎ ∨ * (, ) ≥ ℎ) (7) (9)
Thus, the final cost function is achieved by the combination of the Equation 8 and Equation 9. First (e.g., from the two first constrains), we select a set of candidates [, ] according to: [, ] = max(∑︁(∑︁ 1 ∑*︁ ((, ), ℎ, ℎ))· (, , ℎ , ℎ)) , =1 * =0 * =0 (8) Then, we choose the best * , * among the resulted candidates using the following formula: * , * =
min ( ∑︁ ∑︁ ((, ), ℎ, ℎ)) , ∈[, ] =1 =0
Furthermore, since the dataset includes online runs previously recorded with the discrete protocol, we use the final output for each trial — hit when the predicted class corresponds to the requested task vs. miss otherwise — as ground truth for these pseudo-analyses. aspect, we design the function , that is applied to couples of candidates [, ] found according to the previous criteria:
For sake of clarity, Fig. 4 shows an example of application of the cost function described in the previous section to the control signals related to a both hands trial (upper class) in the continuous modality. The band threshold used in the proposed metric is indicated in grey. For graphical reasons, we only report the control signals achieved with four combinations of and , drawn with green, purple, black, cyan colours in the continuous modality. In the same figure, we also show the control signal in the discrete case achieved via the exponential smoothing that we use as reference and is represented via dashed red colour. The control threshold for each class in the discrete case is marked with dashed blue lines. The best combination of and is associated with the green curve. Indeed, the control signal in cyan does not satisfy the temporal constraint because it overcomes the band before 1 second. The control signal in black will cause a miss - namely it overcomes the threshold for the other class (lower one). The control signal in purple is less performing than the green.
The comparison of the accuracy via the discrete (exponential smoothing) and the continuous control approaches (optimised dynamical control framework) are highlighted in Table 2 that also lists the best couples of and for each subject. Coherently with the results in [18], overall, all subjects improve their performance using the optimised dynamical control framework with the exception of S5, S6, S11. By qualitatively analysing the control signals over the diferent trials, we noticed that them drop in the [ℎ, ℎ] band suggesting the presence of involuntary commands. Further analyses will be needed in future. combinations of and in the continuous case marked with the green, purple, black, cyan colours. The corresponding control signal in the discrete case, taken as reference, is also reported with the dashed red colour line.
Subject
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 0.2 0.025 0.025 0.475 0.425 0.025 0.35 0.175 0.25 0.4 0.325 0.05 1.00 0.95 0.05 0.05 1.00 0.05 0.20 0.60 0.05 0.05
Accuracy discrete case 67.5% 72.5% 73.75% 92.5% 95% 92.5% 65% 66.67%
80% 81.11% 96.67%
Accuracy continuous case, with discrete prediction as ground truth 96.25% 92.5% 85% 97.5% 90% 80% 76.67% 68.33% 83.33% 93.33% 91.67% control framework).
The best and derived from the optimisation per each subject. Comparison of the accuracy in the discrete case (e.g., via the exponential smoothing) vs. continuous case (e.g., via the optimised dynamical
Then, from the detected best configurations of the two parameters for each subject, we perform a regression analysis in order to find a relation between them and verify our hypothesis. The results are displayed in Fig. 5. We found a second-degree polynomial function equal to = 6.6652 · 2 − 5.2772 · + 1.0884. To evaluate the goodness of the achieved model, we use 2 which measures the percentage of the dependent variable variation that our model can explain. Our model has a high 2 with value greater than 81.67%, hence it confirms that there is a relation between and .
In this preliminary work, we investigate how to optimise the continuous teleoperation of brainactuated robotic devices based on a dynamical system presented in [18] using AI. With this purpose, we propose a metric that we exploit as cost-function to optimise the parameters of the dynamical system to convert the user’s intention into continuous robot’s movements. In addition, we found a possible relation between and to facilitate their tune and simplify the system. The main limitation of this study is that the analyses were made ofline on the available dataset without involving new users. Future works will include the validation of such a hypothesis with an appropriate protocol for driving a powered wheelchair. Furthermore, we will also investigate the possibilities of keeping the same relation in the case of asymmetrical free force (diferent and for each class).
This research was partially supported by MIUR (Italian Minister for Education) under the initiative “Departments of Excellence" (Law 232/2016), by the Department of Information Engineering of the University of Padova, with the grant TONI_BIRD2020_01. [14] A. J. Doud, J. P. Lucas, M. T. Pisansky, B. He, Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface, PloS one 6 (2011) e26322. [15] S. Tortora, A. Gottardi, E. Menegatti, L. Tonin, Continuous teleoperation of a robotic manipulator via brain-machine interface with shared control, in: 2022 IEEE 27th International Conference on Emerging Technologies and Factory Automation (ETFA), IEEE Press, 2022, p. 1–8. URL: https://doi.org/10.1109/ETFA52439.2022.9921526. doi:10.1109/ETFA52439. 2022.9921526. [16] A. Satti, D. Coyle, G. Prasad, Continuous eeg classification for a self-paced bci, in: 2009 4th International IEEE/EMBS Conference on Neural Engineering, IEEE, 2009, pp. 315–318. [17] D. Coyle, J. Garcia, A. R. Satti, T. M. McGinnity, Eeg-based continuous control of a game using a 3 channel motor imagery bci: Bci game, in: 2011 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), IEEE, 2011, pp. 1–7. [18] L. Tonin, F. C. Bauer, J. del R. Millán, The role of the control framework for continuous teleoperation of a brain–machine interface-driven mobile robot, IEEE Transactions on Robotics 36 (2020) 78–91. doi:10.1109/TRO.2019.2943072. [19] E. S. Gardner Jr, Exponential smoothing: The state of the art—part ii, International journal of forecasting 22 (2006) 637–666. [20] G. Beraldo, S. Tortora, E. Menegatti, L. Tonin, Ros-neuro: implementation of a closed-loop bmi based on motor imagery, in: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2020, pp. 2031–2037. doi:10.1109/SMC42975.2020.9282968.