The paper analyzes the problem of developing and implementing a sensory feedback system into the structure of lower limb prostheses, which is especially relevant for improving motor functionality, maintaining balance, mobility and comfort of the prosthesis users. For the designed sensory feedback system, the concept of using IoT and machine learning technologies with indirect adjustment of system parameters using a smartphone was used. For this purpose, it is proposed to use a piezo transducer as a sensor, which will be placed in a specialized insole. The latter will be placed in the shoes in which the prosthesis user will move. The signal from such a sensor will contain information about dynamic changes in the load on the prosthesis during walking. It is also proposed to use miniature vibration motors as actuators in such a system, which will be placed in the stump-receiving sleeve. It is proposed to add additional modules into the structure of the designed system, which provide the possibility of wireless transmission of signals from the sensor to a smartphone, on which the system operation will be configured in a specialized mobile application. It is also possible to individually configure the actuator operation parameters by the user, which in the future will allow the user to distinguish the type of surfaces on which he moves without using visual control.
Modern lower limb prosthetics are rapidly developing in response to the growing number of amputees, particularly in Ukraine, where this problem has become particularly acute in the context of a full-scale war. The loss of a lower limb leads not only to impaired motor function, but also to the loss of proprioceptive feedback - the sense of position, force of pressure and interaction with the supporting surface, which significantly reduces the efficiency and safety of movements.
Classical mechanical prostheses do not provide adequate sensory feedback, which limits the user's adaptation, reduces his confidence in walking, makes it difficult to maintain balance and increases energy expenditure during movement. Bionic systems, which are capable of two-way communication with the user's nervous system, open up new prospects in medical rehabilitation, allowing to significantly improve the functionality of the artificial limb.
The implementation of sensory feedback in lower limb prostheses involves registering mechanical loads, processing the corresponding signals and transmitting tactile information back to the user. This requires the creation of a comprehensive system that would include appropriate sensors, amplifier modules, actuators, as well as appropriate software. Solutions that combine piezoelectric sensors, energy-efficient amplifiers, as well as compact and wearable designs that can be integrated into shoes or the prosthesis itself are particularly promising.
Despite the rapid development of prosthetics, users of lower limb prostheses face a number of
significant difficulties [
The lack of sensitivity to pressure, position and surface relief limits adaptation to changes in
movement conditions, increases the risk of falls, and makes fine motor coordination impossible (for
example, in sports or fast walking) [
The integration of prostheses with the nervous system is a key condition for overcoming the
limitations of classic mechanical prostheses and achieving a level of functionality close to that of a
natural limb. Such integration involves the use of neurointerfaces that provide a two-way
connection between the human body and the artificial limb: recording electrical signals from
muscles or nerves to control the prosthesis and transmitting sensory information from the
prosthesis back to the brain. Invasive neurointerfaces are particularly promising, which, thanks to
direct contact with neural structures, allow for high accuracy in controlling movements and
creating feedback [
A necessary component of such systems is the development of highly sensitive electrodes and sensors, which must be biocompatible, miniature, energy-efficient and able to function in the dynamic conditions of the human body. Artificial skin that imitates tactile sensitivity is one of this technology development directions. An important role in the functioning of bionic prostheses is played by machine learning algorithms that analyze neural signals, adapt the prosthesis to the individual motor characteristics of the user, and also form a feedback sensory signal understandable to the brain. Thus, an artificial system is created that allows a person not only to control the prosthesis, but also to "feel" it.
The implementation of sensors in the lower limbs has its own specific features that differ from
the upper limbs due to fundamentally different functional requirements. For the lower limbs, the
most important are the feeling of pressure, support, the position of the foot and lower leg in space
(proprioception), as well as information about the nature of the surface. These sensations are
critical for maintaining balance, stability during walking, preventing falls, and adapting to uneven
terrain [
INAIL Össur + Integrum
AB
EPFL, Lausanne Ottobock, Germany
Year 2014 2019 2020 2021
More advanced systems can use electrical stimulation of nerves in the stump to induce a sensation of pressure or even the position of the phantom foot, which greatly improves balance and allows for more natural adaptation to changes in the environment. Individual tasks of creating sensory elements correlate with papers [25-29].
Unlike fine motor skills in the upper limbs, where tactile sensations are important, information about weight distribution and stability is prioritized in the lower limbs. Therefore, tactile sensors are typically placed on the sole of the prosthetic foot (e.g., in the heel, forefoot, and toe areas) to measure pressure points and intensity. This information can then be transmitted to the user through vibrating elements located on the stump or through mechanical stimulation in a receiving sleeve.
The designed system must provide the ability for users to feel the occurrence of loads on the lower part of the prosthesis when it comes into contact with a hard surface, for example, when walking. For this, two options for implementing such a system are possible. In the first case, the sensitive element - the sensor - will be placed in the base of the prosthesis, for example, in the heel area. In the other case, the sensor will be placed in a special insole that will be inserted into the shoes that the user will use. The first case is universal, but difficult to implement, since it will require additional technological operations during the manufacture of the prosthesis itself. This, in turn, may impose certain restrictions on the reliability and durability of both the prosthesis and the sensitive element, since the structural integrity of the prosthesis base will be violated, and the sensitive element will be difficult to perform in such a way that it can be quickly replaced in case of damage or failure. The second case is more promising, since in any case the user will use certain shoes for walking comfort, better visual perception by others, etc., and accordingly the sensitive element itself can be made independent of the prosthesis and to a certain extent universal. For this purpose, shoe insoles can be used, which will be specially designed to accommodate the sensor of the designed system and will be to a certain extent universal for one user, since they can be freely inserted into different shoes depending on the user's needs or desires. Also, such insoles will be completely replaceable, which will be convenient for users in the event of a sensor failure.
It is proposed to use a piezoelectric transducer as a sensor. Due to the piezoelectric effect, this transducer will perceive external loads and generate a signal in the form of an electric charge, which will be proportional to the magnitude of the load on the sensor and, accordingly, on the prosthesis and the user's stump. This sensor is a transducer of dynamic loads, that is, it will respond to changes in the load on the prosthesis. However, when the loads are constant (static), this sensor will not generate a charge. This is especially useful for the designed system, since the user does not need to constantly feel pressure on the prosthesis, but only when the prosthesis touches the surface during walking. If the sensor constantly generated a load signal and the user constantly felt it (for example, when standing on the prosthesis), this would constantly distract the user and these sensations would be redundant.
The signal from the sensor is proposed to be amplified at the next stage and to ensure real-time operation immediately fed to the actuator. As an actuator, it is proposed to use miniature electromechanical transducers that will form small vibrations that the user will perceive with skin mechanoreceptors. In this case, such actuators can be placed either in a special cuff that will be applied to the free area of the stump skin, or inside the stump-receiving sleeve of the prosthesis itself. As actuators, piezoelectric transducers of a similar type can be used but with a much lower mechanical resonance frequency. However, in this case, it is necessary to install special power amplifiers due to the complexity of exciting such actuators.
In the designed system, it is proposed to use miniature vibration motors from mobile phones. Despite their small size, these motors also have a small current consumption, and their control is very simple.
This sensor was connected to the input of the OWON SDS 1022 digital oscilloscope. Fig. 2 shows the view of the oscillograms on the screen of this oscilloscope with small mechanical effects on the sensor surface.
Fig. 2 shows that the level of the generated signal is significant, which makes the system itself quite sensitive.
However, the main disadvantage of this sensor when used in the designed system is that it has a rather bulky housing and it will be difficult to place it in the insole. Therefore, another transducer was additionally analyzed - KP27242, which is a housingless piezo transducer with an outer diameter of 27 mm. Its appearance is shown in Fig. 3.
The output signal level for such a sensor was also estimated under the same external influences as for the previous case.
However, since the sensor itself does not have a housing, a housing was designed for it to ensure mechanical strength and printed on a 3D printer from ABS filament.
Also, to transfer the load to the sensor itself, a special cover was developed for the pre-designed housing, which contains cylindrical protrusions inside to transfer the load to the sensitive element. This cover was also made by 3D printing from TPU-90 filament. Due to this, it turned out to be elastic, and when loaded, its internal cylindrical elements deform and limit the impact on the sensitive element surface. Thus, mechanical destruction of the piezoelectric element due to excessive loads on its surface is practically impossible. Fig. 4 shows the view of the assembled sensor structure.
Fig. 5 shows the view of the signal oscillograms from such an assembled sensor, which were recorded for the same external influences as for the previous case.
As can be seen from Fig. 5, the output signal level for the KP27242 element is higher compared to the previous sensor. Therefore, it was used as the basis of the designed system.
As can be seen from the above oscillograms, the signal at the output of the piezoelectric transducer when using a flexible cover to transmit dynamic loads is a simple low-frequency signal, the peaks of which fall at the very moments of maximum load on the element. Therefore, a low-frequency audio amplifier can be used as an amplifier. Different types of amplifiers were analyzed, in particular classes A, B and D. Additionally, taking into account the fact that to ensure the miniaturization of the system itself, it must be powered by lithium-ion batteries with a voltage of 3.7 V, a class D amplifier was used. Amplifiers of this class are pulse-type amplifiers and can develop significant output powers when operating from very low voltages. In addition, in the pulse mode, the dissipated powers are negligible, so such amplifiers differ in significantly smaller overall dimensions and power consumption.
At the prototyping stage, an amplifier based on the PAM8403 chip was used. Its main technical characteristics: - Amplifier class: D; - Output power: 2 x 3W; - Load resistance: from 4Ω to 8Ω; - Supply voltage: from 2.5V to 5V; - Limit supply voltage: 5.5V; - Dimensions: 18.5 x 21.1 mm.
According to the described technical characteristics, this microcircuit is ideally suited in class, supply voltage and output power for the designed system.
Also, two basic designs of amplifiers based on this microcircuit are available on the electronic components market. They are shown in Fig. 6.
According to Fig. 6, the second option differs from the first only in the presence of an input signal level regulator. And at the previous stage, it was used to prototyping the designed system. In the final version, it is proposed to use the first option, since it has significantly smaller dimensions and weight.
Taking into account the operating modes of the designed system and the requirements for energy consumption (low supply voltage and current consumption), it is proposed to use miniature DC vibration motors from mobile phones in the designed system. They should be placed in a stump-receiving sleeve or in a special cuff and mechanically affect the surface of the user's skin, which will ensure the feeling of the surface when walking on the prosthesis. As an option, a SHICOH N7 micromotor was used. Its internal resistance is approximately 35 Ohms.
The view of this motor is shown in Fig. 7.
To evaluate the performance of the designed system, a model of the insole in .stl format, which is in the public domain, was used. Using the Solidworks program, the thickness of the insole was increased and a place was made in its lower part (near the heel area) to place the sensor. The insole itself was made by 3D printing from TPU-90 material. Its final view is shown in Fig. 8.
To use the last type of sensor (Fig. 4), a cover was made separately so that it could freely enter the base of the insole and form a solid object with it (Fig. 9).
Fig. 10 shows the view of the sensor, amplifier and vibration motor connected to each other. With the same impact on the sensor, the vibration motor starts to work and create vibrations, which will later be perceived by the user's skin through mechanoreceptors.
Thus, according to the selection of constituent elements the system structure shown in Fig. 11 was implemented as a prototype.
All components – sensor, amplifier and vibration motor – were integrated and tested as a single system. The results of experimental verification confirmed the correctness of the choice of technical solutions, sufficient sensitivity to dynamic loads and functionality of the system in conditions close to real operation.
Thus, an effective, compact and energy-saving system for providing a sense of load in a prosthetic limb was developed, which opens up prospects for further clinical adaptation and improvement of sensory feedback systems in prosthetics. Also, as can be seen from Fig. 11, such a system is quite primitive and represents a direct amplification channel without additional feedback, and it does not provide for additional settings for the operation of actuators except for the vibration force of vibration motors by adjusting the gain of the RAM8403. This limits its functionality and reduces stability.
Also, in real conditions, a significant number of factors affecting the operation of the sensor of such a system should be taken into account. In this case, the output signal of the sensor can be described quite approximately by expression (1): (1) here:
– the output signal, which will be a random process, and can be presented as a function of separate groups of influencing factors, which may include the following: – the group of factors determined by the dynamics of the impact on the prosthesis of the user's mass and dimensions during walking;
– the group of factors determined by the dynamics of changes in the load force on the prosthesis depending on the user's active lifestyle; – the group of factors determined by the dynamism and activity of the user's movements during walking, walking style, motor activity (slow/fast walking, running, etc.); - the type of shoes used by the user and the type of shoe sole (sneakers etc.); - a group of factors determined by the parameters of the surface on which walking is performed (flat hard surface, flat soft surface, grass, sand, small stone, surface with potholes, etc.). The first three groups of factors are individual biometric parameters of each individual user.
Thus, it is necessary to configure the designed system for each individual user for each group of described factors. It is practically impossible to do this using classical methods, since this would require registering a set of signals, processing them using appropriate methods within an adequate method of mathematical description, highlighting informative signs for each group of factors (1) and forming the corresponding actuator control signals. In this regard, the optimal use of machine learning methods and combining all components of the sensory feedback system into a single whole based on IoT technologies is optimal. Modern approaches to the use of data transfering, machine learning and predictive analytics in medical and technical systems, including prosthetics, are actively being developed, which can significantly improve the quality of sensory feedback and user adaptation [30-34].
For the designed sensory feedback system, the concept of using IoT and machine learning technologies with indirect adjustment of system parameters using a smartphone was used. This was partly described in [35]. The structure of such a system is shown in Fig. 12.
Accordingly, unlike the structure in Fig. 11, a preamplifier module is additionally introduced, after which a Bluetooth module is installed to implement a wireless data exchange channel with a smartphone. The smartphone itself is intended for performing basic settings of the sensory feedback system by the user and their subsequent storage in the controller, which after setting will automatically generate the corresponding actuator control signals during constant use of the prosthesis without connecting a smartphone.
During the setting process, the signal from the preamplifier output will be sent to the smartphone, on which all necessary calculations and system operation settings will be performed. To do this, it is necessary to develop a specialized mobile application with the ability to specify the main output parameters by the user (according to the first three groups of factors (1)). The application itself will calibrate the input signal values, since, for example, the range of the sensor output signal will depend on the user's weight, and the rate of change of this signal will depend on age and activity. The ability to enter this data will be required during registration and the first login to the mobile application. Regarding the fourth group of factors (1), the ability to change the type of footwear will be constantly active in the application, since this will affect the change of correction factors that will determine the parameters of the signal that will go to the actuators. Regarding the last group of factors, their settings in the mobile application will be performed last. At the same time, the smartphone screen will display the inscription “Select surface type”, as well as several active buttons with the names of a specific type of surface, for example: “flat hard surface”, “flat soft surface”, “grass”, “sand”, “small stone”, “surface with potholes”, etc. When clicking on the first button with the surface type “flat hard surface”, the application will ask the user to take, for example, 20 steps on such a surface and in the process the signal from the sensor will be registered. Next, the application will ask the user to confirm the completion of 20 steps and return him to the previous settings menu and ask him to select the next type of surface and also take 20 steps. This will be repeated until the application receives signals from the sensor about all the provided surface types. Taking into account the possibility of a long stay of the user in the room (for example, a rehabilitation center), the application will provide the ability to ignore individual surface types (since it will be impossible to receive the corresponding signals from the sensor). Now, having signals from the sensor when the user moves over different types of surfaces based on machine learning methods, the application will form groups of signal informative signs for each type of surface and transmit them via the reverse Bluetooth channel to the controller, which will save them. In the future, when moving over different surfaces (for example, when moving from a flat road to grass), the controller will be able to automatically identify this type of surface and adjust the parameters of the actuator control signal. After that, the user will be able to individually configure the actuator operation parameters for different types of surfaces in the mobile application. So, in the mobile application menu, you can select the “surface sensing settings” button, after which buttons with the types of surfaces that the user selected at the previous stage will appear on the smartphone screen. When clicking on each button, the user will see several options on the screen with the ability to set the actuator vibration frequency, vibration intensity, additionally set the depth and frequency of the actuator modulation, in the case of a multi-actuator system, it will be possible to assign the operation of each individual actuator or group of actuators for each type of surface. These settings will be necessary and important for creating comfortable effects for the user from the actuators on the stump skin mechanoreceptors and the possibility in the future of unambiguous identification of the type of surface on which he moves. Actually, this will provide the principle of sensory feedback without the need for constant visual control of the movements performed. Actually, the movements of the user with such a prosthesis should become more natural, since, albeit indirectly, his body will receive information about the surface and the features of moving on it.
This paper examines the scientific and technical principles of designing, implementing, and functionally integrating a sensory feedback system in lower limb prostheses, which is a relevant area of modern biomedical engineering. It has been established that the introduction of sensory feedback significantly improves motor functionality, increases balance, mobility, and comfort of users, and also opens up new opportunities for military and sports rehabilitation.
A variant of a tactile sensory feedback system using a KP27242 piezoelectric sensor, an energyefficient class D amplifier based on the PAM8403 chip, and an actuator in the form of a SHICOH N7 vibration motor has been proposed. Design options for placing the sensor have been analyzed, of which its installation in an adaptive insole has proven to be the most effective. The experimental test confirmed the operability, energy-saving properties and functional efficiency of the developed system in modes close to real operating conditions.
However, to expand the functionality and stability, it is proposed to introduce additional modules into the designed system structure that provide the possibility of wireless transmission of signals from the sensor to a smartphone, on which a specialized application will be used to configure the system depending on the characteristics of the effects of external factors, such as the activity of the prosthesis user, the type of surface on which the user moves, the type of footwear used. It is also possible to individually configure the parameters of the actuator or group of actuators by the user, which in the future will allow the user to distinguish the type of surfaces on which he moves without using visual control. This will actually improve motor functionality, increase balance, mobility and comfort of prosthesis users with such a sensory feedback system.