=Paper=
{{Paper
|id=Vol-2590/short11
|storemode=property
|title=Development of a Hardware-Software System for a Bionic Prosthesis of a Human Hand
|pdfUrl=https://ceur-ws.org/Vol-2590/short11.pdf
|volume=Vol-2590
|authors=Kirill Markin,Pavel Rozhkin,Alexey Platunov
|dblpUrl=https://dblp.org/rec/conf/micsecs/MarkinRP19
}}
==Development of a Hardware-Software System for a Bionic Prosthesis of a Human Hand==
https://ceur-ws.org/Vol-2590/short11.pdf
Development of a Hardware-Software System for
a Bionic Prosthesis of a Human Hand
Kirill Markin1[0000−0002−1933−4604] , Pavel Rozhkin2[0000−0002−6037−846] , and
Alexey Platunov3[0000−0003−3003−3949]
1
ITMO University, Kronverksky prospekt, Saint Petersburg, 197101, Russian
Federation
kirill1997 markin@mail.ru
2
blackiiifox@gmail.com
3
aeplatunov@itmo.ru
Abstract. For a modern person, the loss of any limb or organ is a big
problem. Not everyone can measure it and try to remove these restric-
tions. Thanks to modern means and technologies, it becomes possible
not to be a person with disabilities, but to purchase augmented ones.
Already, bionic prostheses of the upper extremities use myoelectric po-
tentials for control, have wireless data transfer technologies and their
functionality will soon be inferior to modern smartphones. As a result,
hand prostheses begin to constitute a powerful tool for improving the
functional capabilities of human limbs, thereby improving the quality of
the user. The paper presents the goals and objectives set during the devel-
opment of a hardware-software system for an automated reprogrammed
hand prosthesis for people with disabilities, as well as the rationale for
its implementation. Parts of the developed system, technical characteris-
tics and software and algorithmic solutions used in the development are
described. In conjunction with direct performing motor mechanisms and
a full-scale model of a bionic prosthesis, the development is intended for
use in the field of medical rehabilitation of the upper limbs. The system
under development will allow you to customize gestures implemented
by the prosthesis through an application on a personal computer. Di-
rect control over the prosthesis can be carried out both on the basis of
bioelectric potentials and on the basis of additional human-machine in-
terfaces, in particular, voice and graphic control interfaces in a mobile
application. Through the use of additional interfaces, it will be easier
for a person to go through the initial rehabilitation period in which he
needs active muscle training to use myoelectric sensors. Also, the sys-
tem provides the opportunity to use a bionic prosthesis for those people
whose nerve endings were damaged and are no longer able to conduct
neural impulses. In the developed system, it was possible to configure up
to 1000 gestures, which the user can independently create and modify.
The system was tested on a patient with an amputated left upper limb
and was noted by him and rehabilitologists as promising.
3
Copyright c 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0).
2 Markin Kirill Anatolevich et al.
Keywords: Robotic systems · Automated reprogrammable computing
systems · Bionic prostheses · Embended system ·
1 Introduction
Unfortunately, in modern society the percentage of people with disabilities is
constantly growing [6]. On average, about one million limb amputation opera-
tions take place in the world per year. Help for people with amputated limbs is
carried out by specialists in the field of rehabilitation and prosthetics.
Over the past decades, advances in the development of 3D printing, light and
powerful engines, capacious batteries, as well as the growth of the capabilities of
electronic computing devices have created the conditions for the appearance of
prostheses that are as close as possible in their functions to human limbs to such
an extent that they can be replaced by cosmetic and traction types of prostheses
[8] of the upper limbs came bionic prostheses [4], which are able to restore the
functionality of the lost limb [2]. Already there are bionic prostheses resembling
a mini-computer [1].
All today’s companies developing bionic prostheses are focused on two areas
of development [10]:
– Cheaper prosthesis.
– Improving the prosthesis control system.
The most interesting area of study is the management system, since work in
its field has just begun and it has a long way to go to become more convenient
to use. The main problems of the control system of modern bionic prostheses
are [9]:
– Lack of feedback in most prostheses [3].
– A limited number of executable commands.
– The inability to use a prosthesis with a myoelectric control system for people
with neuropathy (lack of nerve conduction).
– A long period of post-traumatic rehabilitation associated with increased dif-
ficulty in training the necessary muscle groups for myoelectric control. The
training will require a lot of concentration and spatial thinking skills, so
necessary for working with it.
The prosthesis being developed will reduce the period of post-traumatic re-
habilitation of a person and will make it possible for those people who are not
available to use myoelectric sensors.
2 Aims and objectives
The aim of this work is the development, research and testing of the hardware-
software system of an automated reprogrammable bionic prosthesis of the hand,
with modular control. The prosthesis can be controlled both on the basis of
Title Suppressed Due to Excessive Length 3
bioelectric potentials and on the basis of additional interfaces. Through the use
of additional interfaces, it will be easier for a person to go through the initial
rehabilitation period in which he needs active muscle training to use myoelectric
sensors. Also, the system will provide the opportunity to use a bionic prosthesis
for those people whose nerve endings were damaged and are no longer able to
conduct neural impulses, which are required for using myoelectric sensors.
To control the bionic prosthesis, it will be possible to use controls based on
the graphical user interface, voice control, as well as based on bioelectric control
using myoelectric sensors. The presented interfaces will convert the recognized
prosthesis commands into control signals for the movers to form gestures for the
prosthetic hand. In addition to providing the basic functions of the prosthesis,
such as squeezing / unclenching individual fingers and the arm as a whole, the
modular control approach will allow the user to independently configure and add
new functionality for personal use by creating or editing gestures implemented
by the prosthesis.
3 Scientific novelty of the solutions proposed in the draft
1. The proposed use of natural language recognition for the control system
of the prosthetic arm has no analogues in the areas of rehabilitation and
prosthetics of the upper limbs of a person according to the analysis of open
sources.
2. The developed system for storing, transmitting and performing prosthetic
gestures allows us to abstract from the implemented system of motor mech-
anisms and control devices and to create new ones or change old gestures.
3. A modular control system based on the interaction of myoelectric sensors,
voice and graphic control, will significantly expand the functionality of the
system.
4 Justification for the need for development
In process rehabilitation, during the management of the prosthesis, the patient
must think what kind of simple movement he would like to make with his phan-
tom arm, for example, bend his arm at the elbow, squeeze his hand into a fist,
etc. Myoelectric sensors read signals from spinal motor neurons and convert them
into prosthetic commands. The problem with prostheses is that their sensors are
attached to the remains of the muscles at the patient’s amputated limb. Twitch-
ing of these muscles set the prosthesis in motion. But it often happened that the
muscles were damaged too much, due to which the movements of the prosthesis
were very limited. About 50 percent of patients simply refused the prosthesis
due to its incorrect work. That is why many experts have long been trying to
find an alternative approach to solving this problem.
The system of automated reprogrammable control of the prosthetic arm is a
system based on voice and combined (voice-myoelectric) control. No more trying
to recreate or trying to simulate the movements of a phantom hand. Instead,
4 Markin Kirill Anatolevich et al.
you can pronounce a number of simple phrases and not necessarily in a strictly
specified form or case, as well as realize even more complex movements as a
result of a combined command.
5 Description of the hardware-software system of a bionic
prosthesis of a human hand
The structure of the hardware-software system of the prosthetic hand can be
divided into three main parts (see Fig. 1):
Fig. 1. Structure of hardware-software system of bionic prosthesis of a human hand.
1. The full-scale model is a complex consisting of a model of the forearm from
the point of view of ergonomics, as close as possible to the appearance of a
real forearms and sensors and actuators integrated into it devices;
2. The microcontroller part of the control system provides control of the me-
chanical effect of the drives on the prototype part of the prosthesis, taking
into account the given command, as well as reading / transmitting data
between the sensors and the controller;
3. The software part of the control system is the interface of the system’s inter-
action with the user and provides the ability to control the prosthetic hand,
create new actions performed by the prosthesis, and change the current ones.
5.1 Full-scale model
The hand is a complex system and is of particular importance to humans, and
the anatomy of the hand is characterized by the presence of small bones and
articulating joints of various types. As a result of the analysis of the anatomical
features of the arm, the minimum necessary group of muscles, joints and tendons
was selected that were sufficient to perform basic movements:
Title Suppressed Due to Excessive Length 5
1. Muscles - extensor and superficial flexor of the fingers, dorsal interosseous,
vermiform;
2. Joints - distally and proximal interphalangeal, metacarpophalangeal;
3. Tendons - superficial and deep.
Fig. 2. The model of the prosthesis in the horizontal section.
Based on the assumptions made, a graphic 3D model of the hand prosthesis
was developed in Google SketchUp 2018 and Autodesk 3ds Max 3D modeling
environments. When considering the horizontal section of the prosthesis (see
Fig. 3), you can see that the model provides channels that simulate the work
of the deep flexor tendons of the finger flexors (A). For the supporting part of
the model are the elements of the articulation of the phalanges of the fingers be-
tween themselves, which mimic the distal (D), proximal interphalangeal (C) and
metacarpophalangeal (B) joints. A group of servos connected to the phalanges
of the fingers by small cables that mimic the functioning of the above muscles
and tendons is responsible for the model [7].
5.2 Microcontroller control system
The STM32F407VET6 microcontroller [5] is used to control the executive devices
and process incoming and read information from sensors and a communication
6 Markin Kirill Anatolevich et al.
device. Development was conducted in the C language of the 2011 standard using
the HAL library for the configuration of MK peripheral devices. This solution
allows you to port the code to different versions of STM. Based on the technical
parameters and design features of the system, the microcontroller implements
the following tasks (see Fig. ??):
Fig. 3. Functional-modular structure of the controller of the hardware-software system
for controlling the prosthesis of a human hand.
1. Providing messaging with user interface. Bluetooth is used as the transmis-
sion interface, as Wireless connection is required both on a mobile device
and on a personal computer;
2. Receiving and processing readings of myoelectric sensor. As myoelectric sen-
sors are used Ottobock Electrode 13E200. To read the readings from the
sensor, the ADC is built into the microcontroller with a sampling frequency
of 160 Hz (maximum sensor frequency is 60 Hz);
3. Storage and processing of gestures programmed on a personal computer.
Gestures are stored on the SD card located in the prosthesis. Work with the
sd card file system is carried out through the FatFs library;
4. Management of actuators of the full-scale model. In particular, the display
of information for the user, engine driver control via CAN bus;
Title Suppressed Due to Excessive Length 7
5.3 Interfaces of human interaction with the system
User interaction with prostheses is carried out through the use of two applica-
tions. The first is the HandControl desktop application, which provides the user
with the ability to reprogram the actions performed by the prosthetic arm. The
second is an android application that acts as a control device for the prosthetic
arm. It contains the choice of the prosthesis operating mode (combined, direct
or myoelectric control mode), sending a request for the execution of gestures, as
well as the implementation of voice control.
The desktop application is developed in C# using following means:
1. Technology WPF (Windows Presentation Foundation), which is a replace-
ment for WinForms technology for developing graphical applications in C#.
The technology is based on the use of DirectX when rendering graphics in
an application and using XAML to describe application graphics.In addi-
tion, this technology allows you to most effectively implement the MVVM
programming pattern, which was used in this application;
2. Framework Material Design for styling and extension functionality of the
basic elements of WPF;
3. The Json.NET Framework to enable conversion of .Net objects to their JSON
equivalent and vice versa by matching the names of the properties of the
.NET object with the names of the JSON properties and copying them;
4. Framework Fody, which implements PropertyChanged events in classes that
implement the INotifyPropertyChanged interface.
6 Findings
The system under development expands the capabilities of existing prosthetic
analogues by integrating voice control and makes it possible to abstract the
development of the prosthesis model from the control system. In addition, users
of prostheses will be able to pass the adaptation period to the bionic prosthesis
much faster at the initial stages due to the combined control system, when they
can refuse myoelectric control.
References
1. Bioelektricheskij protez “Stradivari” [Bioelectric prosthesis ”Stradivarius”],
https://motorica.org/protezirovanie/stradivary. Last accessed 29 Dec 2019
2. Cipriani C., Controzzi M., Carrozza M.C.: The SmartHand transradial prosthesis.
J. NeuroEng. Rehabil. 2011, 8, 1–14.
3. Cipriani, C., Zaccone, F., Micera, S., Carrozza, M.C.: On the shared control of
an EMG-controlled prosthetic hand: Analysis of user–prosthesis interaction. IEEE
Trans. Robot. 2008, 24, 170–184.
4. Ermakova E. Ju. Vidy izdelij dlja protezirovanija kisti ruki [Types of prod-
ucts for prosthetics hands], https://ortocure.ru/kosti-i-sustavy/drugoe/protez-
ruki.html. Last accessed 2 Nov 2019
8 Markin Kirill Anatolevich et al.
5. Geoffrey B.: Discovering the STM32 Microcontroller. Indiana University, USA
(2016)
6. Kuznecov R. A. MSJe i invalidnost’ pri amputacijah
konechnostej [ITU and disability in limb amputations],
http://www.invalidnost.com/publ/mediko socialnaja ehkspertiza pri nekotorykh
zabolevanijakh/mseh i invalidnost pri amputacijakh konechnostej/2-1-0-460. Last
accessed 2 Nov 2019
7. Petrov V. G.: Tehnologija izgotovlenija protezov verhnih konechno-stej [The tech-
nology of fabrication of upper limb prostheses.]. Gippokrat, Russia (2008)
8. RoboSapiens BIONICHESKIE PROTEZY: ISTORIYA, PRINCIP RABOTY,
POSLEDNIE DOSTIZHENIYA [BIONIC PROSTHESES: HISTORY,
PRINCIPLE OF WORK, LATEST ACHIEVEMENTS] https://robo-
sapiens.ru/stati/bionicheskie-protezyi/ Last accessed 29 Dec 2019
9. Stuart Nathan Future prosthetic: towards the bionic human
https://www.theengineer.co.uk/future-prosthetic/
10. Zhvansky E. Ne opuskajte ruk: pochemu bionicheskie protezy ne stanovyat-
sya dostupnee? [Do not give up: why bionic prostheses do not become more
accessible?] https://www.forbes.ru/tehnologii/345329-ne-opuskayte-ruk-pochemu-
bionicheskie-protezy-ne-stanovyatsya-dostupnee Last accessed 29 Dec 2019