=Paper=
{{Paper
|id=Vol-2564/shortarticle_4-CRoNe2019
|storemode=property
|title=Computational vision and machine learning to evaluate Metacarpophalangeal and Interphalangeal deviation in fingers for clinical purpose
|pdfUrl=https://ceur-ws.org/Vol-2564/shortarticle_4-CRoNe2019.pdf
|volume=Vol-2564
|authors=Matías Salinas,Astrid Cancino,Alejandra Zazueta,Rodrigo Salas
|dblpUrl=https://dblp.org/rec/conf/crone/SalinasCZS19
}}
==Computational vision and machine learning to evaluate Metacarpophalangeal and Interphalangeal deviation in fingers for clinical purpose==
https://ceur-ws.org/Vol-2564/shortarticle_4-CRoNe2019.pdf
Proceedings of the 5th Congress on Robotics and Neuroscience
Computational vision and machine
learning to evaluate
Metacarpophalangeal and
Interphalangeal deviation in fingers
for clinical purpose
Matías Salinas MSc1* , Astrid Cancino MSc1 , Alejandra Zazueta MSc1 , Rodrigo Salas
Phd2
*For correspondence:
matias.salinas@postgrado.uv.cl 1 Programa Doctorado en Ciencias e Ingeniería para la Salud, Universidad de Valparaíso;
(FMS); astridcancino@gmail.com
(FS)
2 Escuela de Ingeniería C. Biomédica, Universidad de Valparaíso
Present address: † Doctorado en
Ciencias e Ingeniería para la Salud,
Universidad de Valparaíso, Chile
Abstract Metacarpophalangeal deviations can be present from congenital diseases, nerve
injuries, direct trauma such as fractures, autoimmune arthropathies such as that associated with
lupus erythematosus (Jaccoud), to rheumatoid arthritis (RA), which is an inflammatory disease of
joint components with varying degrees of destruction of the small joints of hands, producing a
deformation that is evidenced in the deviation of the axis of the fingers having an impact on the
functionality of the hand. The application of computational vision and machine learning in the
quantification of finger angulation and joint thickening, can provide an objective way to record
measures that are routinely performed subjectively if there is no specialized equipment. These
measurements can be very useful to have an adequate pre and post intervention record or simply
have a standardized record in the case of degenerative pathologies. Through digital image
processing, it is possible to define markers that allow us to calculate the angulation and width of
the metacarpophalangeal segments at the service of the clinic or useful in research. The results
generated were collected in a csv file showing 80% effectiveness in images with arthritis.
Introduction
The activity of the hands in daily life is extremely delicate and precise, such as writing, painting
or playing an instrument. They also allow us to perform heavy labor, such as digging with a
shovel or hammering. The manual skills are almost imperceptible in health condition until the
appearance of a hand injury or the manifestations of a degenerative or autoimmune disease begins.
In hand pathology that show alteration of the axis of the fingers, the anthropometry of hands is
responsible for studying the dimensions and orientation of the human body and its relationship
with its environment. The field of anthropometry is very extensive, it could be adopted in any
field, from manufacturing industries, tool design to topics of medicine and sports. The most
commonly used measurements of the hand, have to do with the manufacturing field of tools,
operating room items, gloves, specialized handles by hand size, equipment for disabled people,
prostheses even in the pre and post operative clinical records of hands, tracking of finger fractures
and degenerative joint pathologies. The incorporation of the metacarpophalangeal and interfalange
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Proceedings of the 5th Congress on Robotics and Neuroscience
deviation metric would be very useful in the specialized field of medicine and rehabilitation of
traumatic and degenerative pathologies such as the sequels of lupus and arthritis. This measure
is not widely developed in a standardized way as the measures applied to the industry but can
provide great support by being quantified in an automated way(Jonsson, 2017). Machine learning,
which has shown great success, poses an opportunity in the field of medicine and rehabilitation
with a quantification that leads to better records before and after the intervention and monitoring
of joint pathologies. Learning machines that extract characteristics of a problem and then relate it
to an answer have shown a high degree of adaptation to dynamic events, which will allow us to
work with different types of hands and pathologies.
Results
Two databases of healthy hands were obtained to extract characteristics using the developed
algorithm.
From the collection of images of hands with arthritis from the article by (Jonsson et al., 2012),
automated feature extraction was achieved to obtain anthropometric measurements of the hands
for the purpose of clinical record of evolution in a supervised way in degenerative pathologies.
For the test stage in hands with pathology described below, it was applied in 10 random hands
with arthritis were processed, the result is shown in Table 1, the collection of images contain 28
arthritic hands from 14 users, with a degree of deformity greater than the healthy hands. The results
indicate that from 28 hands, it was possible to extract in 23 the characteristics of interest such as
length, width and angulation of each of the fingers, while in the remaining ones it is necessary
to improve the code to adapt to all possible image records, such as for example the use of rings,
watches, among others.
Table 1. Finger Deviation Results
User_ID TD/TL ID/IL MD/ML RD/RL LD/LL
𝟣 5.72°/4.65/2.01 7.66°/7.94/2.0 6.76°/8/1.95 7.66°/7.93/1.8 5.71°/7.41/1.53
𝟥 13.15°/4.81/1.94 7.6°/7.83/1.92 0.7°/8.70/1.8 1.36°/8.38/1.66 17.9°/7.63/1.49
𝟧 7.66°/5.4/1.89 7.72°/8.13/1.81 10.68°/8.76/1.67 15.15°/8.70/1.5 15.32°/6.7/1.24
𝟫 9.88°/4.53/1.86 11.51°/7.9/1.76 1.36°/8.3/1.61 8.25°/8.3/1.46 3.63°/7.52/1.3
𝟣𝟣 6.16°/4.51/1.87 8.21°/7.36/1.91 0°/7.4/1.89 5.81°/7.2/1.57 14.64°/7.08/1.39
𝟣𝟥 3.21°/5.07/2.14 0.04°/7.24/2.02 9.68°/8.04/1.93 0.89°/7.6/1.71 17.6°/7.22/1.47
𝟣𝟫 22.38°/4.12/1.56 2.7°/7.2/1.84 2.33°/8/1.6 3.58°/7.95/1.31 20.72°/6.78/1.14
𝟤𝟣 5.51°/4.21/1.64 0°/7.95/1.66 0°/8.38/1.61 7.66°/7.9/1.33 21.85°/6.62/1.1
𝟤𝟥 3.55°/4.73/1.66 10.71°/7.51/1.87 6.12°/7.92/1.78 7.72°/7.47/1.51 10.09°/6.36/1.57
𝟤𝟧 1.14°/4.59/1.79 3.43°/6.96/1.76 10.5°/7.93/1.61 3.64°/7.73/1.33 23.9°/6.93/1.2
Table 1–source data 1. Thumb deviation/Thumb length/Thumb width, Index deviation/Index length/Index
width, Middle deviation/ Middle length/Middle width, Ring deviation/ Ring length/Ring width, Little deviation/Little
length/ Little width.
Discussion
The finger anthropological measurement algorithm achieved 82% success in feature extraction. The
main problem was the deep learning model for key points, that failed to find one point of the 20 key
points, usually the nail in the thumb. A solution, for improve detection is using computational vision
developed in the preprocessing of image for finding the key point not detected. For that reason, it
is necessary analyzing larger volume of images with pathological deviations, for generalizing and
automatized the missing detection. To avoid complication the algorithm works better without a
ring or other accessories.
Proceedings of the 5th Congress on Robotics and Neuroscience
Methods and Materials
Access to 2 large volume databases was obtained, used to determine knuckle patterns and another
to determine sex according to the characteristics of the hands. A third database of images of the
hands of people with Arthritis, the request was made, but the response was negative.
“The Hong Kong Polytechnic University contactless hand dorsal images database”.
Description: The database of dorsal images from the Polytechnic University of Hong Kong is a
contribution of male and female volunteers. This database was acquired at the IIT Delhi Campus of
the Polytechnic University of Hong Kong and in some villages in India during the period 2006-2015,
mainly through the use of a mobile and handheld camera. This database has 2505 dorsal images
of the right hand of 501 different subjects that illustrate three knuckle patterns on each of the
subject’s four fingers. All images are in bitmap format (* .bmp). This database is available by direct
and justified request (Kumar and Xu, 2016).
“11k Hands. Gender recognition and biometric identification using a large dataset of hand
images”.
Description: The database contains the “11k Hands” data set, a collection of 11,076 hand images
(1600 x 1200 pixels) of 190 subjects, of different ages between 18 and 75 years. Each hand was
photographed from both sides dorsal and palmar with a uniform white background and placed
approximately the same distance from the camera. The database is free for reasonable academic
use (Afifi, 2019).
Methodology With the images of the database
We proceeded to extract the following characteristics, nail position, knuckles and centroid of the
hand, for these purposes we worked only with the dorsal images of the hand. Train the machine
learning to extract the characteristics of the image and generate a skeleton of the hand, a binarized
image and an image with the baselines to calculate the angulation. The development of the
methodology contains the following stages: Binarization, Edges, Framing, Centroid, Threshold,
Skeleton, Finger detector, Finger count, Length and Width Measurement, Identifying key points of
the hand, Identifying wrinkles, Identify nails and knuckles (proximal IF) and Deviation Measurement.
The detail of each stage is explained step by step.
The first three stages are summarized in the Figure 1, the image is binarized (A) and the
properties of the region are analyzed, the edge of the object is removed (B), frame stage the area
of interest is enclosed in a red box (C).
Figure 1. A)Binarized Image, B)Egde Extraction, C)Framing
The next steps Figure 2, are the localization of the centroid, correspond to the center of mass
of the object(A), then the threshold (B), the image is thresholded to identify the distance from the
center to the edge. The skeleton was extracted (C) with the highest threshold values. With this
stages completed we are ready to parcel the fingers.
In Figure 3, the finger detector, a mask that only involves the fingers is created to delimitate the
region of interest for the problem (A), to verify the previous procedure we count the fingers (B) and
Proceedings of the 5th Congress on Robotics and Neuroscience
Figure 2. A)Centroid, B)Threshold, C)Skeleton
Figure 3. A)Mask Finger, B)Finger labels, C)Length and Width Measurement
finally length and width of each fingers are measured (C), the unit of measurement is centimeters.
We consider the 20 key points of the hand, these points are identified by the search of ROI,
then the wrinkles of the fingers are identified by the technique of "deep learning" (DL), finally the
identification of the nails and knuckles (proximal inter phalangic joint ) are the points of interest
to identify for the measurement of the deviation that is made with the calculation of the angle
between the points of interest detected in the previous stage.
Figure 4. A)Regions of interest, B)Projection key points, C)Combination of the previous steps
The Figure 4 show the ouput of DL model, Regions of Interest (ROI) were detected; the points
of interest that will be 3 for each finger, one at the base, another at the proximal interphalangeal
joint and finally at the end of the finger, represented by the nail (A). The fingers length and width
were obtained measurement the distance between limits of the finger, this measure was made
calculating euclidean distance between the previous points detected (B). Finally, the summarized of
the previous methods is the combination of the skeleton and the key points detected (C).
For make the DL model we have trained with 11.000 images labeled, and then 1.000 images
was reserved for subsequently tested Wei et al. (2016). The model was trained with hand images,
Proceedings of the 5th Congress on Robotics and Neuroscience
Figure 5. Summarized Result of previous steps
Figure 6. Faster RCNN model
did focus on nails, base of the finger and proximal knuckle. The detection of the 3 elements was
made with a fine-tuning, conserving output layer architecture in the model Simon et al. (2017) and
manages to understand the differences of the 4 fingers and the thumb.
The algorithm is based on ROI Convolutional Neural Network models like the one below:
The model calculates a map of characteristics, then generates a probabilistic mapping of the
areas indicated by a coordinate vector, finally in a fully connected stage, both the output of the
classification and the output of the area of interest are generated by a probabilistic map, which can
be used by the tensorflow API to indicate ROI in new images.
We use LabelImg, for them first a file is created with the classes to be considered, then the same
classes are selected with manual segmentation of the area of interest, the program does the rest by
generating the feature vector:
The ROI previously described will be the basis for forming 2 lines with a point in common. We
must determine with the tangent arc of the angle that forms them and thus see the inclination of
the slopes of these lines between them. Finally, subtract the angles of both slopes.
Projections
In a first iteration, the algorithm presents a bad measurement of the Thumb and width of the
fingers, in the last, the width are the same for all fingers. Then in a new iteration, the problem was
fixed improving the object size method, changing the method of calculus with a new more precision
Proceedings of the 5th Congress on Robotics and Neuroscience
approach. For future works, an increase in the number of images of hands with pathological
deviations will be useful for establishing a, standardize the acquisition of images and increase the
variability of images to improve the algorithm of detection of regions of interest. This latter will
allow us to confirm the actual measurements with conventional measurement. The objective of
this article was to make the quantification system known to the clinical environment and propose it
as a simple and easily accessible tool for the standardization of patient evolution and response to
treatment.
Acknowledgments
We acknowledgments to the project PMI 1402 "Desarrollo de una plataforma Interdisciplinaria para
la innovación en Salud: Un referente Internacional en el desarrollo de medicina de Precisión"
References
Afifi M. 11K Hands: Gender recognition and biometric identification using a large dataset of hand images.
Multimedia Tools and Applications. 2019; doi: 10.1007/s11042-019-7424-8.
Jonsson H. Age related prevalence of hand osteoarthritis diagnosed by photography (HOASCORE). BMC
Musculoskeletal Disorders. 2017; doi: 10.1186/s12891-017-1870-0.
Jonsson H, Helgadottir GP, Aspelund T, Sverrisdottir JE, Eiriksdottir G, Sigurdsson S, Eliasson GJ, Jonsson A,
Ingvarsson T, Harris TB, Launer L, Gudnason V. The use of digital photographs for the diagnosis of hand
osteoarthritis: The AGES-Reykjavik study. BMC Musculoskeletal Disorders. 2012; doi: 10.1186/1471-2474-13-
20.
Kumar A, Xu Z. Personal Identification Using Minor Knuckle Patterns from Palm Dorsal Surface. IEEE Transac-
tions on Information Forensics and Security. 2016; doi: 10.1109/TIFS.2016.2574309.
Mollard E, Michaud K. A Mobile App With Optical Imaging for the Self-Management of Hand Rheumatoid
Arthritis: Pilot Study. JMIR mHealth and uHealth. 2018; doi: 10.2196/12221.
Simon T, Joo H, Matthews I, Sheikh Y. Hand keypoint detection in single images using multiview bootstrapping.
In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition; 2017. p. 1145–1153.
Wei SE, Ramakrishna V, Kanade T, Sheikh Y. Convolutional pose machines. In: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition; 2016. p. 4724–4732.