Proceedings of the 5th Congress on Robotics and Neuroscience Computational vision and machine learning to evaluate Metacarpophalangeal and Interphalangeal deviation in fingers for clinical purpose Matías Salinas MSc1* , Astrid Cancino MSc1 , Alejandra Zazueta MSc1 , Rodrigo Salas Phd2 *For correspondence: matias.salinas@postgrado.uv.cl 1 Programa Doctorado en Ciencias e Ingeniería para la Salud, Universidad de Valparaíso; (FMS); astridcancino@gmail.com (FS) 2 Escuela de Ingeniería C. Biomédica, Universidad de Valparaíso Present address: † Doctorado en Ciencias e Ingeniería para la Salud, Universidad de Valparaíso, Chile Abstract Metacarpophalangeal deviations can be present from congenital diseases, nerve injuries, direct trauma such as fractures, autoimmune arthropathies such as that associated with lupus erythematosus (Jaccoud), to rheumatoid arthritis (RA), which is an inflammatory disease of joint components with varying degrees of destruction of the small joints of hands, producing a deformation that is evidenced in the deviation of the axis of the fingers having an impact on the functionality of the hand. The application of computational vision and machine learning in the quantification of finger angulation and joint thickening, can provide an objective way to record measures that are routinely performed subjectively if there is no specialized equipment. These measurements can be very useful to have an adequate pre and post intervention record or simply have a standardized record in the case of degenerative pathologies. Through digital image processing, it is possible to define markers that allow us to calculate the angulation and width of the metacarpophalangeal segments at the service of the clinic or useful in research. The results generated were collected in a csv file showing 80% effectiveness in images with arthritis. Introduction The activity of the hands in daily life is extremely delicate and precise, such as writing, painting or playing an instrument. They also allow us to perform heavy labor, such as digging with a shovel or hammering. The manual skills are almost imperceptible in health condition until the appearance of a hand injury or the manifestations of a degenerative or autoimmune disease begins. In hand pathology that show alteration of the axis of the fingers, the anthropometry of hands is responsible for studying the dimensions and orientation of the human body and its relationship with its environment. The field of anthropometry is very extensive, it could be adopted in any field, from manufacturing industries, tool design to topics of medicine and sports. The most commonly used measurements of the hand, have to do with the manufacturing field of tools, operating room items, gloves, specialized handles by hand size, equipment for disabled people, prostheses even in the pre and post operative clinical records of hands, tracking of finger fractures and degenerative joint pathologies. The incorporation of the metacarpophalangeal and interfalange Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Proceedings of the 5th Congress on Robotics and Neuroscience deviation metric would be very useful in the specialized field of medicine and rehabilitation of traumatic and degenerative pathologies such as the sequels of lupus and arthritis. This measure is not widely developed in a standardized way as the measures applied to the industry but can provide great support by being quantified in an automated way(Jonsson, 2017). Machine learning, which has shown great success, poses an opportunity in the field of medicine and rehabilitation with a quantification that leads to better records before and after the intervention and monitoring of joint pathologies. Learning machines that extract characteristics of a problem and then relate it to an answer have shown a high degree of adaptation to dynamic events, which will allow us to work with different types of hands and pathologies. Results Two databases of healthy hands were obtained to extract characteristics using the developed algorithm. From the collection of images of hands with arthritis from the article by (Jonsson et al., 2012), automated feature extraction was achieved to obtain anthropometric measurements of the hands for the purpose of clinical record of evolution in a supervised way in degenerative pathologies. For the test stage in hands with pathology described below, it was applied in 10 random hands with arthritis were processed, the result is shown in Table 1, the collection of images contain 28 arthritic hands from 14 users, with a degree of deformity greater than the healthy hands. The results indicate that from 28 hands, it was possible to extract in 23 the characteristics of interest such as length, width and angulation of each of the fingers, while in the remaining ones it is necessary to improve the code to adapt to all possible image records, such as for example the use of rings, watches, among others. Table 1. Finger Deviation Results User_ID TD/TL ID/IL MD/ML RD/RL LD/LL 𝟣 5.72°/4.65/2.01 7.66°/7.94/2.0 6.76°/8/1.95 7.66°/7.93/1.8 5.71°/7.41/1.53 𝟥 13.15°/4.81/1.94 7.6°/7.83/1.92 0.7°/8.70/1.8 1.36°/8.38/1.66 17.9°/7.63/1.49 𝟧 7.66°/5.4/1.89 7.72°/8.13/1.81 10.68°/8.76/1.67 15.15°/8.70/1.5 15.32°/6.7/1.24 𝟫 9.88°/4.53/1.86 11.51°/7.9/1.76 1.36°/8.3/1.61 8.25°/8.3/1.46 3.63°/7.52/1.3 𝟣𝟣 6.16°/4.51/1.87 8.21°/7.36/1.91 0°/7.4/1.89 5.81°/7.2/1.57 14.64°/7.08/1.39 𝟣𝟥 3.21°/5.07/2.14 0.04°/7.24/2.02 9.68°/8.04/1.93 0.89°/7.6/1.71 17.6°/7.22/1.47 𝟣𝟫 22.38°/4.12/1.56 2.7°/7.2/1.84 2.33°/8/1.6 3.58°/7.95/1.31 20.72°/6.78/1.14 𝟤𝟣 5.51°/4.21/1.64 0°/7.95/1.66 0°/8.38/1.61 7.66°/7.9/1.33 21.85°/6.62/1.1 𝟤𝟥 3.55°/4.73/1.66 10.71°/7.51/1.87 6.12°/7.92/1.78 7.72°/7.47/1.51 10.09°/6.36/1.57 𝟤𝟧 1.14°/4.59/1.79 3.43°/6.96/1.76 10.5°/7.93/1.61 3.64°/7.73/1.33 23.9°/6.93/1.2 Table 1–source data 1. Thumb deviation/Thumb length/Thumb width, Index deviation/Index length/Index width, Middle deviation/ Middle length/Middle width, Ring deviation/ Ring length/Ring width, Little deviation/Little length/ Little width. Discussion The finger anthropological measurement algorithm achieved 82% success in feature extraction. The main problem was the deep learning model for key points, that failed to find one point of the 20 key points, usually the nail in the thumb. A solution, for improve detection is using computational vision developed in the preprocessing of image for finding the key point not detected. For that reason, it is necessary analyzing larger volume of images with pathological deviations, for generalizing and automatized the missing detection. To avoid complication the algorithm works better without a ring or other accessories. Proceedings of the 5th Congress on Robotics and Neuroscience Methods and Materials Access to 2 large volume databases was obtained, used to determine knuckle patterns and another to determine sex according to the characteristics of the hands. A third database of images of the hands of people with Arthritis, the request was made, but the response was negative. “The Hong Kong Polytechnic University contactless hand dorsal images database”. Description: The database of dorsal images from the Polytechnic University of Hong Kong is a contribution of male and female volunteers. This database was acquired at the IIT Delhi Campus of the Polytechnic University of Hong Kong and in some villages in India during the period 2006-2015, mainly through the use of a mobile and handheld camera. This database has 2505 dorsal images of the right hand of 501 different subjects that illustrate three knuckle patterns on each of the subject’s four fingers. All images are in bitmap format (* .bmp). This database is available by direct and justified request (Kumar and Xu, 2016). “11k Hands. Gender recognition and biometric identification using a large dataset of hand images”. Description: The database contains the “11k Hands” data set, a collection of 11,076 hand images (1600 x 1200 pixels) of 190 subjects, of different ages between 18 and 75 years. Each hand was photographed from both sides dorsal and palmar with a uniform white background and placed approximately the same distance from the camera. The database is free for reasonable academic use (Afifi, 2019). Methodology With the images of the database We proceeded to extract the following characteristics, nail position, knuckles and centroid of the hand, for these purposes we worked only with the dorsal images of the hand. Train the machine learning to extract the characteristics of the image and generate a skeleton of the hand, a binarized image and an image with the baselines to calculate the angulation. The development of the methodology contains the following stages: Binarization, Edges, Framing, Centroid, Threshold, Skeleton, Finger detector, Finger count, Length and Width Measurement, Identifying key points of the hand, Identifying wrinkles, Identify nails and knuckles (proximal IF) and Deviation Measurement. The detail of each stage is explained step by step. The first three stages are summarized in the Figure 1, the image is binarized (A) and the properties of the region are analyzed, the edge of the object is removed (B), frame stage the area of interest is enclosed in a red box (C). Figure 1. A)Binarized Image, B)Egde Extraction, C)Framing The next steps Figure 2, are the localization of the centroid, correspond to the center of mass of the object(A), then the threshold (B), the image is thresholded to identify the distance from the center to the edge. The skeleton was extracted (C) with the highest threshold values. With this stages completed we are ready to parcel the fingers. In Figure 3, the finger detector, a mask that only involves the fingers is created to delimitate the region of interest for the problem (A), to verify the previous procedure we count the fingers (B) and Proceedings of the 5th Congress on Robotics and Neuroscience Figure 2. A)Centroid, B)Threshold, C)Skeleton Figure 3. A)Mask Finger, B)Finger labels, C)Length and Width Measurement finally length and width of each fingers are measured (C), the unit of measurement is centimeters. We consider the 20 key points of the hand, these points are identified by the search of ROI, then the wrinkles of the fingers are identified by the technique of "deep learning" (DL), finally the identification of the nails and knuckles (proximal inter phalangic joint ) are the points of interest to identify for the measurement of the deviation that is made with the calculation of the angle between the points of interest detected in the previous stage. Figure 4. A)Regions of interest, B)Projection key points, C)Combination of the previous steps The Figure 4 show the ouput of DL model, Regions of Interest (ROI) were detected; the points of interest that will be 3 for each finger, one at the base, another at the proximal interphalangeal joint and finally at the end of the finger, represented by the nail (A). The fingers length and width were obtained measurement the distance between limits of the finger, this measure was made calculating euclidean distance between the previous points detected (B). Finally, the summarized of the previous methods is the combination of the skeleton and the key points detected (C). For make the DL model we have trained with 11.000 images labeled, and then 1.000 images was reserved for subsequently tested Wei et al. (2016). The model was trained with hand images, Proceedings of the 5th Congress on Robotics and Neuroscience Figure 5. Summarized Result of previous steps Figure 6. Faster RCNN model did focus on nails, base of the finger and proximal knuckle. The detection of the 3 elements was made with a fine-tuning, conserving output layer architecture in the model Simon et al. (2017) and manages to understand the differences of the 4 fingers and the thumb. The algorithm is based on ROI Convolutional Neural Network models like the one below: The model calculates a map of characteristics, then generates a probabilistic mapping of the areas indicated by a coordinate vector, finally in a fully connected stage, both the output of the classification and the output of the area of interest are generated by a probabilistic map, which can be used by the tensorflow API to indicate ROI in new images. We use LabelImg, for them first a file is created with the classes to be considered, then the same classes are selected with manual segmentation of the area of interest, the program does the rest by generating the feature vector: The ROI previously described will be the basis for forming 2 lines with a point in common. We must determine with the tangent arc of the angle that forms them and thus see the inclination of the slopes of these lines between them. Finally, subtract the angles of both slopes. Projections In a first iteration, the algorithm presents a bad measurement of the Thumb and width of the fingers, in the last, the width are the same for all fingers. Then in a new iteration, the problem was fixed improving the object size method, changing the method of calculus with a new more precision Proceedings of the 5th Congress on Robotics and Neuroscience approach. For future works, an increase in the number of images of hands with pathological deviations will be useful for establishing a, standardize the acquisition of images and increase the variability of images to improve the algorithm of detection of regions of interest. This latter will allow us to confirm the actual measurements with conventional measurement. The objective of this article was to make the quantification system known to the clinical environment and propose it as a simple and easily accessible tool for the standardization of patient evolution and response to treatment. Acknowledgments We acknowledgments to the project PMI 1402 "Desarrollo de una plataforma Interdisciplinaria para la innovación en Salud: Un referente Internacional en el desarrollo de medicina de Precisión" References Afifi M. 11K Hands: Gender recognition and biometric identification using a large dataset of hand images. Multimedia Tools and Applications. 2019; doi: 10.1007/s11042-019-7424-8. Jonsson H. Age related prevalence of hand osteoarthritis diagnosed by photography (HOASCORE). BMC Musculoskeletal Disorders. 2017; doi: 10.1186/s12891-017-1870-0. Jonsson H, Helgadottir GP, Aspelund T, Sverrisdottir JE, Eiriksdottir G, Sigurdsson S, Eliasson GJ, Jonsson A, Ingvarsson T, Harris TB, Launer L, Gudnason V. The use of digital photographs for the diagnosis of hand osteoarthritis: The AGES-Reykjavik study. BMC Musculoskeletal Disorders. 2012; doi: 10.1186/1471-2474-13- 20. Kumar A, Xu Z. Personal Identification Using Minor Knuckle Patterns from Palm Dorsal Surface. IEEE Transac- tions on Information Forensics and Security. 2016; doi: 10.1109/TIFS.2016.2574309. Mollard E, Michaud K. A Mobile App With Optical Imaging for the Self-Management of Hand Rheumatoid Arthritis: Pilot Study. JMIR mHealth and uHealth. 2018; doi: 10.2196/12221. Simon T, Joo H, Matthews I, Sheikh Y. Hand keypoint detection in single images using multiview bootstrapping. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition; 2017. p. 1145–1153. Wei SE, Ramakrishna V, Kanade T, Sheikh Y. Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016. p. 4724–4732.