Extensible Disease Classification Tool that can be Dynamically Trained by the End User Nikos Petrellis 1 1 University of Peloponnese, Electrical and Computer Engineering Dept., Patras, Greece Abstract The Disease Classifier tool that is capable of classifying images displaying sick plant parts as well as human or animal skin disorders is presented in this paper. It is based on earlier applications (Plant Disease and Skin Disease) and classification techniques developed by the author for Windows Phones. These tools have been redesigned using Microsoft Xamarin platform and thus they can be deployed on Android or iOS smartphones. Images displaying either plant or skin disorders can be classified in the Disease Classifier environment based on the features of the lesions appearing on a plant leaves or fruits or the skin. New useful services have also been integrated in the Disease Classifier tool: a) the diagnosis rules are dynamically loaded without requiring recompilation and b) the features extracted when a new photograph is analyzed can be used automatically to train the tool on new diseases using a relatively small training set. These services, in conjunction with the employed simple but efficient classification algorithm, make the Disease Classifier a powerful assistant for end users such as agronomists since they can extend the tool for the support of new diseases. Keywords 1 Plant disease, skin disorders, dynamic configuration, trained by the end user, Xamarin 1. Introduction Several mobile applications have been recently presented for plant disease diagnosis [1]. Maize diseases are diagnosed using a smartphone application in [2]. Crop diseases are recognized using an intelligent mobile application based on fuzzy inference in [3]. Artificial Intelligence (AI) methods based on Convolutional Neural Networks (CNN) have been extensively used to classify plant diseases. A CNN learning algorithm performing early plant disease diagnosis and classification with 98% success rate is demonstrated in [4]. Twelve plant diseases are recognized by a CNN network as described in [5]. Similar image processing techniques have also been recently presented e.g., for hand, foot and mouth diseases [6]. Several medical image processing techniques have been simulated and compared in [7]. Various image processing techniques that have been applied both for plant disease diagnosis and medical imaging are discussed in [8]. A simple but efficient classification method has been proposed by the author initially for plant disease diagnosis. It was implemented as a Windows Phone application (Plant Disease) since it was developed using Microsoft Visual Studio 2015. The classification method employed in this application is based on grading the supported diseases according to whether the extracted features fall within strict or loose limits that have been defined during training. The training can be performed using a small fraction of photographs from the available dataset [9]. The same classification method has been also applied for skin disorders in the Skin Disease application described in [10]. One of the drawbacks of the Plant and Skin Disease applications was the practically abandoned target platform (Windows Phone). Moreover, the limits of the features in each disease had to be estimated manually, converted to C# classes and embedded to the application source code. Consequently, the application had to be rebuilt for a different set of supported diseases. Proceedings of HAICTA 2022, September 22–25, 2022, Athens, Greece EMAIL: npetrellis@uop.gr (A. 1) ORCID: 0000-0003-3578-8494 (A. 1) ©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 382 The Disease Classifier application described was developed using Microsoft Xamarin platform thus, it can be deployed either on Android or iOS smart phones. Moreover, feature extraction from photographs displaying a non-supported disease can be used to automatically train the system on new diseases. In the Disease Classifier application, the disease recognition rule files cab be loaded dynamically and the application can switch between different sets of supported diseases. In this way, the end user (e.g., an agriculture research institute or an agronomist) can customize the tool for the requirements of his customers (pricing policy, supported diseases). The classification method and the early Plant and Skin Disease applications are described in Section 2. The new Disease Classifier application and its use are discussed in Section 3. 2. The Plant and Skin Disease applications Rules’ File Select Image Statistical Processing Segmentation Feature Extraction Disease Grading Disease Classification Figure 1: The basic steps followed in the Plant and Skin Disease applications. The Plant Disease and Skin Disease applications described in [9] and [10], respectively are based on the operations shown in Figure 1. Initially the Rules file has to be selected that contains the feature ranges for each disease. Then the image is selected and segmented, followed by feature extraction. The feature values are compared with the ranges in the Rules file to grade each disease. The image is classified as the disease with the highest grade. Before giving more details on the steps displayed in Figure 1, the features used in the employed classification method have to be described. Table 1, lists all the classification features that represent meaningful aspects of the image such as the number of spots and their area. When a new image is selected in the second step of Figure 1, it is segmented in 4 regions using appropriate thresholds. For the sake of simplicity, it was assumed that the background was white. The rest of the image it is assumed to display the plant or skin part. Pixels in this part, that are darker than another threshold are assumed to belong to the lesion spots. In some diseases, the lesion spots are brighter than the normal part and in this case they are marked as spots if their color is brighter than the threshold. Finally, a halo with a configurable pixel thickness (e.g., 5 pixels) is defined around the spots since in most diseases the spots are surrounded by a region of distinct color. As shown in Table 1, the number of recognized spots, their relative area, and the gray level of the spot, the normal part and the halo regions are some of the features used in the classification process. Then, for each one the three regions of interest, a histogram is constructed for each color [9]. These histograms indicate how many pixels exist with a specific color level (0-255) in one the three regions. It has been experimentally shown that in most cases these histograms are single lobes. It is expected that the histogram lobes extracted from pictures displaying the same disease would be similar. However, instead of using a complicated pattern matching method for histogram similarity, we simply compare the start, the end and the peak of the lobes. In this way, an overall number of 32 features are used as shown in Table 1. 383 Table 1 Features used in the employed classification method Feature Description Number of lesion spots (islands). Very small spots can be considered as Spots noise Spot Area The fraction of the lesion spot pixels in the displayed plant or skin part Spot gray level The average gray level of the spot pixels Normal part gray level The average gray level of the normal part pixels Halo gray level The average gray level of the halo pixels The color (red, green or blue) level where the histogram lobe (spot, Histogram lobe start normal, halo) starts (9 parameters) The color (red, green or blue) level where the histogram lobe (spot, Histogram lobe end normal, halo) ends (9 parameters) Histogram lobe peak The histogram lobe peak position (9 parameters) After the segmentation of the image, the extraction of the 32 features listed in Table 1 takes place (4th step of Figure 1). During training on a new disease d, the feature f values of the training photographs are examined to find its range in terms of strict limits (fds_min, fds_max). Loose limits (fdl_min, fdl_max) are arbitrary defined by extending the strict limits either by a constant number or by a fraction. The strict and loose limits for each disease form the rules file required when a new photograph is analyzed (1st step of Figure 1). Each feature f can participate in the disease d grade with a different weight wfd. When a new photograph has to be classified in one of the supported diseases, its extracted feature values are compared with the strict and loose limits of each disease d and a grade Gd is estimated as follows: (1) 𝐺𝑑 = βˆ‘ π‘₯𝑠𝑓 𝑀𝑠_𝑓𝑑 + βˆ‘ π‘₯𝑙𝑓 𝑀𝑙_𝑓𝑑 , 𝑓 𝑓 π‘₯𝑠𝑓 = 1, 𝑖𝑓 𝑓𝑑𝑠_π‘šπ‘–π‘› ≀ 𝑓𝑑 ≀ 𝑓𝑑𝑠_π‘šπ‘Žπ‘₯ , otherwise 0 (2) π‘₯𝑙𝑓 = 1, 𝑖𝑓 𝑓𝑑𝑙_π‘šπ‘–π‘› ≀ 𝑓𝑑 ≀ 𝑓𝑑𝑙_π‘šπ‘Žπ‘₯ , otherwise 0 (3) (a) (b) (c) Figure 2: The main page of the Plant Disease (a)and the Skin Disease application before (b) and after segmentation (c) [9][10]. The analyzed image is classified to the disease d with the highest grade Gd. The user selects the input photograph from the main page of the Plant Disease application displayed in Figure 2. In Skin Disease 384 ver. 1.0, a Threshold in gray level was used to segment the lesion spots and a second threshold for the Background. The field Min Spot Area determines the minimum number of pixels, a spot has to consist of, in order to be taken into consideration (otherwise it is recognized as noise and ignored). In Plant Disease ver. 2.6 different spot and background thresholds are determined for each one of the red, green and blue colors. More sophisticated automatic segmentation methods can also be tested by checking the Altern checkbox instead of Manually. If the spots are brighter than the normal plant or skin part, then the Invert checkbox has to be ticked. In the segmented image shown in Fig. 2c, the background is displayed in white, the spots in black, the halo in gray and the normal part in darker gray. The histograms for each one of these regions can be previewed from the corresponding buttons (Spot Col, Norm Col, Halo Col). 3. The Disease Classification Application The Disease Classification application presented in this paper has been developed using Microsoft Xamarin platform in Visual Studio 2019. Porting the older Plant and Skin Disease applications from Visual Studio 2015 to Xamarin is not straight forward. Thus, the interface of the new application had to be developed from scratch, incorporating the necessary improvements. The development of the Device Classification in Xamarin allowed its deployment as an Android or iOS mobile application. (a) (b) (c) Figure 3: The entry page (a) of the Disease Classification application, the main page for photograph selection and analysis (b) and the page displaying the extracted feature values (c). The most important advantage of the new application that has been developed, is the support of the dynamic loading of the disease recognition rules (step 1 in Figure 1). This step required a new project build in the earlier applications. The strict/loose limits estimated from the training phase are now stored in a plain text file that can be picked in the first page of the application (see Figure 3a). The format of the rules file is relatively simple: the number of diseases D is listed in the first line followed by D pairs of lines. Each pair contains the disease name (first line) and a message in the next line. This message can be displayed if the image is classified to the corresponding disease. Then, D more pairs of lines follow with the strict/loose limits and the weights of each feature. More specifically, 385 in the first line of the pair for a specific disease d, the features limits are listed in the order: fdl_min, fds_min, fds_max, fdl_max (4Γ—32 features=128 values in a single line separated with commas). In the following line of the pair, a weight is listed for the specific feature loose limits and one for the feature strict limits (64 values). The next page (Figure 3b) is similar to the main page of the Plant Disease ver. 2.6. After successfully segmenting the input image, the feature extraction takes place and the feature values are displayed in the page shown in Figure 3c. Although, the feature values were also listed in the earlier Plant and Skin Disease applications, the administrator had to copy the feature values to a spreadsheet in order to extract manually their ranges. Then, the strict and loose limits of these feature ranges had to be transformed in C# classes, incorporated in the source code of the applications (the path with the dash-line in Figure 1). In the Disease Classification application, the button Train has been added that accumulates the feature values from the training images in a single file that is processed automatically in order to generate the rules file that is now loaded dynamically after the training is finished. 4. Conclusions A mobile application for disease classification, developed in Microsoft Xamarin for Android and iOS, was presented in this paper. It is based on earlier Windows Phone applications that have been developed by the author for plant and skin disease diagnosis. The Disease Classification tool allows the dynamic disease rule definition. This service in conjunction with the automatic training that is performed using a small image set, allow the extension of the supported diseases by an end user that is not necessarily qualified in computer science. Future work will focus on experimenting with plant and skin diseases using publically available datasets and testing extensions of the employed classification algorithm. 5. Acknowledgements This work is protected by the patents 1009346/13-8-2018 and 1008484/12-5-2015 (Greek Patent Office). 6. References [1] I.N. Oteyo, M. Marra, S. Kimani, W. De Meuter and E. Gonzalez Boix. β€œA Survey on Mobile Applications for Smart Agriculture.” SN COMPUT. SCI. 2, 293 (2021). https://doi.org/10.1007/s42979-021-00700-x. [2] N. Moawad, A. Elsayed, Smartphone Application for Diagnosing Maize Diseases in Egypt, in: Proceedings of the 14th IEEE International Conference on Innovations in Information Technology (IIT) Al Ain, United Arab Emirates, pp. 24-28, 2020. doi: https://doi.org/10.1109/IIT50501.2020.9299067 [3] M. Toseef and M. J. Khan. β€œAn intelligent mobile application for diagnosis of crop diseases in Pakistan using fuzzy inference system." Computers and Electronics in Agriculture, 153(2018): 1- 11. doi: https://doi.org/10.1016/j.compag.2018.07.034. [4] A. Xenakis, G. Papastergiou, V. C. Gerogiannis and G. Stamoulis, Applying a Convolutional Neural Network in an IoT Robotic System for Plant Disease Diagnosis, in: Proceedings of the 11th IEEE International Conference on Information, Intelligence, Systems and Applications (IISA), Piraeus, Greece, Dec. 11, 2020, pp. 1-8, doi: 10.1109/IISA50023.2020.9284356 [5] G. Shrestha, Deepsikha, M. Das and N. Dey, Plant Disease Detection Using CNN, in: Proceedings of the IEEE Applied Signal Processing Conference (ASPCON), Kolkata, India, Dec, 7, 2020, pp. 109-113, doi: 10.1109/ASPCON49795.2020.9276722. [6] M. Farhad Aryan, W. Krathu, C. Arpnikanondt, and B. Tassaneetrithep. Image Recognition for Detecting Hand Foot and Mouth Disease, in: Proceedings of the 11th International Conference on Advances in Information Technology (IAIT2020). Association for Computing Machinery, New York, NY, USA, 2020, Article 35, 1–11. DOI:https://doi.org/10.1145/3406601.3406640 386 [7] P. Salehi, N. Behzadfar, and H. Movahedi-Nasab. β€œInvestigation and Simulation of Different Medical Image Processing Algorithms to Improve Image Quality Using Simulink-Matlab.” Journal of Novel Researches on Electrical Power, 10(3), 33-42 (2021). [8] N. Petrellis. "A Review of Image Processing Techniques Common in Human and Plant Disease Diagnosis" Symmetry 10, no. 7: 270 (2018). https://doi.org/10.3390/sym10070270 [9] N. Petrellis. "Plant Disease Diagnosis for Smart Phone Applications with Extensible Set of Diseases" Applied Sciences 9, no. 9: 1952 (2019). doi: https://doi.org/10.3390/app9091952 [10] N. Petrellis, Using color signatures for the classification of skin disorders, in: Proceedings of the 7th International Conference on Modern Circuits and Systems Technologies (MOCAST), Thessaloniki, Greece, 7-9 May 2018, pp. 1-4, doi: 10.1109/MOCAST.2018.8376573 387