A Windows Phone Application for Plant Disease Diagnosis Nikos Petrellis Department of Computer Science and Engineering, Technological Educational Institute of Thessaly, Larissa, Greece, e-mail: npetrellis@teilar.gr Abstract. Although professional agriculture engineers are responsible for the recognition of plant diseases, intelligent systems can help in their diagnosis in early stages. Using such systems, low cost continuous plant monitoring can be applied. Some expert systems have been proposed in the literature for this purpose that are based on user descriptions and image comparison. The symptoms of a disease include lesions or spots in various parts of a plant. The color, area and the number of these spots can determine to a great extent the disease that has mortified a plant. Higher cost molecular analyses and tests can follow if necessary. In this paper, a Windows Phone application capable of measuring the plant lesion features is described. The accuracy in the plant disease recognition is higher 90% according to the experimental results performed using grape diseases as a case study. Keywords: plant disease, lesions, image processing, agricultural production. 1 Introduction Plant diseases can increase dramatically the cost of agricultural production if they are not detected and treated in their early stages. The plants have to be monitored all the time in order to detect the first symptoms of a disease before it is spread to the whole crop. Professional agriculture engineers may not be available to continuously monitor a crop especially if its size is small or medium and the cost for such a process is high, or if the crop resides in a distant rural region. Remote monitoring through machine vision can offer an alternative option. For example, the user can send photos to professional agriculturists and ask opinions based on the visible symptoms. Several additional tests may have to be performed in order to confirm if a plant is affected by a specific disease. The plant disease diagnosis can be based on several symptoms that are described in detail in (Riley et al., 2002). The symptoms can be grouped as follows: a) Underdevelopment or, b) Overdevelopment of tissues or organs and c) Necrosis or death of plant parts and alteration of normal appearance. The progression of the symptoms can vary significantly and it is associated with problems caused by biotic agents. The symptoms can be classified as primary and secondary. For example, the 709 initial stages of a disease can include decayed roots of a tree, while the secondary symptoms may include the toppling over of the tree. The invaders of the later disease stages may also obscure the original disease symptoms misleading the diagnosis. The improper herbicide usage can cause symptoms similar to spots caused by an infectious agent. The sudden appearance of the symptoms and the absence of progression can be an indication that the cause is the herbicide usage. Moreover, new leaves are generally free of symptoms. More than one pathogens can often infect a plant and the symptoms associated in this case may be significantly different from the symptoms shown by each of the different pathogens when they act separately. The symptoms of a pathogen can be often expressed as fungal or bacterial leaf spots. Vein banding, mosaic and ringspot can also appear. The leaves can be distorted or a powdery mildew can appear. Spore structures may be present. The needles may drop in conifer trees. The plants can be injured by chemical spray or air pollution or by soil/air chemicals. Cankers can appear at the branches of a tree. The fruits can have decays and rots or discoloration. Abnormal wilts or dying branches can appear. In most of the cases listed above, an image processing technique could have been used to locate the lesions and quantify them by estimating the number of spots, their area, their color, etc. Several image processing techniques and molecular tests are reviewed in (Sankaran et al., 2010). The sensitivity of molecular tests, depends on the detectable minimum amount of microorganism. The ELISA is a popular molecular diagnosis, based on the use of a microbial protein associated with the plant disease. The antibodies are produced by an animal after this protein is injected to it. Another popular technique based on DNA analysis is PCR (Schaad et al., 2002). The author of this paper has been recently involved in the development of a portable low cost equipment capable of performing molecular tests (Georgakopoulou et al., 2016). Observation (non-destructive) techniques like spectroscopic and image processing can be used for plant disease diagnosis based on its symptoms. Spectroscopic techniques can identify water stress levels, nutrient deficiency, measure the fruit quality after the harvest, etc. Infrared spectroscopy is described for example in (Purcell et al., 2009). Reviews of image processing techniques in visible light for plant disease detection can be found in (Kulkarni et al., 2012). In that paper an image segmentation takes place in the CIE L*a*b color scale, then a Gabor filter is used to generate the input of a neural network that achieves a disease recognition with 91% accuracy. A plant part shape, its texture, fractal dimensions, lacunarity, dispersion, grey levels, grey histogram discrimination and the Fourier descriptors have also been taken into account in disease diagnosis. A quantification technique for fruit traits is presented in (Mix et al., 2003). An expert system based either on graphical representation or a step-by-step descriptive method is discussed in (Abu-Nasser et al., 2008). Step by step description is based on a questionnaire answered by the user. In graphical representation, the user compares manually photos stored in a database to find the disease that matches his case. A customized solution for corn diseases has been presented in (Lai, 2010). The most of the image processing and spectroscopic techniques require the analysis to be performed by high cost equipment or computational intensive software packages. In this paper, a mobile application is presented that is based on a low complexity image processing technique that isolates the normal leaf area from the sick lesions. 710 The described image processing technique can be used in the framework of a scalar system that can operate in standalone mode as a single smart phone application or in cooperation with remote clouds or databases or with a portable molecular analysis equipment like the one described in (Georgakopoulou et al., 2016). In the present version, focus is given on the lesions that can appear at various parts of a plant like the leaves, or the fruit. The lesions will also be called “spots” in this paper and the developed application counts the number of these spots in the part of the plant that appears in the photo, their area, their gray level and extracts color histograms. The spot color Red-Green-Blue (RGB) features or their CIE L*a*b scale can be extracted since a map of the spot positions is available. This allows the support of other referenced techniques like the one presented in (Kulkarni et al., 2012). In real time operation, the producer would use a smart phone with the application installed, in order to take pictures of mortified plant parts. Then, the application would ask him to provide some additional information that is useful for a more reliable diagnosis. The application in its current form has been tested for a number of plant diseases and can be easily extended for several other cases. In this paper we use the proposed application to measure the features of the spots that appear in pear, orange, tangerine tree or grapevine leaves. The application can successfully discriminate between pear tree diseases like Venturia Pirina, Septoria Pyricola, Erwinia Amylovora, etc., orange/tangerine tree diseases like fungus Capnodium oleae. Grape diseases like Powdery or Downy Mildew, Esca, etc. are used in this paper to extract experimental results. The experimental results also show that the measurement of the number of spots, their gray level and area in any plant part photo can be achieved with higher than 90% accuracy. The image processing technique used for the characterization of the spots is described in Section 2. The implementation of the Windows Phone application is presented in Section 3 and experimental results are discussed in Section 4. Fig. 1. Measurement of the leaf and spot size (Petrellis, 2015). 711 2 Plant disease recognition method The developed application operates on a smart phone that has to be equipped with a camera. The mobile phone should be capable of connecting to the Internet only if a richer plant disease database or cloud has to be accessed in order to support a diversity of plants. Moreover, this would be useful if the photos taken by the phone camera have to be accessed by an agriculture engineer for approving the final decision of the expert system. Otherwise, the application can be used offline and this is useful especially if it has to be used in distant fields where the 3G/4G coverage may be weak. In this case the application installed on the smart phone can support only the specific plants, that the producer is interested of. He can take photos of plant parts with lesions like leaves and fruits and then run the plant disease recognition application using the captured images. The application asks him some additional information like the kind of plant, its part that has been photographed (e.g., upper or lower leaf surface), the draft distance the photo was taken from the leaf, an estimation of how many leaves have been affected in a single plant or how many plants of the field have been affected. The producer can also be asked about the time the symptoms appeared. Environmental conditions (history of temperatures, moisture, etc) can be retrieved by using the GPS locator of the smart phone. The image processing algorithm incorporated in the application extracts the lesion features like number of spots, their grey level and area using an algorithm that has already been described in (Petrellis, 2015). The present version is extended to extract the color features of the normal leaf parts and the leaf spots too. These results can be exploited by the decision module of the application in conjunction with the information given by the user in order to draw a conclusion on the condition of the plant. In future versions the decision module can be an advanced neural network. The recognized disease here is the one that gets the higher score in a grading system that will be defined below. In a more advanced setup, the mobile phone can cooperate with a molecular analysis module like the biosensor readout circuit described in (Georgakopoulou et al., 2016) that has been developed for the Corallia/LabOnChip project. The communication with this module can be performed in a wired or wireless manner (USB, Wi-Fi, Bluetooth, etc.). In this paper, we will focus only at the stand-alone smart phone application. If the absolute spot dimensions have to be taken into consideration, the following estimation method has to be followed. The spot detection algorithm assumes that the photo with the plant part of interest has been captured from a known distance D (see Fig. 1) e.g., one palm between the camera and the leaf. However, if the relative leaf area is important instead of the absolute spot area, then the distance D and the following estimations are not important. If the camera angle φ is also known then, the half leaf length L is estimated by: L=D·tan(φ). (1) The length L can be estimated if the photo is taken from a known distance D and it corresponds to P pixels. The ratio S=L/P can be used to estimate another distance or area from the number of pixels. For example, if the length of a spot in an axis is L’ and corresponds to P’ pixels, it can be estimated as: 712 P' (2) L'= L P Moreover, if the two points do not reside on the same vertical or horizontal axis, equation (2) can also be used to estimate their distance if P’ is the number of pixels in the shortest diagonal distance between them. In a plane, if the dimensions of the covered area are Lx, Ly in the horizontal and vertical axis respectively and they correspond to Px and Py pixels, then each pixel occupies an area Ap that is estimated as follows 1 Lx Ly (3) Ap = S 2 Px Py Using equation (3), a spot of any shape that consists of P pixels will correspond to an area that is equal to P times the area of a single pixel Ap. In order to apply the spot recognition algorithm, the plant part should be initially separated by its background. This task could be implemented by a complicated segmentation procedure or a dedicated image processing library that would consume valuable smart phone resources. In the present version, it is assumed that the background is much brighter than the plant color. For this reason, a leaf for example is placed on a white sheet of paper as its background before it is captured by the smart phone camera. In this way, no background separation algorithm is needed at all. Moreover, it is also assumed that the photos have been captured under a canopy or during a cloudy day so that the shadows are not important. The pixels with a grey level higher than a configurable threshold BG are assumed to belong to the background. More sophisticated algorithms for the separation of the background and the leaf shadows will be implemented in future versions of the application. The three color components of an image (Red, Green, Blue) can be easily handled to extract detailed image features but they are ignored initially, handling the image in grey scale. The captured image is converted into a grey image separating the background, the normal leaf surface and the spots. The pixels belonging to the leaf are used to estimate an average grey level Ag. Then, the matrix of the image is scanned to check if there are leaf pixels i with a grey level Gi that fulfil the following condition: (4) Gi − Ag > Th If the difference between the grey level of a leaf pixel and the average Ag is higher than the threshold Th, the specific pixel is assumed to belong to a spot. A matrix BGW1 is constructed with the same dimensions as the original image. Each cell in BGW1 can be one of the following three grey levels: white for background (255), grey for normal leaf surface (e.g., 120) and black for spot (0) as shown in Fig. 2. The BGW1 is scanned again to group neighboring pixels belonging to the same spot. The resulting matrix BGW2 has an integer number in each one of its cells. This number is the identity of the spot that it belongs to. If a position in BGW2 is 0, then the corresponding pixel does not belong to a spot. 713 (a) (b) (c) Fig. 2. Original photograph (a), visualization of BGW1 matrix (b). Color histogram generated from the spot pixels (c). The following algorithm is used to construct the matrix BGW2: a) the rows are scanned from left to right and the neighboring pixels are assigned with the same identity, b) if the previous pixel on the left does not belong to a spot, the neighboring pixels at the row above are checked and if one or more of these has been assigned to an identity different than 0, this identity is also used for the current pixel, c) the scanning of the BW2 matrix is repeated merging spot identities until no changes are performed. All of the desired features of the spots are made available through the BGW2 matrix: the maximum spot identity is the number of spots. The area covered by the spots is estimated by the non-zero cells of BGW2 and equation (3). The fraction of the plant part that is occupied by spots is estimated as the number of non- zero cells divided by the total number of cells (excluding the background). The average grey level of each spot can be estimated by the non-zero cells. The coordinates of each spot and its dimensions can be estimated by the BGW2 matrix cells with the same identity. A filtering can also be applied discarding spots consisting of very few pixels (less than a threshold: MinArea) because either they are noise or they are too small to be considered. 714 Using either BGW1 or BGW2 matrices the spot pixels in the original image can be isolated. The Red-Green-Blue (RGB) values for each spot pixel are used to construct three histograms, each one with 256-positions. These histograms are displayed by the application as shown in Fig. 2c. Each position of the Red the Green or the Blue histogram corresponds to the number of pixels that have the exact color level. For example, if the position 100 of the Red histogram is equal to 150, this means that 150 spot pixels have Red level equal to 100. As can be seen from Fig. 2c, each color histogram of the spot consists of a single lobe. The starting point (s), the ending point (e) and the peak position (p) of each lobe is checked to see if it resides within predefined limits: (smin, smax), (emin, emax), (pmin,pmax) respectively. These limits have been extracted statistically by observing a number of photographs with plant parts infected by the same disease when the disease parameters are defined. A disease matching score Gr is extracted: Gr = ∑ X sWs + ∑ X eWe + ∑ X pW p (5) s e p Where Ws, We, Wp are the weights of each condition (if a condition should not be taken into consideration, then the corresponding weight can be 0) and Xs, Xe, Xp are binary values indicating whether a parameter s, e, or p is within the predefined limits. For example, Xs is defined as ⎧1, if ( smin ≤ s ≤ smax )⎫ (6) Xs = ⎨ ⎬ ⎩0, otherwise ⎭ Similar conditions and score weights can be derived by the color histograms of the healthy part of a plant as well as from the global parameters like the number of spots, their area, the average grey level of all spot pixels and the average grey level of all the pixels that belong to healthy plant part, the compliance of weather conditions, etc. The disease with parameters like (smin, smax), (emin, emax), (pmin,pmax) that lead to the highest score Gr is selected as the one that matches the symptoms of the plant. 3 Smart Phone Application The smart phone application described in this paper was developed for Windows Phone platform using Visual Studio 2015 and Silverlight. In our future work, it will be ported to Universal Windows Platform in order to support more Microsoft Windows platforms. It will also be ported to Xamarin in order to support smart phone platforms different than Microsoft Windows like Android. The initial screen of the developed application is shown in Fig. 3a where the supported plant parts are displayed according to the plant disease database used. The user selects the plant or tree that he is interested in. The selection can be implemented by Combo boxes if the number of supported plants is small otherwise a List box or Long List Box can be used. Then, the user selects the photo that displays the mortified plant part. The 715 photo is displayed in the same page as shown in Fig. 3b that has been captured when the application runs on a real Lumia 535 device. BGW1 images and histograms like the ones shown in Fig. 2 are also displayed in this page. The next page (Fig. 3c) asks the user to determine the part of the plant that the selected photo displays. (a) (b) (c) (d) (e) (f) . Fig. 3. Pages of the developed application. Initial page (a) and initial page after the photo selection (b). Selection of the plan part the photo displays (c). Additional Information entered by the user (d). Geolocation for retrieving statistical and weather information about the area (e) and the results page (f). In the next page (Fig. 3d), the user is asked to give additional information and calibrate the image-processing algorithm. Some general questions that would make easier the disease recognition would concern the number of the leaves and the number of trees that are mortified, how long ago did the producer notice that the symptoms appeared, age of the plants, etc. Calibration can concern the parameters MinArea, BG and Th described in the previous section. Additional information like 716 current date and the geographical position of the plant can be defined in the next page (Fig. 3e). The local weather statistics (moisture, temperatures, etc) can be retrieved automatically if the specific date and the geographical position of the plant are known. The last page (Fig. 3f) shows the results of the analysis described in the previous section. The analysis results include the number of spots and their relative area as well as their grey level. The diagnosed disease of the plant is also listed. Moreover, advice can be given that includes some practical actions like cutting the sick branches and leaves and burning them away, or some prevention actions that can restrict the spreading of the disease. The application could even suggest appropriate herbicides, fungicides, etc, for the treatment of the disease if it is certified by the appropriate authority. 4 Experimental Results (a) (b) (c) (d) Fig. 4. Original image of a pear tree leaf (a), and the isolation of the spots (in white color) with Th=60 (b), Th=80 (c) and Th=40 (d). In this section, the ability of the described application to accurately estimate the spot number and area is investigated. Some cases where the application fails to isolate the lesions are also discussed. Finally, experimental results from the use of the proposed application in recognizing grape diseases are presented. The spot recognition method described in Section 2 has been applied to several photos of leaves from pear and citrus trees, and grapevines. The number of spots and the area they occupy on the leaf are significant inputs for the decision module of the mobile disease recognition application described in the previous section. The accuracy in estimating the features of these spots depends heavily on the value of the parameter Th. For example, in Fig. 4b, Th=60 and using this value, the number of estimated spots is 49. The number of real spots in the photo of Fig. 4a, is 45 or 50 if some quite smaller ones are taken into account. The real area of the spots is 14% of the leaf. The spot area may be a more credible measure than the number of spots since the algorithm may split some spots or may consider multiple spots as one depending on the value selected for Th. The estimated spot area of Fig. 4b is 12.34%. The MinArea parameter for this case was selected equal to 4 pixels i.e., spots consisting of fewer than 4 pixels are not considered. In order to understand the 717 importance of the appropriate Th value selection, two additional images are included in Fig. 4. Fig. 4c shows the spot recognition result if Th=80. The number of spots and their area in this case is 51 and 7.6% respectively. Although the number of spots recognized is quite close to the actual number, the estimated area is far from the real spot area. Fig. 4d shows the spot recognition result if Th=40. The number of spots and their area in this case is 77 and 20% respectively. The visualization of BGW1 matrices like the one displayed in Fig. 1b can be used to calibrate the spot recognition algorithm by selecting an appropriate value for Th in real time. (a) (b) (c) Fig. 5. A grape leaf in inverted grey scale (a), spot isolation with Th=80 (b) and Th=75 (c). The spot recognition will be difficult in some cases even if the Th parameter is tuned well. Such a case is demonstrated in Fig. 5 where a vine leaf is displayed. The problems of this case are the following: a) the background is not bright enough and has some spots that are confused with the leaf spots (difficulty in selecting the appropriate BG value), b) the color of the spots is the same as the one of the leaf veins and c) the area of each spot is surrounded by a halo brighter than the leaf color and an internal spot that is darker. As can be seen in Fig. 5, the current algorithm cannot be applied in this case since the spots are not recognized for any Th or MinArea value. Supporting these cases will need a more sophisticated but still simple algorithm. Current development focuses on isolating also the halo of the spots and their perimeters. Table 1. Average error in the estimated number of spots and their area for Pear, and Citrus tree leaves. Tree Part Th Error in the Error in the estimated estimated spot number of spots area Pear Lower surface of the leaf 60 7% 11% Citrus Upper surface of the leaf 110 7% 12% The spot recognition method was tested on spots like the ones shown in Fig. 4 for several Pear, Orange and Tangerine tree leaves that have been mortified by diseases like fungus Capnodium oleae, Erwinia Amylovora, etc. The accuracy in the estimation of the number of spots and their area is demonstrated in Table 1. In all cases MinArea=4 which seems to be a proper global value for the spot dimensions in these specific diseases. The errors could have been further reduced by choosing more 718 accurate Th values but they have been selected as multiples of 10 since the user will be difficult to do a finer tuning in real time. The developed application was also evaluated using grape leaves infected by the following diseases: Downy Mildew, Powdery Mildew, Esca, Phomopsis. The grading method defined by equations (5) and (6) was used taking into consideration the color histogram lobe starting/ending points and the peaks, the number of spots, their area, grey levels and the compliance with weather data. A training set with 20 leaf photographs was used to define the limits. After using such a small number of training samples to define the disease recognition rules, the application was tested using a benchmark of 100 photographs. These photos were classified in diseases by the application, based on the higher grade that they received for each disease. The success rate of the disease classification is shown in Table 2. These experimental results show that the proposed low complexity disease recognition method achieves an accuracy that is comparable or better with state of the art disease classification methods. For example, in (Kulkarni et al., 2012) the success rate in the disease recognition is 91%. Table 2. Grape disease classification results. Grape Disease Successful Classification Downy Mildew 90% Powdery Mildew 98% Phomopsis 88% Esca 98% 5 Conclusions An image processing technique that can be easily implemented on smart phones, capable of recognizing plant lesion features and diseases has been presented. The number of spots and their area on plant leaves showed accuracy higher than 90%. The disease recognition was tested with grape leaves with an accuracy in the classification between 88% and 98%, depending on the disease. In our future work, more diseases will be tested and the application will be ported on different mobile phone platforms. Acknowledgments. This work is protected by the provisional patent 1008484, published by the Greek Patent Office, May 12, 2015. 719 References 1. Abu-Naser, S.S., Kashkash, K.A., and Fayyad, M. (2008) Developing an Expert System for Plant Disease Diagnosis. Asian Network for Scientific Information, Journal of Artificial Intelligence 1, 2, p. 78-85. 2. Georgakopoulou, K., Spathis, C., Petrellis, N., and Birbas, A. (2016) A Capacitive to digital Converter with Automatic Range Adaptation. IEEE Trans. On Instrumentation and Measurements, 65,2 p. 336-345. 3. Kulkarni, A. and Patil, A. (2012) Applying image processing technique to detect plant diseases, International Journal of Modern Engineering Research, 2(5), p. 3361-3364. 4. Lai, J.C., Ming, B., Li, S.K., Wang, K.R., Xie, R.Z., and Gao, S.J. (2010) An Image-Based Diagnostic Expert System for Corn Diseases. Elsevier Agricultural Sciences in China 9, 8, p. 1221-1229. 5. Mix, C., Picó, F.X., and Ouborg, N.J. (2003) A Comparison of Stereomicroscope and Image Analysis For Quantifying Fruit Traits. SEED Technology 25, 1. 6. Petrellis, N. (2015) Plant Disease Diagnosis Based on Image Processing, Appropriate for Mobile Phone Implementation. Proceedings of the 7th HAICTA 2015 conference, Kavala, Sep. 17-20, 2015, p. 238-246. 7. Purcell, D.E., O’ Shea, M.G., Johnson, R.A., Kokot, S. (2009) Near infrared spectroscopy for the prediction of disease rating for Fiji leaf gall in sugarcane clones, Applied Spectroscopy, 63(4), p. 450-457. 8. Riley, M.B., Williamson, M.R., and Maloy, O. (2002) Plant disease diagnosis. The Plant Health Instructor. doi= 10.1094/PHI-I-2002-1021-01 9. Sankaran, S., Mishra, A., Eshani, R. and Davis, C. (2010) A review of advanced techniques for detecting plant diseases. Computers and Electronics in Agriculture, 72. 10. Schaad, N.W. and Frederick, R.D. (2002) Real time PCR and its application for rapid plant disease diagnostics, Canadian Journal of Plant Pathology, 24(3), p. 250-258. 720