Automatic detection and classi cation of pollen species has value for use inside of palynologic allergen studies. Traditional labeling of di erent pollen species requires an expert biologist to classify particles by sight, and is therefore time-consuming and expensive. Here, an automatic process is developed which segments the particle contour and uses the extracted features for the classi cation process. We consider shape features, texture features and aperture features and analyze which are useful. The texture features analyzed include: Gabor Filters, Fast Fourier Transform, Local Binary Patterns, Histogram of Oriented Gradients, and Haralick features. We have streamlined the process into one code base, and developed multithreading functionality to decrease the processing time for large datasets.
Currently, pollen count information is usually limited to generalizing all pollen types with no access to information regarding particular species. In order to di erentiate species, typically a trained palynologist would have to manually count samples using a microscope. Advances in image processing and machine learning enable the development of an automatic system that, given a digital image from a bright- eld microscope, can automatically detect and describe the species of pollen particles present.
We build upon previous work from within our lab which has planned the
structure for a complete personal pollen tracker [
The steps for our process are as follows: 1. Image acquisition and particle segmentation, 2. Feature extraction, and 3. Classi cation.
Our process begins with scanning glass slides of the various pollen species with a digital microscope, then segmenting these images to gather samples of individual pollen particles. These images are then further segmented to identify the pollen boundary, and the area within this boundary is used for feature extraction. 18 shape features, texture features including the Fast Fourier Transform, Local Binary Patterns, Histogram of Oriented Gradients, and Haralick features, as well as aperture features are used. These features are then trained using supervised learning to build a model for the 5 pollen species sampled. The model is then tested with ten-fold cross validation. The process is illustrated in gure 1.
Five di erent species (Alder, Birch, Hazel, Mugwort, and Sweet Grass) have been stained and prepared on glass slides for use with a common digital brighteld microscope. In order to build a robust model, all species had sample images derived from three distinct laboratory slides (using a total of 600 sample images obtained from 15 di erent slides).
For particle segmentation, each digital image is processed in order to locate and segment out a con ning square surrounding a pollen particle. First, a median blur and Gauassian blur are applied to a negative of the image in order to remove smaller particles that are background noise (often dirt or imperfections on the background). Next, a threshold is applied to the image, using the OTSU algorithm to automatically detect the histogram peak. The returned image is an optimized binary image. A second set of lters is then applied using morphological operators (iterations of erosions and dilations) to ll in the particle area. Finally, the image is converted to have a white background in preparation for further processing steps.
A blob detection algorithm is now applied in order to extract a small image surrounding each particle. This algorithm is based on four attributes { Area, Circularity, Convexity and Inertia Ratio, with parameters for \minimum" and \maximum" values for each. By setting the parameters for the expected characteristics of pollen grains, the smaller images are then found and extracted.
The last lter used on the resulting images of particles is depicted in Figure 3. Because the pollen grains settle into the slide adhesive at di erent depths, some particles will be out of focus. These blurry images will provide insu cient data especially concerning texture features, therefore we remove them from our analysis. A blur detection algorithm was developed and applied to each image: a Laplacian lter set to a manually determined threshold value determines which images are too blurry and removed from further processing steps.
Lastly, the contour surrounding each pollen particle is identi ed, using OpenCV's findContours() method. 4
Shape features
We have used 18 shape features already identi ed to be useful through previous
iterations of our research [
Shape features used: Perimeter (P ) Length of contour given by OpenCV's arcLength() function Area (A) Number of pixels contained inside the contour CRoomunpdancetsnses(sR)1 4P 2A
R
Roundness/Circularity Ratio (RC) Another measure of roundness, see [
P +pP 2 4 A Mean Distance (S) Average of the distance between the center of gravity and the contour Minimum Distance (Smin) Smallest distance between the center of gravity and the contour Maximum Distance (Smax) Longest distance between the center of gravity and the contour Ratio1 (R1) Ratio of maximum distance to minimal distance Smax=Smin Ratio2 (R2) Ratio of maximum distance to mean distance Smax=S Ratio3 (R3) Ratio of minimum distance to mean distance Smin=S Diameter (D) Longest distance between any two points along the contour Radius Dispersion (RD) Standard deviation of the distances between the center of gravity and the contour Holes (H) Sum of di erences between the Maximum Distance and the distance between center of gravity and the contour Euclidean Norm (EN2) Second Euclidean Norm RMS Mean RMS mean size Mean Distance to Boundary Average distance between every point within the area and the contour Complexity (F ) Shape complexity measure based on the ratio of the area and the mean distance to boundary 4.2
Texture feature extraction
A variety of texture features were selected due to their performance in prior
research [
Gabor Filters Gabor lters have been proven useful in image segmentation
and texture analysis [
Haralick Features Haralick features [
Local Binary Pattern (LBP) To obtain local binary patterns, a 3 by 3 pixel
window is moved over the image, and the value of the central pixel is compared
to the value of its neighbors. In the case that the neighbor is of lower value,
it is assigned a zero, and in the case of a higher value, a one. This string of
eight numbers ("00011101" for instance) is the determined local pattern. The
frequency of the occurrence of each pattern is used as the texture description.
Aperture Detection The number and type of apertures present on the pollen
surface is a typical feature used by palynologists in order to determine the pollen
species. Therefore, it seems useful to also build an automatic aperture detection
function in order to identify and count apertures as an addition feature set.
Preliminary work identifying apertures [
Once the shape, texture, and aperture features have been calculated, they are added together into a csv le. A data set of 5 species with 40 sample pollen images from 3 separate sample slides led to a total of 600 samples, each with 252 extracted features. A supervised learning process used this data for model creation, which was then tested using ten-fold cross validation. Both support vector machines and a random forest classi er showed promising (and very similar results); for the results reported here a random forest classi er was used due to faster processing on the larger data sets. The n-estimators parameter for this method was set to a typical size of 100 (increasing this number did lead to slightly improved results yet also dramatically increased processing times). 6
Using a random forest classi er on a total of 600 samples (120 each for each species) and 252 features, a model was generated with an accuracy of 87% 2%. Considering that the samples were intentionally selected for variability in their appearance and background (See Figure 2), this is an indication of a robust, reliable model that shows promise for expansion in the future to also include datasets collected from an outdoor environment.
The dataset was further modi ed into di erent versions in order to test the results using only subsets of the features available.
The above table shows the accuracies of the trained models. Using only the 18 shape features, an accuracy of 64% 3% was achieved, and adding texture information either through Gabor Filters or Haralick features substantially improved the results. Features
Accuracy Shape features 64% Shape and Gabor 76% Shape and FFT 65% Shape and LBP 65% Shape and HOG 67% Shape and Haralick 87% Shape and Aperture 67% 3% 2% 2% 3% 2% 3% 2% 7
Through this research, we have tested an expanded sample set of 5 species of pollen particles and used shape, texture and aperture features for use in classication. Use of all features led to an accuracy of 87% 2%. Through testing of individual texture features in combination with shape features, it was found that using only the shape and Haralick features resulted in an accuracy of 87% 3%. Gabor Filters also proved to be a useful feature as seen through the improved accuracy compared to using just the shape features alone. Surprisingly, the other texture features as well as the aperture features did not result in signi cant accuracy gains. One next step of research would be to investigate under which exact conditions certain texture features prove useful. In the case of the aperture features, one known limitation is that the aperture types were trained on a more limited dataset. Because the aperture detection process technique developed did have positive results in determining correct aperture positions, it would be interesting to retrain the aperture type on a wider dataset and see if this results in a more useful set of extracted features. Furthermore, extending the dataset not only beyond 600 images but especially to include more than three microscope slides per species would test against possible over tting to particular slide conditions. Future research would also include application of this process to data collected outside of a laboratory environment, as well as expansion to include more pollen species.