In our paper we use neural networks for tuning of image feature extraction algorithms and for the analysis of orthophoto maps. In our approach we split an aerial photo into a regular grid of segments and for each segment we detect a set of features. These features describe the segment from the viewpoint of general image analysis (color, tint, etc.) as well as from the viewpoint of the shapes in the segment. We also present our computer system that support the process of the validation of extracted features using a neural network. Despite the fact that in our approach we use only general properties of an images, the results of our experiments demonstrate the usefulness of our approach.
use a neural network for tuning these algorithms. The features are then detected automatically. To assess the quality of the detection we use our own application that incorporates the neural network again. This application visualizes how the system assesses a particular images. In the case of discovered inaccuracies, the user can retroactively affect the parameters of automatic feature detection.
In the following section we discuss related approaches. The third section contains a description of features in our detection system, while the fourth section recalls some basics of tools and techniques used. The fifth section is focused on our experiment with orthophoto maps. 2
The basis for all methods and algorithms for analyzing the orthophoto maps
is digital image processing. Digital image processing is a set of technological
approaches using computer algorithms to perform image processing on digital
images ([
Many different approaches dealing with the detection and extraction of
manmade objects can be found in [
Authors of this paper have participated in the development of a
commercial Document Management System, which is used in several institutions of the
Government of Czech Republic. Experiments based on image dataset from one
of these institutions are described in [
Our approach is to describe any image in terms of image contents and in the concepts which are familiar to the users. In the following we present features we are capable of detecting. Some of the features are related to the whole image only, but many of them can also be used to describe some parts of the image. 3.1
Color features According to used colors we are able to find out whether the image is gray-scaled, and if not, whether the image is toned into some specific hue. Also we can say if the image is light, dark or if the image is cool or warm.
– grey-scaled images – color-toned images – bright or dark images – images with cool or warm color tones
The last group of features is color features. We want to describe the image in
terms of colors in the same way as a human will, but it does not suffice only to
count the ratio of one color in the image or in some area of the image. A more
complex histogram is also not enough. We should consider things like dithering,
JPEG artifacts and the subjective perception of colors by people. Using color
spatial distribution, color histograms and below mentioned shapes recognition
we are also able to detect the background color. The colors we are currently able
to detect are:
– red, green, blue, yellow, turquoise, violet, orange, pink, brown, beige, black,
white and gray
– background color
Color features detection Low-level color features were detected using a
combination of their spatial distribution and a comparison with their prototypes. The
first version of the system contained prototypes that were constructed manually
using our subjective perception. However this approach was not general enough,
therefore we have created a set of training image patterns with manually
annotated color features. To deal with human perception we have averaged the
annotation results among several annotators. Using this set we have trained the
artificial single-layer feed-forward neural network (see [
As an input we have used the pixels of particular patterns in different color models (as different models are suitable for different color features). Trained neurons (their input weights and hidden threshold) were then transformed (using the most successfull color model) into the color feature prototypes (see fig. 1). We detect all of the mentioned features as fuzzy degrees, but for selected applications we scale them down to the binary case.
Image segmentation To obtain more precise information about the processed
image, we have decided to employee an image segmentation technique. Using
the Flood fill algorithm (with eight directions, for details see [
Of course, this approach is not general. We use it only for bigger shapes and we ignore possible holes within the shapes. Because we use mostly downsampled versions of source images, we can guarantee the effectivness of processing. And because we use high quality downsampling, our results are similar to a person’s first glance. The shapes we are able to detect are: line, restangle, circle, triangle and quad.
At this moment, we detect shapes separately, but to the resulting description we save only information, whether at least one shape of such kind has been detected (i.e. the image contains one triangle) or whether there are multiple shapes of such kind (i.e. the image contains more triangles).
Currently we are thinking of using obtained distances not only for shape identification, but also for shape description. The same shape can be scaled, moved or rotated on different images, but the description using relative distance changes is still the same (up to index rotation). The more different angles we use, the more precise description we obtain.
Anomalies Since the orthophoto maps are created from long distance, interesting
objects are often relatively small and vaguely bound in the image. For this reason
we have incorporated the concept of anomalies (see [
As you will see in the experiment section, this concept became very important in our approach. Figure 2 contains highlighted samples of various shapes detected in the orthophoto maps. The Neural network (or more precisely artificial neural network) is a computational model inspired by biological processes. This network consists of interconnected artificial neurons which transform excitation of input synapses to output excitation. Most of the neural networks can adapt themselves. There are many different variants of neural networks. Each variant is specific in its structure (whether the neurons are organized in some layers, whether the neurons can be connected to themselves, etc.), learning method (the way the neural network is adapted) and neuron activation function (the way the neuron transforms input excitation to output excitation) and its parameters.
For our purposes we use a structure consisting of an input layer of
neurons, several inner hidden neuron layers and one output layer of neurons. As the
learning method we use supervised learning, where the network is presented
repeatedly with specific samples, which are propagated towards the network
output. This output is compared with expected results and the network is (using
calculated error) adapted to minimize this error. The learning finishes after a
predefined number of learning epochs or if the error rate decreases under a
predefined constant. After the learning phase, the neural network can be presented
with another group of samples and provides its output. For more details on
neural networks consult [
Our particular network is illustrated in figure 8. We have decided to use classical bipolar-sigmoid (because we needed to represent both positive and negative examples) as an activation function of neurons (having β = 2): Simple backpropagation has been used as a learning algorithm: 2 f (x) = 1 + e−βx − 1 4wij (t + 1) = η σE σwij + α4wij (t)
The basic idea of this algorithm is to calculate the total error E of the network (computed by comparing real outputs of the network with expected ones) and then change the weights 4wij (t + 1) of the network to minimize this error. The learning rate parameter η controls the speed of weight changes. To speed up learning, we use momentum α – which updates the weight in each step also with the value from the previous step 4wij (t). 5
To verify that our set of features is capable of representing the user point of view on the images content, we have created a web application for image suggestion. In the first step, the user is presented with several random images from the data. He/she marks these images as interesting (or not interesting). Using this process the user search profile is created. In the second step the application tries to understand this profile (a set of positive and negative examples) using an artificial multilayer feed-forward neural network. In the last step, the trained network is presented with the whole dataset and suggests images which may be potentially interesting to the user.
The user can clarify his/her profile by marking further images and the process is repeated. We have used part of the profile for training and the rest for the validation of the profile to verify the meaningfulness of this profile. The score of presented images is an indication for users to add more positive (if the overall score is too low) or negative (the overall score being too high) examples. Network parameters Parameters of the neural network have been selected as follows (see fig. 8): 610 input neurons (input activation represents the degree of individual feature presence), 5 hidden neurons and one output neuron (representing the degree of image acceptance). The learning rate was η = 0.1 and momentum α = 0.1. The maximum number of iterations per learning epoch was set to 1,000. The number of input neurons correspond to the number of features in different regions of the image. Remaining parameters have been selected after several attempts of being subjectively the best. A larger number of hidden neurons often caused the overtraining of the network (good performance on the training samples with very limited ability of generalization). A larger number of iterations produced no significant improvement. Lower values failed to comply with user judgments.
Application description This application (see fig. 8) has been created as an ASP.NET Web application on the Microsoft .NET platform utilizing several other technologies such as CSS/JavaScript to improve user experience. Most of the computation time is used during the image dataset indexing, which is done only once and can be precomputed offline. The indexing of particular image takes on average 0.76 seconds and can be easily paralelized as the indexing of every particular image is a completely independent.
The neural network is recreated with every request, but in high-load environment can be stored between the requests. Application memory contains indexed image signatures only, therefore the whole application is well scalable. In our testing environment we have been able to run this application easily on an Intel 2.13 GHz processor and 4 GB RAM with a dataset containing several thousand images. Clearly the process of running the neural network with particular image signatures in every step of recommendation has its computational limits, but these limits lie far beyond the boundaries of the purpose of our experiment.
We have performed several user testing sessions where we have selected the presented set of features as being the most suitable for our purposes. Using this process we made sure that normal users can understand image analysis systems based on selected features and these users were normally able to find expected results after giving two or three positive and negative examples. The discrepancy between the user’s expectations and the output of the system is a suggestion for another iteration of feature detection tuning.
Using several mathematical models and methods, such as neural networks, we have developed and described a system which can analyze orthophoto maps, detect user-oriented features in the maps and visualize the structure of the region. In the future work we will investigate the similarities between different image segments and sources of these similarities. We would like to consider different kinds of maps and use our approach on a much wider landscape region.
This work is supported by Grant of Grant Agency of Czech Republic No. 205/09/1079.