This paper presents our approach for semantic concept retrieval based on visual characteristics of multimedia content. The former forms a crucial initial step towards efficient event detection, resulting into meaningful interpretation of available data. In the process, a visual vocabulary is constructed in order to create a representation of the visual features of still image content. This vocabulary contains the most common visual features that are encountered within each still image database and are referred to as “region types”. Based on this vocabulary, a description is then formed to capture the association of a given image to all of its region types. Opposite to other methods, we do not describe an image based on all region types, but rather to a smaller representative subset. We show that the presented approach can be efficiently applied to still image retrieval when the goal is to retrieve semantically similar rather than visually similar image concepts by applying and evaluating our method to two well-known datasets.
It is true that the main obstacle in order to successfully implement the task of (semantic) concept retrieval in multimedia is that the actual semantic description of image objects or even of entire image scenes, is rather difficult to grasp. Several research approaches exist in the literature and they range from text-based to content-based ones. The former tend to apply text-based retrieval algorithms to a set of usually (pre-)annotated images including keywords, tags, or image titles, as well as filenames. The latter typically apply low-level image processing and analysis techniques to extract visual features from images, whereas their scalability is questionable. Most of them are limited by the existing state-of-the-art in image understanding, in the sense that they usually take a relatively low-level approach and fall short of higher-level interpretation and knowledge.
In this paper, we shall provide our research view on modelling and exploiting visual information towards efficient semantic interpretation and retrieval of multimedia content. Our goal is to create a meaningful representation of visual features of images by constructing a visual vocabulary, which will be used at a later stage for efficient concept detection. The proposed vocabulary contains the most common region types encountered within a large-scale image database. A model vector is then formed to capture the association of a given image to the visual dictionary. The goal of our work is to retrieve semantically similar images through the detection of semantic similar concepts within them. This means that given a query image, depicting a semantic concept, only the returned images that contain the same semantic concepts will be considered as relevant. Thus, images that appear visually similar, without containing the semantic concept of the query image will be considered irrelevant.
The idea of using a plain visual dictionary in order to quantize image features has
been used widely in both image retrieval and high-level concept detection. In [
The rest of this paper is structured as follows: Section 2 discusses the idea of using a visual vocabulary in order to quantize image features and presents the approach we adopt. Section 3 presents the algorithm we propose in order to create a model vector that will describe the visual properties of images. Experiments are presented in Section 4 and brief conclusions are finally drawn in Section 5. 2
As it has already been mentioned, the idea of using a visual vocabulary to quantize image features has been used in many multimedia problems. In this Section we discuss the role of the visual vocabulary and we present the approach used in this work for its construction.
Given the entire set of images of a given database and their extracted low-level features, it may easily be observed that for concepts that can be characterized as “scenes” or “materials” regions that correspond to the same concept have similar low-level descriptions. Also, images that contain the same high-level concepts are typically consisted of similar regions. For example, regions that contain the concept sky are generally visually similar, i.e. the color of most of them should be some tone of “blue”. On the other hand, images that contain sky, often are consisted of similar regions.
The aforementioned observations indicate that similar regions often co-exist with some high-level concepts. This means that region co-existences should be able to provide visual descriptions which can discriminate between the existence or not of certain high-level concepts. As indicated in the bibliography, by appropriately quantizing the regions of an image dataset, we can create efficient descriptions. Thus, this work begins with the description of the approach we follow in order to create a visual vocabulary of the most common region types encountered within the data set. Afterwards, each image will be described based on a set of region types.
In every given image Ii we first apply a segmentation algorithm, which results to a
set of regions Ri. The segmentation algorithm we use is a variation of the well-known
RSST [
After the segmentation of all images of the given image dataset, a large set F of the feature vectors of all image regions is formed. In order to select the most common region types we apply the well-known K-means clustering algorithm on F . The number of clusters which is obviously the number of region types NT is selected experimentally.
We define the visual vocabulary, formed by a set of the region types as T = nwko; k = 1; 2; : : : NT ; wk ½ F ; (1) where wk denotes the k-th region type. We should note here that after clustering the image regions in the feature space, we chose those that lie nearest to the centroid of each cluster.
We should emphasize that although a region type does not contain conceptual semantic information, it appears to carry a higher description than a low-level descriptor; i.e. one could intuitively describe a region type as “green region with a coarse texture”, but would not be necessarily able to link it to a specific concept such as vegetation, which neither is necessary a straightforward process, nor falls within the scope of the presented approach. 3
In this Section we will use and extend the ideas presented in [
Let Ri denote the set of the regions of a given image Ii after the application of the aforementioned segmentation algorithm. Moreover, let Ni denote its cardinality and rij denote its j-th region. Let us also assume that a visual vocabulary T = fwig consisting of NT region types has been constructed following the approach discussed in Section 2.
mi = fmi(1) mi(2) : : : mi(NT )g ; denote the model vector that describes the visual content of image Ii in terms of the visual dictionary. We calculated each coordinate as
mi(j) = minrij2Ri fd(f (wj ); f (rij )g ; j = 1; 2; : : : ; NT :
In this work, instead of mi we calculate a modified version of the model vector which will be referred to as m^i. After calculating the distances among each region rij and all the region types, let Wij denote an ordered set that contains all the region types with an ascending order, based on their distances dij to rij , as
In previous work we constructed a model vector by comparing all regions Ri of an image to all region types. For each region type, we chose to describe its association to the given image by the smallest distance to all image regions. Let
Wij = fwij j 8k; l · NT ; k · l : wik · wilg : (2) (3) (4) (5) (6) (7) (8)
If we compare Eq.8 with Eq.3 we can easily observe that the resulting model vector m^i, it becomes obvious that it is not constructed based on the full visual vocabulary. Instead, our method selects an appropriate subset.
The method we followed in order to construct m^i contains an intermediate step when compared to the one for the construction mi. The latter has been used successfully in a high-level concept detection problem. The use of a neural network classifier practically assigned weights to each region type. Thus, those that were not useful for For each region rij we select its closest region types, which obviously are the first K elements of Wij . This way and for each region we define the set of its K closest region types as
WiK = fwij : j · Kg : To construct a model vector m^i, instead of using the whole visual vocabulary, we choose to use an appropriate subset. This will be the union of all ordered sets WiK W K = [
WiK : i This way, the set W K consists of the closest region types of the visual dictionary to all image regions. We will construct the model vector using this set, instead of the set of all region types. Again,
m^i = f m^i(1) m^i(2) : : : m^ i(NT )g :
We define as m^i(j) the minimum distance of a region type to all image regions, thus it is calculated as m^i(j) = ½ minfd(f (wij ); f (rij ))g if wij 2 W K 0 else : the detection of a certain concept had been ignored. However, in the case of the retrieval we do not assign any weights to the region types. This means that if the model vector consisted from all region types, those with a small distance to the image regions would act as noise. In this case, retrieval would fail, as many images would have similar descriptions despite being significantly different in terms of their visual content.
To further explain the aforementioned statement, we also give a semantic explanation on why the choice of K instead of one region types for each image region is meaningful and crucial. From a simple observation of a given data set, but also intuitively, it is obvious that many high-level concepts are visually similar to more than one region types. For example, let us assume that the concept sand appears “brown” in an image of the database and “light brown” in another. Let us now consider a query image containing the concept sand. If the given visual vocabulary contains both a “brown” and a “light brown” region types, in order to retrieve both the aforementioned images of the database, their description should contain both region types and not the most similar. Thus, this way we tackle the problem of quantization.
An artificial example of the K most similar region types to each image region is depicted in Fig.1, for the case of K = 2. 0.32
0.23 0.24
0.18 0.44
0.21
0.37
0.16
In order to test the efficiency of the proposed approach, we selected two descriptive
image collections, one dataset1 created by Oliva and Torralba and one comprised by
images derived from the Corel image collection [
In order to meaningfully evaluate our work, we calculated the mean Average Precision (mAP) measure for each concept. At this point we should remind the reader that given a query image belonging to a certain semantic category, only those images within the results that belong to the same category were considered to be relevant. In addition, the well-known Euclidean distance was applied in order to compare the visual features between regions and region types. The mAP that has been achieved for several visual vocabularies and for several cases of the region types that were considered to be similar to the image regions is depicted in the following Tables.
Nt=150, K=1 Nt=150, K=2 Nt=150, K=4 Nt=270, K=1 Nt=270, K=2 Nt=270, K=5 especially with respect to concept sea, which is to be explained due to the actual nature of the concepts themselves. More specifically, concepts vegetation and sky performed also very well (i.e. mAP above 0:70), whereas mAP values obtained for concepts road and sand were average. On the other hand, mAPs for concept wave was not as high. This result can again be explained based on the actual visual properties of the particular concept, i.e. a wave is difficult to segment and discriminate in a visual manner. Fig. 5 depicts again the evolution of mAP vs. K and NT for all Corel concepts. In this case, we observe a more unified distribution of mAPs for higher values of K, which results to rather small variations to the actual values, e.g. ranging from a low of 0:401 up to 0:457 for concept wave or a low of 0:786 up to 0:891 for concept sea. In this paper we presented an approach for semantic image retrieval by exploiting a region-based visual vocabulary. More specifically, we introduced an enhanced bag-ofwords model for capturing the visual properties of images and instead of using the entire vocabulary, we selected a meaningful subset consisting of the closest region types to the image regions. This led to a simple yet effective representation of the image features that allow for efficient retrieval of semantic concepts. Early experimental results on two well-known still image datasets are promising.