Today image retrieval methods are rapidly growing but still, image as a type of information is not sufficient for some specific cases, especially when it comes to search for information in social networks. For this reason, we introduce a combined method including both text and image representations for identical object search. The task is solved with practical relevance to lost pets finding in social networks. The suggested method shows approximately 14.66% better quality result in comparison with the same method in which image retrieval technique reproduced only.
When solving different problems, a human uses different sources of information,
including audio, video or text information. Multimodal task settings are intended to study
approaches to exploit different modalities to improve the results of task solution.
Crossmedia (or multi-media) retrieval aims to search for information when queries and
retrieval results are of different media types: text, image, or video [
Multimodal machine translation approaches consider techniques of translation with
the use of available pictures [
Multimodal machine learning approaches aim to build models that can process and
relate information from multiple modalities [
In the current paper, this issue is studied for pets` search over the social network posts, where each post may include text (or image) only or image with its textual description. It can be useful in case a user could upload the photo or the textual portrait of his lost pet to a system and then get a response that contains the most similar recent posts aggregated from the social network. 2
In this section, we briefly discuss the related methods where text and image are both processed.
In [
In [
In [
There are various techniques to calculate the attention distribution and "value" is
used to generate the selected information. Experiments have proved that the attention
mechanism is applied in abstract generation [
We define post or document as an entity containing image or text or both image and text. We need to find images or texts where an identical object is mentioned. Formally, if we have training samples subset of texts or images, the task is to minimize the distance between documents if they contain the same object.
We propose two methods of identical objects search. The first one is represented in Section 4. For that case we consider a method based on image retrieval only. It includes image data collection (4.1), further inappropriate images filtration (4.2) and augmentation (4.3), object detection (4.4) and similarity evaluation (4.5). The second method is described in Section 5. It uses image-text joint representations. It contains collection of posts (image and corresponded description) (5.1), posts data transformation to joint feature vectors and vectors similarity evaluation (5.2). All steps for each of methods are shown in Fig. 1. Also, results are valid for the configuration below: 1. Processor: Intel Core i9-9820X, 3.3 Ghz; 2. Graphics: nVidia 1080TI, 11 Gb GPU memory; 3. Memory: 64 Gb, 4.2 Ghz; 4. OS: Ubuntu 18.04 x64. 4 4.1
Our goal is to build a dataset containing folders with images that show the same animal in each of them. For this purpose, we use crawling which is an automated data gathering method. In order to collect data, we should get all available images from the social networks and then filter out improper cases. To do that, we find Instagram and Flickr accounts that aggregate images of the desired animal (cat or dog), where for each post there is a text description with a link to the profile of the user who uploaded this image to a social network. The user profile specified in this post usually contains the desired identical animal images as expected (Fig. 2). The first 100 images are saved from every such a profile.
By means of offered algorithm, we extract 9502 cat accounts and 8070 dog accounts. 4.2
For every account received from the previous step, it is necessary to filter the images that do not contain the desired object (cat or dog). To select a neural network architecture, which is most efficient for image filtering, we gather a test dataset, which is manually labeled. ─ 1000 random images from the different Instagram accounts; ─ 118 selected images representing complicated cases on which animal is situated but at the same time the image does not have any clear outlines or ambiguity of presence takes place (Fig. 3). Such images may contain disguise, difficult view (angle), merge with background, blur, close-up, graphical effects, and so on.
To meet the classification challenges, pretrained neural network models are used. It allows us not to spend time and computational resources as well as answer the question if there is a required animal on the image. After that, we are able to make a decision if it is necessary to filter out a specific image, with sufficient precision.
In this paper, we compare the following CNN architectures each of which has the
unique properties for the classification tasks:
1. VGG16/VGG19 [
Architecture name VGG16 VGG19 ResNet50 Inception v3 Xception Inception Resnet v2
Cats Dogs
In case an account contains lower than 16 images, then the augmentation technique is applied. Based on available photos, some geometric operations (like Affine Transformation, brightness level changing, rotation, reflection and others) are performed. That is necessary in order to increase number of photos per account to ensure convergence of the subsequent algorithm. 4.4
We use object detection to focus on an object instead of image background. As well as for the classification task, we use pretrained neural networks for this topic.
The following architectures were compared:
1. Yolo v3 [
Grid-RCNN [
CenterNet [
The obtained results of object detection are specified in Table 4 and Table 5. We use subset of 118 dog photos and 155 cat photos as a test set. mAP 0.9814 0.9876 0.5975 0.9642 0.9914 0.5078 We have found the most qualitative results of the anchor-based RetinaNet neural network. We use RetinaNet + FreeAnchor neural network option for the dog and cat detection. 4.5
Siamese neural networks are used to calculate the similarity between texts or images.
We take a modified FaceNet version called EmbeddingNet1 with triplet loss
function [
If result has not changed more than 10 epochs in a row, then we conclude training is completed.
We use Instagram data for training and VK2 posts for test. VK test set contains 582 photos. Training has taken 79 epochs. As a result, we have top-1 accuracy equals 54.10% and top-5 accuracy equals 71.81% for test data. Every account takes 56.62 ms. for processing. To compute similarity, we use nVidia 1080TI graphics processor configuration.
5.1
To make identical objects search, much like the previous method we make automatic gathering firstly. This time, we collect specific VK social network posts and filter out inappropriate ones after that.
In general, the algorithm is comprised of the following stages: 1. Data crawling from social network. We make some groups crawling from VK. The first half of these groups (6 pieces) has different general-purpose topics and the last half (6 pieces) is relevant for animal shelter and volunteer particularities. We extract 39995 posts in total. 2. Texts preprocessing. We split and lemmatize all the words from the previous stage; also, we remove all punctuation marks. Words within hashtags commonly don't provide any added value therefore it should be removed primarily. After that, we remove numbers, proper names, English words (we work with the texts in Russian only), Latin symbols, and special symbols. We make basic stop-words processing (with default parameters) with Python nltk3 framework. 3. Tokens sorting. We sort all the frequent words (unigrams) in descending order and then take the first 10%. Also, we consciously increase the frequencies number of some words in order to raise the priority of the locations' references.
As a result, we have k the most frequent meaningful words ( = 370 for our case). Further each unigram frequency is normalized in relation to the total number of frequencies with the following expression:
= ∑10==0 1 .
= 1 ∑ =1 .
The obtained in (1) pw values are the unigrams tokens. The more frequently word is mentioned, the higher priority it has.
In (1): fw – frequency of the word w; pw – priority of the word w.
If we summarize all the priorities for every post, then we are able to calculate text c priority.
In (2): l – post words number; pc – priority of the text c.
If there is no any word in most frequent unigrams list, then the final priority is assigned to 0.
Text priority values are added to the array where the median is specified more precisely with every new priority. Median defined as a threshold: if priority is higher than
(1) (2) the median value, then the text is referred to our specific (cats and dogs mentions) and the corresponded target is set to 1 for that reason. Otherwise, the target is set to 0.
Median value converges with a growing number of posts as shown in Fig. 4. (3) (4) There are about 350-400 iterations for sufficient algorithm convergence. The final threshold after 1200 iterations equals 118,02.
We can calculate the current post error (residual) ec as difference between the current text priority pci and median threshold value. Applying linear interpolation, we get confidence that current post is assigned to the correct class: = ( ( ) − ).
In (3): pc – vector containing all the priorities before current priority value; pci – current priority value; f – linear interpolation function.
We use scipy 1.5.0 of Python programming language for linear interpolation function computation.
Thus, in accordance with detection results, each image in the post is corresponded to mean average precision metric quality of the proper class (dog or cat).
As a result, we have the following full probability equation representing if post contains at least one mention of dog or cat:
1 = 4
3 + 4 .
In (4) we work on the image and text data in the post. For verification, we take 100 posts for algorithm testing and receive 94 posts of 100 are relevant.
As a result, we have a method capable to collect posts of required topics automatically. We collect topics with cats and dogs and achieve sufficient data precision for our case. We make a 7292 posts dataset totally. 5.2
We compare post vectors for object identity calculation. Both image and text were
transformed into vector representations. We use pretrained DistilBERT [
Ultimately, in this article, we present new methods of training set creation. Further identical object recognition task solution is based on a search of similar objects on the images or image-text joint representations. Such representations consist of numerical vectors including both image and text features. We compare these representations to find out how close the similar objects are located in feature space after applied methods.
All the calculations are produced with Python 3.6.8 programming language and libraries: keras 2.3, tensorflow-gpu 1.15.0, mxnet+cu 1.5.1, CUDA 9.0, numpy 1.18.0, scipy 1.5.0, matplotlib 3.1.3, mmcv 0.5.4, torch 1.4.0.
As a result, we get text as an additional property has a significant positive impact. We obtain quality metrics increased approximately in 14.66%. 6
In this paper, we solve the problem of how to recognize identical objects in the social network. This issue is studied for lost pets’ search. Two methods are suggested. In the first case, we collect a dataset of identical lost pet photos and then try to find the most similar objects of required class using a siamese neural network. The second case assumes we use combined technique including both text and image representations. As they are received, we try to transform them into the joint feature vector for further comparison with the cosine distance metric.
As the results demonstrate, approach including various data types is more preferable, it gives us more high result. We obtain 14.66% better metric quality in comparison with a method in which the image retrieval method presented only. Also, we provide Github repository5 source code for reproducibility.
Potential future research will focus on designing a retrieval system that will employ image captioning additionally.