In this paper we describe our approach for the ImageCLEFlifelog Moment Retrieval task. A total of ve runs were submitted, which used visual features, textual features or combination. The rst run was based only on the concepts gived by the organizers. In the second and third runs, we used respectively ne-tuned Googlenet and Alexnet for images description. The fourth run was based on the fusion of the two previous runs. For the fth run, we crossed the results of our best run (based on Alexnet model) with the result of XQuery FLWOR expression applied to the XML le containing the semantic location and activities data. Our architecture is implemented using Neural Network Toolbox, Parallel Computing Toolbox and GPU coder which generates CUDA from MATLAB. The results obtained are promising for a rst participation to such a task, with F1-measure@10=0.424 which placed us at third behind AILabGTi Team with 0.545 and HCMUS Team with 0.479.
CNN
LSTM ne-tuning lifelog mothe ImageCLEFlifelog dataset is to analyse the multiple multimodal source of information : the images, the semantic content and the biometric information. There is no precise format for the data. The accuracy of the images returned depends extremely on the exploitation of these data.
The main objective of experiments is to nd an automatic way to extract and analyse these multimodal data. Also, it is necessary to nd a way to translate each query (topic) into a set of concepts to match with image concepts. 2
Despite the rapid increase in the number of publications in multimedia retrieval
[
Thus, the proposed works depends on the concepts gave by benchmarks'
organizers. They make use of manual annotation to overcome this gap. Some of
them, use only CNN trained on IMAGENET to extract concepts. According to
[
Creating a new convolutional neural network is expensive in terms of expertise, equipment and the amount of annotated data needed. The training can take several weeks for the best CNNs, with many GPUs working on hundreds of 1 www.image-net.org/ 2 http://places2.csail.mit.edu/download.html 3 http://cocodataset.org thousands of annotated images. The complexity of creating CNN can be avoided by adapting publicly available pre-trained networks. We exploit the knowledge acquired on a general classi cation problem to apply it again to the lifelog context. We choose Alexnet and Googlenet and adapt the last three layer. The proposed approaches which is based on ve phases follow the schema as illustrated in Figure 1. It is divided into two parts: one online and the other o ine. The o ine part contains the (1)query analysis, (2)the ne-tuning CNN, (3)the data extraction and the (4)image inverted index generation. The online part use these several steps to (5)retrieve relevant images according to a speci c query.
The rst phase consists in analyzing the query using LSTM to match concepts with queries. The second phase is based on ne-tuning CNN to improve the search performance of the neural network. In the third phase, we use XQuery FLWOR expression to extract relevant images related to location, activity or time. The fourth phase consists in image inverted index generation to facilitate and speed up the processing time of the retrieval. The fth phase which based on doc2sequence aims at retrieving the data that is matching the query. We detailed in the following each of these phases. As rst step, we build a document which contains labeled textual descriptions of queries moments. Then we convert the words to numeric vectors by training a word embedding with dimension 100 and 50 epochs. After that, we create and train an LSTM network based on the sequences of word vectors using the stochastic gradient descent with momentum (SGDM) optimizer with learning rate 0.05. 2.2
In this section, we describe how we ne-tuned Googlenet and Alexnet. We organize 2184 images into 48 classes. To retrain GoogLeNet to classify new images, we replace the last three layers of the network: a fully connected layer, a softmax layer, and a classi cation output layer. We set the nal fully connected layer to have the same size as the number of classes in the new data set. We freeze the weights of the rst 110 layers in the network by setting the learning rates in those layers to zero to speed up network training and to prevent those layers from over tting to the new data set. We also generate data augmentation to prevent the network from over tting and memorizing the exact details of the training images. We use 70% of the images for training and 30% for validation. To retrain Alexnet, we followed the same steps as Googlenet except for freezing initial layers. Finally, we classify each images from the IMAGECLEFLifelog2018 dataset using the ne-tuned network and generate a csv le which contains image, concepts and scores. The scores are obtained from the classi cation of the CNN. 2.3
We need to extract the location, the activity or the time coming from sensor readings on mobile devices described in XML format. Indeed, some topics like 'Find the moments when I was taking a road vehicle in foreign countries' or 'Find the moments when I was having dinner at home' need other information than those contained in the images themselves. Therefore, we use XQuery FLWOR expression to extract relevant images related to location (home and longitude/latitude), to activity (transport) and to time (night). 2.4
Building an image inverted index is an important step in our approach. In this le, the images are organized in a matrix which represents the occurrence (or not) for each concept. Indeed, each image contains one or more concepts which describes the content. If a concept is contained in the image, a score between 0 and 1 is assigned to the image. Since the Narrative Clip wearable camera captures one image every 30 seconds, we obtain at the end of each day about 1440 images. So, we chose to generate an image inverted index for each day of the lifelogger. Create one image inverted index for all the images will cause a considerable loss of time in the generation of the le and also a slowness in the retrieval. To build this matrix, we rst extracted all the concepts contained in the dataset and we sorted them alphabetically. Then, we generated a matrix with images names as column and concepts as rows. For our rst run, we used the data provided by the organizers to build the image inverted index. For the other runs, we generated the matrix using the output le of the ne-tuned CNN step previously described. 2.5
All the steps described above are done o ine. Only the retrieval is online. The proposed approach for the retrieval process is based on the trained LSTM networks, the ne-tuning and the XQuery results. Firstly, we classify the query using the trained LSTM network. We then obtain the concepts that we are working on in the inverted index. Secondly, we search the concepts in the inverted index then we extract the relevant images with scores. After that, we realized an aggregation between the results obtained by the ne-tuning and those obtained by Xquery. Finally, we sort the results based on highest scores. 3
Our approach is implemented using Intel(R) Core(TM) i5-4430 CPU @3.00Ghz with 16Go RAM. We work on Windows 10 Professional using Matlab 2018a. We use Neural Network Toolbox with GPU coder which generates CUDA from MATLAB code for deep learning.
We train the ne-tuned Alexnet and Googlenet using Intel(R) Core(TM) i54200M CPU @2.50Ghz with 6Go RAM.
It lasts respectively 215 and 751 minutes for a dataset containing 2184 images. 4
We submitted 5 runs on the retrieval LMRT subtask summarized in Table 1. The rst run is exploiting only the concepts provided by the ImageCLEFlifelog organizers. We generate an inverted index for each lifelogger's day based on the information gived by the organizers of ImageCLEFlifelog2018. The retrieval returns the images that contains the concepts extracted from the topics based on the inverted index and the trained LSTM.
The second run is using the Googlenet network ne-tuned with batch size 10 and learning rate 0.0001. We train the network for 6 epochs with 912 iterations. The third run is using Alexnet network ne-tuned with same parameters as Googlenet. The fourth run is based on the result of the two previous runs. We merged the two results, sorted the scores from highest to lowest and take the n rst results where n=50. For the fth run, we crossed the results of our best run (that of Alexnet) with the result of XQuery FLWOR expression applied to the XML le containing the semantic location and activities data.
The gure 2 shows the o cial ranking metrics F1-measure@10 for the ve runs,
which gives equal importance to diversity (via CR@10) and relevance (via P@10)
[
The tables 2, 3 and 4 show the results of the runs submitted to the retrieval LMRT subtask. The results con rm that relying only on the textual concepts provided by the organizers does not give good results. We then focus on working directly on images so we choose to apply ne-tuning. We rst ne-tune with AlexNet, then with Googlenet which improved signi cantly retrieval performance. After that, we compare the results from the two previous runs and we take the images with highest scores. The fth run which gave the best result in term of F1@10 is based on ne tuning with Alexnet and extracted information from the XML les which contains semantic locations and semantic activities. We extracted this information using XQuery. 6
This paper focuses on the problem of retrieving speci c moment in lifelogger's life during ImageCLEFlifelog2018 LMRT task. We proposed a deep-learning based approach established on ve phases using ne-tuning and LSTM. The rst phase consists in analyzing the query using LSTM to match concepts with queries. The second phase is based on ne-tuning CNN to improve the search performance of the neural network. In the third phase, we use XQuery FLWOR expression to extract relevant images related to location, activity or time. The fourth phase consists in image inverted index generation to facilitate and speed up the processing time of the retrieval. The fth phase which based on doc2sequence aims at retrieving the data that is matching the query. Promising results has been o cially reported, demonstrating the e ectiveness of the proposed retrieval approach. As future work, we will focus on ne-tuning with other CNNs like Inception or Resnet. Moreover, we will consider face detection based on training cascade object detector. 7
The research leading to these results has received funding from the Ministry of Higher Education and Scienti c Research of Tunisia under the grant agreement number LR11ES48.