Lifelogging o ers a mean for people to record their day to day life. However, without proper tools for retrieving speci c moments, the large volume of collected data is rendered less useful. We present in this paper our contribution to the ImageCLEF Lifelog evaluation campaign. We describe a modular approach that covers both textual analysis and semantic enrichment of queries and targeted concept augmentation of visual data. The proposed retrieval system relies on an inverted index of visual concepts and metadata and accommodates di erent versions of clustering and ltering modules. We tested over 10 fully automatic combinations of modules and a human guided approach. We observed that with even minimal human intervention, we can obtain a signi cant increase of the F1 score, ranking fourth in the competition leader board.
Recently, interest in lifelogging appeared with the breakthrough of a ordable wearable cameras and smartphones apps. It consists in logging the daily life of an individual using various data (i.e. image, biometrics, location etc.). These objects produce a huge amount of personal data, in various forms, that could be used to build great applications to improve the users life (help for the elderly or semantic video search engine). To research this eld, a few datasets were constituted to allow researchers to compare methods through workshops, competitions and tasks.
Since 2017, IMAGEClef [
New approaches with beyond state-of-the-art performances are expected to be found during this challenge. This would allow, for instance, to build a powerful semantic search engine and to retrieve speci c moments easier. Besides this, there are a wide range of possible application domains, which explains the popularity of related workshops. In this paper we detail multiple automatic approaches as well as a human guided one which give users possibility to choose between prede ned lters. 2
The origins of image retrieval can be traced back to 1979 when a conference
on Database Techniques for Pictorial Applications was held in Florence [
For the Lifelog Moment Retrieval (LMRT) task of the LifeLog challenge, the
two major criteria to achieve are relevance and diversity. To improve the
relevance, most of the approaches relied on the extraction of concepts or features
by existing pre-trained models. This is an important time-saver in these
challenges. Some systems [
Other approaches used one binary classi er per topic or classi ers with a
minimum con dence level for each topic. Many teams also used a blur detector
to lter the images before selection. To ensure diversity, a team [
In the multimedia research eld, IMAGEClef is not alone. Many other
related workshops and tasks occur on a regular basis such as NTCIR [
Among the data delivered by the organizers, we have chosen to keep some of them and to add others. We kept the qualitative and persistent data and, therefore, did not take into account the following: lat, long, song, steps, calories, glucose, heart rate and distance. The metadata UTC-time was especially very useful and we will detail its contribution in the clustering part.
In the following, we will provide more details on the extraction of visual
concepts. The user's images were transmitted with visual concepts extracted
from several neural networks :
{ The top 10 attributes predicted by using the PlaceCNN [
In order to diversify the learning databases we added the following:
{ The top 3 labels with their scores predicted by using VGG19. This
architecture was created by VGG (Visual Geometry Group) from the University of
Oxford [
As another approach, we also built one or two SVM (Support Vector Machines)
[
The very rst step of the online system is the interpretation of a topic. How could we de ne how an image corresponds to an unde ned topic? We tried several techniques based on the idea to obtain a semantic representation of each topic. The way the query is interpreted has a direct impact on the ranking, as it will change the input of the ranking system. We chose to keep a set of keywords for each topic. In our toolbox, Python-RAKE 1 is a package used to extract basic keywords from a topic and WordNet allows to generate pre-de ned synonyms. Finally, Word2Vec models provide another way to generate synonyms. It will be more detailed in the next section. The way we combined these tools will be detailed in the next section. 3.3
For a given query, our system returns a ranking (full or partial) of the images. "How could we rank them?" is the general problem of this part. We explored several methods, from a basic counting of labels selected with a match, to the average of semantic similarities between labels and a topic. Each ranking is speci c to a well-de ned method and may be more e cient than another depending on the next processing (i.e. clustering, SVM weights...) 3.4
Another way to change the images selection, is to adjust some requirements. For instance, if we try to retrieve the images corresponding to "restaurant" in our index we would not nd 'fast food restaurant' with an exact match criterion. It may look trivial, but changing the way an image is selected opens a wide range of possibilities, especially within the NLP eld. We can easily recover images depending on their visual concepts and metadata. However, we may use some Word2Vec models to apply a similarity threshold or use a Levenshtein distance threshold and much more. In our guided method, we also used conditional lters. For instance, the topic "Driving home" where the user must be driving home from work, was treated with two conditions: the user must have been at work in the last hour and he must arrive at his home in the next fteen minutes. Combined with a primary ranking, the lters allow to narrow down the selection to suppress the noise. Both precision and recall were usually improved thanks to these approaches. 3.5
Since the evaluation method is F@10 - harmonic mean of precision and recall - it is important to augment the diversity of the selected images. That's the reason why it matters to focus on clustering in order to improve diversity without changing relevance. Consequently, we chose to explore two di erent approaches, one using temporal similarity and one using visual similarity.
We realized 12 runs (11 graded) with a combination of di erent methods for each run. This section aims at explaining these methods. Every run followed the global pipeline described below. The process will be detailed further in this section.
Keywords - For each topic, we extract the associated keywords with the Python-RAKE package. Python-RAKE cuts the sentence into words, lters with a list of stop-words and returns the remaining keywords.
Synonyms - This method helps to diversify the topic representation by adding synonyms to the basic extracted keywords. We use WordNet to generate prede ned synonyms based on the WordNet hierarchy. Then, we compute the similarity between these synonyms and the basic keywords. This similarity score enables us to select the relevant synonyms. In our runs, the similarity was computed by the GoogleNews-vector-negative300 which is a Word2Vec model.
Another method to compute similarity uses the vector representation of each words in a Word2Vec model. This similarity is useful to lter irrelevant synonyms. Our nal method consists of selecting 5 synonyms from WordNet with a threshold (0.5) on the Word2Vec model similarity. Furthermore we keep only synonyms which don't match partially to keywords. 4.2
We tested di erent solutions to choose a subset to work on. We usually used one of the following approach to reduce the working set, but in a few runs we kept the whole dataset.
Match - We rst create an inverted index. An inverted index in this case will take a label or a metadata as an input and will return all the images owning the input. Using an inverted index is time-saver and useful to interact with the dataset. Then, we recover through the index the images for which at least one label or metadata is part of the keywords.
Partial match - Similarly to match, we select labels that match partially at least one of our keywords. Two words are partial match if and only if one is a part of the other (i.e. \food\ and \fast-food\). The veri cation was done through the clause `in` in Python. This aims to increase the recall but it introduces some noise as well.
Similarity - Another approach consists in computing the semantic similarity between each key of the index and keywords. Only images that have a label similar enough to a keyword are selected. We didn't use this approach in our submitted run, however a similarity threshold at 0.6 should be e cient. 4.3
Label counting - In this basic method, the score is the number of time the image was picked in the inverted index detailed earlier in section 4.2. Topic Similarity - In this alternative method, we compute the similarity score between each label and keywords. We then pick the maximum similarity for each keyword as a score. The nal score for an image is the mean keywords scores. SVM - In terms of a visual approach, we used transfer learning to train a dedicated binary SVM for each topic. In the end, this provided us with a probability score for each image to be part of the topic based on visual features.
The rst step to create our SVM was to establish a training set, so we used the Google Search API to get images from Google Images2. The negative examples are shared between the SVMs and these are images of scenes and objects from daily life. Here are the words we chose to use for the negative images : bag, car, cat, daily life photography, dog, sh, footballer, horse, newyork, python, smartphone and sunset . The positives were scrapped by querying the title of the topic and a combination of keywords and synonyms. After the scrapping, we ltered manually the positive to avoid the noise.
Every SVM had around 2000 negative and 200 positive examples. Each image was then processed through VGG16 to extract the visual features, which are the input vectors of our SVM. As we intend to use the probability given by the SVM to establish a threshold in a few runs, we tried to use an isotonic calibration. However, as our training set wasn't big enough, we chose to keep the output probability.
SVM performances could be upgraded with a bigger training set, calibration and data augmentation. Unfortunately, we didn't have the time and the resources to deal with at this moment. 4.4 Weighting - The re nement by weighting is based on the idea to combine different scoring methods. In fact, we de ne a new ranking based on the prediction of multiple approaches.
Thresholding - In a similar way, the thresholding considers a minimum score in other methods as a prerequisite to be kept in the ranking.
Visual Clustering - To maximize the diversity of returned images, one approach was to cluster the rst 200 images with the highest score on the features extracted by our neural networks. We extracted these features with the second output layer of VGG16 as explained earlier. We clustered the images using the K-means algorithm to retrieve 10 clusters. It greatly improved the diversity of images (i.e. recall score) but inevitably reduced the precision score as only one of many correct images is returned sometimes. It did not improve the overall F@10 score. The second approach for clustering that we will present next proved to be more reliable.
Temporal Clustering - In order to nd the maximum number of moments for a speci ed topic, the temporal approach seemed to be the most logical. We wanted to form di erent clusters spaced at least one hour, and to detect noise. Thus, we chose to use DBSCAN3 (Density-based spatial clustering of applications with noise) on the rst 250 images with the highest score.
This algorithm, created in 1996 just needs two arguments: a distance and a minimum number of points to form a cluster. Consequently, we converted the UTC times into minutes and then applied the algorithm implemented in ScikitLearn. The best parameters on average for the topics were one hour for the maximal distance between two points to be considered as the same moment and 5 images minimum to form a moment.
In addition, we also used a blur lter from the OpenCV library to reject all blurry images. Once the blur lter and the clustering were completed, we had to select ten images. Finally, we chose to show the images with the best scores from 3 https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html di erent clusters. However, many other methods may be used to implement the image selection between clusters. 4.5
The last process before sending the run to the evaluation system, is to normalize
our score to obtain a con dence score ([
Automatic runs With these selected runs we wanted to establish a few comparisons between our methods. They will be analyzed in section 5. Table 1 shows the automatic runs. 1 Keywords Entire dataset Topic Similarity Time Clustering 2 Keywords Entire dataset Topic Similarity Visual Clustering 3 Keywords Entire dataset Topic Similarity | 4 Keywords Partial match Topic Similarity | 5 Keywords + Synonyms Entire dataset Topic Similarity | 6 Keywords + Synonyms Partial match Topic Similarity | 7 Keywords + Synonyms Partial match Topic Similarity Weighting by SVM 8 Keywords + Synonyms Partial match Topic Similarity Tresholding by SVM 9 Keywords + Synonyms Entire dataset SVM | 10 Keywords + Synonyms Entire dataset SVM Visual Clustering 11 Keywords + Synonyms Entire dataset SVM Time Clustering Interactive runs During our interactive runs, the user had to interpret a topic through lters. The subset selection is done by these lters. However, the keywords and synonyms are still extracted as they will be needed to compute the scoring of a few methods. Table 2 shows the interactive runs.
Automatic* Filters by Human Automatic* Automatic*
Table 3 shows the results obtained by our di erent runs in the test set. The scoring is F@10, which is the harmonic mean of two measurements computed on each topic. The rst one is the precision, whether the ten rst are linked to the given topic. The second one is the proportion of di erent moments found on the given topic. Thus, good precision is not enough to achieve a high score; both recall and precision should be high to reach a great score.
From run 1 to 6, \Mixed run\ means that additional concepts were extracted (visual) and that process were done on the label (textual). The \Mixed\ aspect of run 7 and 8 is the same, but we add the re nement by the SVM, which rely on the visual approach. Run 9 to 11 do not use the textual approach during the process, however they were trained beforehand. 1 Automatic | Mixed 0.077 2 Automatic | Mixed 0.036 3 Automatic | Mixed 0.036 4 Automatic | Mixed 0.078 5 Automatic | Mixed 0.053 6 Automatic | Mixed 0.083 7 Automatic | Mixed 0.101 8 Automatic | Mixed 0.068 9 Automatic | Visual 0.099 10 Automatic | Visual Not evaluated 11 Automatic | Visual 0.116 12 Human-Guided 0.255
Table 4 presents the comparison that our runs allow us to do. Thus we are able to de ne which method worked best in this speci c context. An attempt to generalize does not guarantee identical results. Despite the risk for Time Clustering to reduce the precision if there is not enough moments found in the K rst images, its usage constantly increased the F@10 score. In a similar way, weight a ranking with SVM's prediction shows an increase in F@10. Combining these two re nement methods may be great.
However, the winner of these runs is clearly the human-guided method. Combining the human understanding and the mixed automatic runs, it reaches 0.255 at F@10 where our automatics runs didn't surpass 0.116. Then it would be interesting, in future works, to establish a hybrid framework which asks the operator to represent topics through lters and then make a ranking with the automatic approaches. It would use SVM to weight the con dence and Time Clustering to ensure cluster diversity. Figure 2 gives an example of how it works. In this working-note we presented our approach to the LMRT task of the LifeLog IMAGEClef competition. We chose two frameworks: one fully automatic, and one with human assistance. We extracted feature vectors, and used meta-data on location and time for each image. Our approach relies on linking keywords and their synonyms from topics to our data. We made the human assistance framework available because some lters work better on some topics than others. To cluster the remaining images, we found out that the time clustering had better results than the visual clustering. The sole purpose of clustering was to improve the diversity of moments retrieved. It seems logical that the more images are separated in time, the more they can t in di erent moment. Moreover, the DBSCAN algorithm selects automatically the number of clusters and identi es noise images. Therefore, it is superior to the k-means algorithm used in visual clustering. DBSCAN did not achieve great results on visual clustering because the distance between feature vectors is not well de ned.
The main di culty regarding the LMRT task was to process a great deal of multimodal data and nd the optimal processing. Fine tuning parameters and thresholds by hand was often a good method but it limits the scalability of the system. As each topic requires slightly di erent approaches, there is still some work to do to achieve a fully automatic and adaptive system for moment retrieval such as improving the topics interpretation and the automatic setting of appropriate lters.