In our working notes we discuss our proposed strategies for the ImageCLEFlifelog LMRT task. We used word vectors to calculate similarity scores between queries and images. To extract important moments and reduce the image amount we used image segmentation based on histograms. We enriched the given data with concepts from pretrained models and got twelve concept types for which similarity scores were calculated and accounted. Furthermore, we used tree boosting as a predictive approach. Our highest F1@10 on the training queries was 27.41% and for the test queries we obtained a maximal F1@10 of 11.70%. All of our models were applicable to generic queries in a fully automated manner.
A lifelog is a collection of structured data from the life of a person [
In most cases, a lifelog contains big amounts of data and can be used for
multiple use cases. One example is lifelogging as memory extension for patients
who su er mild cognitive impairment and episodic memory impairment [
As processing of the lifelogs by hand takes enormous e ort, computer systems
are deployed. The goal of the LMRT task is to create a system which processes
shot images and meta data from the life of a person with prede ned queries
and return appropriate images [
In this paper we will brie y discuss existing solutions and will afterwards present our proposed strategies. We also review our experiments which we will summarize at the end. We decided to only focus on queries given for user 1 because the training queries only contain queries to user 1 and in the test queries only one query is on behalf of user 2.
In the last years, we worked with deep convolutional networks [
Lately the topic of lifelog moment retrieval gained increasing attention. In the
last year's challenge six teams in total participated whereas in the year 2017 it
was only one team [
In such tasks the metrics cluster recall, precision and F1 play a great role
because as many di erent situations as relevant as possible needed to be found.
Therefore, methods were found which work automated and some of them used
human interaction with relevance feedback. The latter performs generally better
than the automated one [
The usage of image segmentation before the further query processing can
result in better scores [
Furthermore, NLP approaches with word embeddings and LSTMs had been
successfully used to compare query and image concepts [
Our approach combines segmentation, NLP methods and similarity scores without human in the loop. 3
Our main strategy was to account multiple similarity scores which were calculated between images and queries. Therefore, we extracted concepts which describe di erent areas out of a lifelog and compared these with each token from a query. Furthermore, we include an image segmentation for grouping similar moments and reducing the image amount by removing segments containing only one image. We also present a model based on supervised machine leaning which we trained with similarity scores to process the test queries. 3.1
We read all images which were shot with the wearable camera of a person. To recognise di erences between two images we extracted the colour histograms of the images by the day. Then we perform agglomerative clustering by comparing the euclidean distances of the histograms. To remove the hierarchical structure of the clusters, we attened them afterwards by merging clusters which had a distance lower than a certain threshold.
After those steps we had clusters which did not necessarily contained images from only one moment, for instance the moments user 1 is driving to work and back were in the same cluster. Because of that we separated the clusters into single segments out of coherent images. To reduce the total image amount, we removed segments which contained only one image because these images were mostly blurry or represented changes of the situation.
In the last step we combined segments cluster-wise which were maximum 15 images apart because they were most likely similar and represented the same moment. Finally, we took the segments out of the clusters because we did not needed the clusters anymore.
Results We performed experiments where we tried di erent thresholds and found a value with which we were able to extract 4302 segments containing in total 21643 out of the 63696 images of user 1. This corresponded to a reduction of 66%. Afterwards we took one image of each segment as representative and calculated the cluster recall for those 4302 images. As result we attained 88.50% cluster recall by anew reduction of 80%. The selection of the representative image did not decrease the cluster recall in comparison to taking all images. 3.2
We extracted multiple label types to describe moments from di erent views. Therefore, we used many of the given data but performed own detections with pretrained models as well. We used following types: { Categories: We used the given categories which contained Places365 labels predicted through Place CNN. Some of the categories include information about indoor and outdoor, for example museum indoor so we decided to extract two extra label types from the categories: Base Categories and Indoor/Outdoor Labels. Those labels described the location of a moment. { Attributes: We used the given attributes which contained labels from the SUNattribute dataset predicted on Place CNN. With those labels we described the environment of a moment. { COCO Concepts 1: We used the given concepts which contained labels from the COCO dataset predicted on Faster R CNN. We used them to identify objects in the images. { COCO Concepts 2: Besides the given concepts we used a pretrained YOLOv3 model to detect COCO labels. { COCO Concepts 3: Because COCO labels describe common situations very well we extracted COCO labels a third time but with a pretrained Detetron model. { Open Images Concepts: We used a pretrained model on YOLOv3 again for the detection of labels from the Open Images dataset. { Image Net Concepts: We used a pretrained model on YOLOv3 again for the prediction of labels from the ImageNet dataset. These concepts were classi cation labels, they did not identify multiple objects in images. { Daytimes: We mapped the recording times of the images into four daytimes: morning, afternoon, evening and night. { Locations: We generalised the names of the locations by de ning common locations like store, bar or bakery. { Activities: We used the activities transport and walking. { Cities: We extracted from the given time zones the city part.
Besides the raw labels we also calculated the IDF of each label to weight them later in the comparison. For each label type with scores a threshold could be de ned at which the label is seen as accurate for a certain image. In those cases all other labels were ignored and the IDFs were adjusted. For labels which were not in the word vectors we tried to nd a valid representation and if we could not nd any, we ignored the label.
In the following table all label types are shown with their amount of valid labels and if they were generated from the given data. For every image/segment a vector is created for each label type whose size is equal to the amount of labels. There were two possible ways to create these vectors, one for the images and one for the segments.
Image based Vectors For the creation of an image vector we looked a every label of a label type if it occurred in the image. If this was the case, the score of the label was written into the vector. For label types without scores an one was written into the vector if the label occurred in the image. Labels which did not occur were represented by a zero.
Segment based Vectors For the segment based vector creation we either selected the rst image of a segment and produced the vector the way described above or we combined the image vectors of all images of a segment by taking the maximum of each label value. 3.4
For the query processing we rst had to de ne the query source which could be either the query description or query title. Then we removed punctuation from the query, lowercased and tokenized it. Afterwards we removed stop words as well as query typical tokens like nd, moment or u1 and duplicates. 3.5
In the following tables the results of our tokenization process are shown. In Table 2 it is visible that the description token contain token which do not have a clear representation in any of the label types like view, taking, using or beside.
We recognized di erences in the query types comparing training and test queries, for example test query 10 contains a location but in the training queries no locations occurred. Furthermore, test query 7 asked for two persons but a special number of objects were not asked in any of the training queries. Token Vectors For each query token a vector with the same size as the image vector is created. The entries of the vector were the cosine similarities between the token and each of the labels retrieved from our word vectors. These similarities were inside the interval -1 and 1, whereby 1 represented the highest similarity. 3.6
After we retrieved both image/segment vectors (I) and token vectors we
calculated a similarity score between them. Therefore, we rst transformed the
co-domain of the token vectors into the interval [
Similarity(I; T ) = 1 kI pn
T k
We implemented an option called ceiling which replaces the entries (scores) of the image/segment vector with 1. We also include a factor p which potentiate the di erence of the vectors and an option use idf which de nes if an IDF vector should be included in the calculation. It contained the IDF of the label if the di erence of the vectors was greater than 0.5 and otherwise the reciprocal of the IDF. This way we could boost similarities and punish dissimilarities of rare labels. 3.7
After we calculated similarity scores between all images/segments and tokens for each label type we needed to account them in the next step. Therefore, we created a similarity matrix Mk for each image/segment k whose rows represented each label type j and whose columns represented each token of a query i.
We also included weights vij for each token and weights wj for each label type. Weights vij were the maximum cosine similarity that a token i could have with any label of label type j. If the maximum was negative the weight was set to zero. Weights wj were set manually.
(Mk)ij = vij wj Similarity(Ik; T ij )
Based on this matrix we calculated two di erent similarity scores. The rst method was to calculate the mean of the matrix and the second method was to take the maximum similarities of each label type and calculate the mean. We called the rst method mean and the second method labelmax. We also tried to take the maximum of the token and calculate the mean but the result on the training queries was always worse than that of both other methods. Token Clustering As extension to the described accounting method we implemented a clustering of the query tokens to group similar tokens. We merged tokens which had a cosine distance lower than 0.5 because they were very similar.
Then we calculated the similarity score as described above but for each cluster. The resulting cluster scores were taken and the mean was computed.
The idea behind this clustering was that if a query contains di erent things all of them should be considered equally. For example the token icecream and sea were in di erent clusters whereby food and restaurant were in the same cluster. If the token beach would be added to the rst query it would be added to the second cluster. Without clustering the images with the labels icecream and sea would be taken less into account than images with all tokens. 3.8
After we calculated the similarity scores, they were sorted descending for each topic. From the segments we only extracted the rst image because it was a valid representative for all other images as shown in 3.1.
To improve the cluster recall we added the option sort submission which de nes if only one image per day should be selected for the top submission entries per topic because many of the test queries contained situations which were likely occur only once per day like driving home from o ce or looking for items in a toyshop.
We only used resources which were open source. Our word vectors were
pretrained GloVe vectors from Common Crawl which had 300 dimensions and a
vocabulary of 2.2 million tokens [
Furthermore, we used real-time object detection system YOLOv3 in
combination with pretrained models to detect labels from the Open Images dataset,
ImageNet 1000 dataset and COCO dataset [
We published our source code on GitHub3. It was written in Python and uses di erent third party libraries. The project page provides instructions on how to execute the code. We used an Intel Core i5-7500T in combination with a NVIDIA GeForce GTX 1070 Mobile to run our code. 5
In this section we will present our experiments which all based on one base model which combined multiple label types and calculated similarity scores regarding to them. 5.1
As we had many parameters we rst optimized each label type for itself by running a grid search for di erent parameter settings. The best comparing settings of a label type were these which achieved the highest F1@10 on the training queries. We tried the following parameters and ran our model without segmentation, with mean comparing, unsorted, without label optimization and queries based on the description. 3 https://github.com/stefantaubert/imageclef-lifelog-2019 Results We obtained the results shown in Table 5. We found out that the Places Labels achieved a maximal F1 of 23.21%. The label types from the minute based table on the other hand achieved really low scores.
The best score of the 3 COCO Concepts was achieved by Detectron at a threshold of 0.95. The labels from ImageNet performed best using a threshold of 0.99. The IDF values were only useful for the Open Images labels which was really unexpected. The ceiling for the Places related labels was useful which was expected because the scores of the places labels were really low in comparison to the other label types. 5.2
Because the category labels achieved such high F1 we decided to process the test queries with that model.
The resulting score was 3.3% which was much lower than expected. The reasons could be that the di erence between the training and test queries were to big.
Our second model was based on all visual concept labels: Attributes, Places and COCO Concepts 1. We ran four di erent settings. Run 5 was accidentally submitted with the same submission of run 4. The results showed that the score increased after sorting the submissions for the training queries but the test queries only bene ted from it in the runs with segmentation. For the segmentation variants we considered all images of a segment.
It was noticeable that the run with segmentation achieved almost three times better scores than the normal runs. One reason could be the decreased amount of possible images which increased the probability of taking relevant images from di erent clusters. The di erence between the training and test scores was this time lower than in run 1 but still high especially for run 2 and 3. 5.4
We decided to create a model which uses tree boosting as predictive approach.
We used the similarity scores from run 2 and 4 for training a classi er. Therefore,
we took 20% of the irrelevant images and 20% of the images of each cluster for
the evaluation set and the rest for the training set. Then we trained our model
with XGBoost and binary logistic regression and depth three for the best log
loss [
Results of run 7 and 8 showed that the predictive approach was likely not suitable for such a task because the similarities simply were memorized and could not be used to predict unknown queries. In run 7 no relevant images were found and in run 8 the found images could be just random images because in total there were just 4302 images to select from. 5.5
The Visual Concept Metadata Model was an attempt to improve the Visual Concept model by adding the given metadata. We did not sort the submissions in these runs and switched over to the labelmax comparing method and optimized the labels for all following runs. We also included the weights for each token. For the segmentation variant we considered only the rst image of a segment.
The scores were really low for both test and training queries. The main reason could be that the given metadata pulled the scores down because they were really unsuitable as found out in the label optimization experiments. We obtained mostly better scores on the training queries with the labelmax comparing method which was why we used this method. It could be guessed that an adjusting of weights for the metadata could be resulted in higher scores. 5.6
As the extension with metadata did not increased the score, we tried to extend the Visual Concept Model with the self predicted concepts COCO Concepts 2 & 3, Open Images and Image Net Concepts. This time, we decided to take the titles as queries to update the previous model.
Using this model we achieved our highest score with 11.70% and we also had the highest score for the training queries with 26.16%. The segmentation decreased the score for both training and test queries. We had an increase of 7.8% F1 by taking the extra label types into account.
We combined the two last mentioned models into one big model which contained twelve label types. We also used token clustering in our models and added the common location costa co ee because two topics were asking for co ee. We obtained the results shown in Table 11.
From the results we could read out that the score could be increased by weighting the label types. The segmentation caused a decrease in the score again like in the previous model. A new highest score could not be accomplished. The following images in Fig. 1 show the returned images for this run. (a) Top 1: 1st image (b) Top 2: 1st image (c) Top 3: 1st image (d) Top 6: 5th image (e) Top 7: 7th image (f) Top 8: 2nd image (g) Top 9: 1st image (h) Top 10: 1st image Fig. 1. The returned images from run 15 showed that our model had found some relevant images at top positions. Topics 1, 3, 6, 7 and 8 contained at least one relevant image whereby for topics 2, 4, 5, 9, 10 no relevant images were returned.
For topic 1 correct images from a toyshop were found. All top ten images for topic 2 showed images of the same day when u1 was driving to work but not back home. Three relevant images were returned for topic 3 but none were found for topic 4 and 5. The reason for the latter may be that the uninformative token time was still in the query after the tokenization process.
Almost all selected images for topic 6 were relevant but were taken on the same day which led to a small cluster recall. For topic 7 we got two di erent relevant moments and for topic 8 half of the images were relevant. For the 9th topic we only got an image of u1 wearing not a plaid but a striped shirt and the images returned for the last topic contained the airport in Shanghai with many people but not a meeting. 6
We tried di erent strategies for the lifelog task. The best results could be achieved without using segmentation. One reason for that could be, that too much of the relevant images were removed or that the histogram based approach was not precise enough. Sorting of the submissions was only useful in the models with few label types. In our evaluation, we found out that the weighting of the tokens improved the scores immense, so did the use of thresholds for the label types. We could also achieve an improvement in the score by considering only the highest similarity score of all tokens per label type. Token clustering led to an increase of the score, too.
Our predictive model showed that this approach may not be suitable for such tasks. The big di erences between the training and test queries led to great di erences in the F1 scores. We found out that the usage of the title did improve the scores because less uninformative words were included.
We could have done more experiments which would have shown that some parameters had more in uences for the resulting scores. Weighting of the label types might improve our best run but because of this great amount of parameters it was di cult to nd the best experiments.
To improve our model, we could include dissimilarity scores and nd labels
which may not occur in the images even if the similarity scores were high. We
also could include more of the metadata like the heart rate and more external
data, for example poses from open pose [