Lifelog data provide potential insight analysis and understanding about people in their daily activities. However, it is still a challenging problem to index lifelog data e ciently and to provide a userfriendly interface that supports users to retrieve moments of interest. This motivates our proposed system to retrieve lifelog moment based on visual concept fusion and text-based query expansion. We rst extract visual concepts, including entities, actions, and places from images. Besides NeuralTalk, we also proposed a novel method using conceptencoded feature augmentation to generate text descriptions to exploit further semantics of images. Our proposed lifelog retrieval system allows a user to search for lifelog moment with four di erent types of queries on place, time, entity, and extra biometric data. Furthermore, the key feature of our proposed system is to automatically suggest concepts related to input query concepts to e ciently assist a user to expand a query. Experimental results on Lifelog moment retrieval dataset of ImageCLEF 2018 demonstrate the potential usage of our method and system to retrieve lifelog moments.
There are two main problems for lifelogging analysis and moment retrieval: (i) to analyze and e ciently index lifelog data, and (ii) to enhance usability and ease-of-use for users to input queries and retrieve moments of interest. To solve the rst problem, we propose to extract concepts (entities and places) [24] and generate text descriptions from images. Currently, we gather lifelog images into visual shots but we still process each image independently. For the second problem, we develop a lifelog retrieval system that supports four types of queries (place, time, entity, and metadata)[24] and automatically suggests hints to users the related concepts from an input query.
Comparing to our initial system [24], our current system for lifelog data
processing and retrieval has two main improvements to further exploit the semantic
from text descriptions for images. First, we propose a new method for text
description generation based on the spatial attention model by Kelvin Xu et. al[25]
and the semantic attention model by Quanzeng You et. al[26]. Second, we
develop the related concept recommendation module into our retrieval system for
query expansion. From an initial concept, our system can automatically suggests
potential related concepts for users to expand the query with the expectation to
cover a wider range of retrieved results. Using our proposed method and system,
we achieve the score of 0.479 for Lifelog moment retrieval (LMRT) with Lifelog
data[
In Section 2, we brie y review recent achievements in Lifelog, place retrieval, and visual instance search. Then we propose our method to o ine process lifelogging data in Section 3 and go deeply on image captioning in Section 4. Our system to assist users to nd a moment of interest based on an arbitrary query is presented in Section 5. The conclusion and open questions for future work are discussed in Section 6. 2
The comparison on evaluating the e ectiveness of information access and
retrieval systems operating over personal lifelog data has been considered for a
long time. In 2012, the tasks in NTCIR-12 which are the rst ones focus on
known-item search and activity understanding applied over lifelog data [
Moreover, they keep on enrich the lifelog data by adding more information in
semantic locations like a cafe, restaurant and physical activity such as walking,
cycling and running in ImageCLEFlifelog 2017 [
Location plays an important role in Lifelog Moment Retrieval Task (LMRT)
that increases the accuracy of the whole system. In the task, the information
about a place is very important so it is better to retrieve as much as possible
diverse images. Duc-Tien et al. have introduced the method to deal with this
problem using the dataset collected on Flickr [
Sivic has introduced rst Bag-of-visual-word (BoW) model which is on of the
most state-of-the-art approaches for video retrieval [23]. The model follows the
key assumption, which is the two similar images sharing the signi cant number
of local path matched against each other. In order to boost the performance of
Instance Search system, many techniques, such as RootSIFT feature [
Word Representations in Vector Space is a problem that learns high-quality distributed vector representations, capturing a large number of precise syntactic and semantic word presentation [17, 16]. The state of the art is Word2Vec which is based on Skip-gram model introduced by Mikolov et. al. They improve the Skip-gram in time consuming by simplifying the `hierarchical softmax' with `negative sampling' and learning regular word representation.
Generating an natural language description for one speci c image is challenging problem in Computer Vision. Although a lot of work concentrates on labeling images with xed set on visual categories, their drawback are relying on hard-coded visual concepts and sentence templates that reducing complex visual scene. To overcome this problem, Karpathy and Fei-Fei introduced NeuralTalk, state-of-the-art model, to generate the caption of an image. 3
3.1
Overview We inherit our framework in the Lifelog Search Challenge [24] to solve the LMRT challenge. However, the main di erence between the two methods is that we further take advantage of the semantic that can be provided by text descriptions generated from an image. To exploit di erent aspects as well as styles of semantics in text description, we use NeuralTalk and our Concept Augmentation with Attention method to generate di erent descriptions corresponding to one image with the expectation to get more insight information about that image.
Figure 1 presents the overview of our proposed method to o ine process lifelogging data. Our method has four main components:
GSR … Step
Concept Augmentation with Attention
{ Visual Clustering: we rst group each sequence of similar contiguous images into a visual shot, then we link visually similar shots to a scene using visual retrieval with our Bag-of-Word framework. { Concept Extraction: we extract three types of concepts from each image, including scene category and scene attributes, entities, and actions. { Image captioning: besides NeuralTalk, we propose our new method using Concept Augmentation with Attention to generate text descriptions for an image. { Augmented Data Processing: we process extra data for query re nement, including biometrics, blood pressure, blood sugar level, text data of computer activities, etc.
Comparing to our proposed method to o ine process lifelogging data [24], our enhanced method further exploit the text descriptions of images. We use NeuralTalk to generate the baseline text descriptions, and we propose our new method with Concept Augmentation and Attention for better description generation. In this section, we brie y review the three components that we reuse from our framework [24], and we reserve Section 4 to present our new method for image captioning.
Visual Clustering: Instead of processing all images, we rst group similar
contiguous images into visual shots[24]. By this way, we can get better context
of a scene in a visual shot. As images are captured with 45 second interval, the
location and pose of an entity may change in two consecutive images. Therefore,
we use FlowNet [
To link shots corresponding to the same scene, we propose a solution to use our Bag-of-Visual-Word (BoVW) framework for visual retrieval[19]. For an image x in a visual shot Si, we retrieve similar images in other shots. The distance between the two visual shots Si and Sj is determined by the mean distance between their images.
distance(Si; Sj ) = meanfdistance(x; y); f or x 2 Si and y 2 Sj g (1) To represent the similarity relationship between visual shots, we create an undirected graph with nodes as visual shots and edges. There is an undirected edge to link two nodes Si and Sj if their distance is less than a threshold maxdistance. Each connected component represents a cluster of similar visual shots, and is expected to represent the same or similar scene in real life.
Concept Extraction: We focus on three types of concepts that can be extracted from an image. We use MIT Place API [27] to determine the scene category and scene attributes of an image. To extract entities, we use Faster RCNN [22]. For possible action detection, we extract the main verb in each description generated from an image. Besides NeuralTalk, we propose our new method for image captioning, presented in Section4.
Augmented Data Processing: We feed the information about bodymetrics of the volunteers provided by the challenge to retrieve a better result. In this LMRT, for each bodymetric information, we cluster them into a range of value. Considering heart-rate for an example, we divide the range from 1 to 150 into fteen 10-unit wide periods. Then the images is clustered into group based on those period. 4
Our proposed model is based on the spatial attention model in work of Kelvin Xu et. al[25] and the semantic attention model in work of Quanzeng You et. al[26]. Our model consists of two main modules: feature extraction and caption generation.
In the feature extraction module, with an input image I, we use a deep convolutional neural network to produce a N N D feature map. We then use an object detection model to extract the labels of the objects in the image. The object detection model produces the probability that an object appears in the image or not. We choose k labels with the highest scores to avoid noise in the image. These labels are represented as one-hot vector and then multiplied with an embedding matrix E to produce L-dimension embedded vectors. Next, both the feature map and the embedded vectors are passed into the caption generation module.
In the caption generation module, the feature map and the embedded vectors are rst processed through two di erent attention models. The rst attention model uses a weight value i produced by the combination of information from previous hidden state and each feature from the feature map to show how much "attention" is on the feature vector ai in the region i of the feature map. The image context vector at the current timestep t is then produced from the feature map and the weight value.
ti = fsoftmax(fattend1(ai; ht 1))
N N X The function fsoftmax helps the model to generate weights i for each region sumed up to 1 so that the context vector would be an expected context at timestep t.
We use the similar method in the second attention model for the embedded vectors of the labels. Each vector bj in the k embedded vectors is multiplied with a weight value j and summed up to produce the label context vector. ti = fsoftmax(fattend2(bi; ht 1))
k ztb = X j bj j=1 (4) (5)
Finally, the image context vector za and the label context vector zb are fed
into an LSTM[
Image
Mask R-CNN
R e s N e t tag vectors feature map
tags attention image attention
tags attention image attention
LSTM
Word1 LSTM
Word2
Unlike the work of Kelvin Xu et. al[25] which only uses the attention on image features and the work Quanzeng You et. al[26] which only looks at the image once and then uses attention on attributes, our model combines information from both features and labels. At each time step, the model will pay attention on certain region of image and on some certain tags to generate image caption.
Our model is trained on MS COCO dataset. In our implementation, we use
the ResNet101[
Strategy In this Section, we present our system overview that can assist users to retrieve a memory in the past corresponding to a given query described in natural language text. Currently, we do not aim to make our system smart enough to automatically analyze and fully understand the query, but the system can help a user to stepby-step retrieve then lter with multiple criteria to get the appropriate results. Our system provides multiple strategies to query for past memory in lifelogging data, but it is the user who actively decides which sequence of strategies to search for a speci c memory. 5.2
Query Query 1: Find the moments when I was preparing salad. To be considered relevant, the moments must show the lifelogger preparing a salad, in a kitchen or in an o ce environment. Eating salad is not considered relevant. Preparing other types of food is not considered relevant.
First of all, we consider on locations of the scenario. In this scenario, we focus on kitchen, o ce. Then we focus on the main context of the scenario. We mainly mining the main keyword \preparing salad". Through Word2Vec [17] model, we explore the similar and relevant words such as vegetable, fruit to extend the retrieval results. Finally, we get the candidate results and we manually choose the best results to submit. To create the variance of the result, we sort result based on time and choose the results at the di erent time. Figure 3 illustrates some image results of the query.
Query 2: Find the moments when I was with friends in Costa co ee. To be considered relevant, the moment must show at least a person together with the lifelogger in any Costa Co ee shop. Moments that show the user alone are not considered relevant.
In the query, the information of time is not mentioned so we could not apply a day periods to retrieve an image. In addition, the number of images have a context in the co ee shop is numerous and the volunteer drink co ee in di erent shops, the user has to decide which shop is Costa. Moreover, because the volunteer visit a co ee shop in di erent day, for each day, we select only one image that meets the information in the query. Figure 4 show the retrieved images corresponding to the query.
Query 3: Find the moments when I was having dinner at home. Moments in which the user was having dinner at home are relevant. Dinner in any other location is not relevant. Dinner usually occurs in the evening time.
For this scenario, we consider time period and location. We mainly explore the context that is dinner indoor in the evening. Besides, we expand the results through keywords which are similar and relevant to dinner such as eat, drink, feed. With the retrieval list, we manually select the best results as a submission. Because the dinner is a daily activity, it almost exists every day so we select images at the di erent time and di erent days to create a variance of the submission. Figure 5 illustrates retrieval images corresponding to the query. Query 4: Find the moments when I was giving a presentation to a large group of people. To be considered relevant, the moments must show more than 15 people in the audience. Such moments may be giving a public lecture or a lecture in the university.
The system could recognize that the scene of this query description is in the lecture hall where lot of students attend. The system suggests the \lecture' as a keyword for this scenario. Furthermore, \chair" and \table" are the object could be considered. From these keywords, the system return a list of images represent presentation. The user has to count the number of student appear in the image to decide which one is satis ed the query. The result images are shown in Figure 6.
The current result demonstrates the potential use of our system for moment retrieval in lifelog data. Our strategy is searching by keywords attention-based on an image description. The location and time period are mainly considered to lter the result corresponding to the context of the query. 6
Our proposal system assists user can retrieve lifelog moments based on the different types of queries (i.g. place, time, entity, extra biometric data). Leverage our system in Lifelog Search Challenge workshop to this challenge, we further explore the image caption to gain a better result. Besides, we proposed a novel method for generating image caption, it makes the description for every image can be diverse. For novice users, they could not know how to search with their existent keywords. To deal with this problem, we proposed a Keywords Recommendation by Word2Vec. We build-up a keywords dictionary which helps users can select the useful keywords for searching based on their existent keywords.
However, there are some weakness in our system. The retrieved results are not diverse. Diversity results play an important role in retrieval systems, especially in the Lifelog Moment Retrieval system. It helps to achieve a comprehensive and complete view on the query. Diversi cation of search results allows for better and faster search, gaining knowledge about di erent perspectives and viewpoints on retrieved information sources.
In following works, we will push new lters to remove noise and non-relative retrieved results. Additionally, we use new way to visualize images, speci cally, we will cluster images into many group based on their features. It helps users can have a good visualization and easier to select images. Furthermore, our longterm goal is build-up a system which automatically searches with a raw textual query, is learning on strategies of users. 15. Ionescu, B., Muller, H., Villegas, M., de Herrera, A.G.S., Eickho , C., Andrearczyk, V., Cid, Y.D., Liauchuk, V., Kovalev, V., Hasan, S.A., Ling, Y., Farri, O., Liu, J., Lungren, M., Dang-Nguyen, D.T., Piras, L., Riegler, M., Zhou, L., Lux, M., Gurrin, C.: Overview of ImageCLEF 2018: Challenges, datasets and evaluation. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018), LNCS Lecture Notes in Computer Science, Springer, Avignon, France (September 10-14 2018) 16. Mikolov, T., Chen, K., Corrado, G., Dean, J.: E cient estimation of word representations in vector space. CoRR abs/1301.3781 (2013) 17. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 26, pp. 3111{3119. Curran Associates, Inc. (2013) 18. Nguyen, V.T., Le, K.D., Tran, M.T., Fjeld, M.: Nowandthen: A social networkbased photo recommendation tool supporting reminiscence. In: Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia. pp. 159{168.
MUM '16, ACM, New York, NY, USA (2016) 19. Nguyen, V., Ngo, T.D., Tran, M., Le, D., Duong, D.A.: A combination of spatial pyramid and inverted index for large-scale image retrieval. IJMDEM 6(2), 37{51 (2015) 20. Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2007) 21. Philbin, J., Isard, M., Sivic, J., Zisserman, A.: Lost in quantization: Improving particular object retrieval in large scale image databases. In: In CVPR (2008) 22. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems (NIPS) (2015) 23. Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. In: Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2. pp. 1470{. ICCV '03, IEEE Computer Society, Washington, DC, USA (2003) 24. Truong, T.D., Dinh-Duy, T., Nguyen, V.T., Tran, M.T.: Lifelogging retrieval based on semantic concepts fusion. In: Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge. pp. 24{29. LSC '18, ACM, New York, NY, USA (2018) 25. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 2048{2057. PMLR, Lille, France (07{09 Jul 2015) 26. You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016) 27. Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 487{495. Curran Associates, Inc. (2014) 28. Zhu, C., Jegou, H., Satoh, S.: Query-adaptive asymmetrical dissimilarities for visual object retrieval. In: IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013. pp. 1705{1712. IEEE (2013)