Lifelogging has taken a wide range of interest from research communities due to the increasing number of wearable and personal devices. With a large amount of data collected every day, an essential task is to organize, manage, and retrieve data e ciently. Therefore, from a collection of photos, it is necessary to rearrange them in some order that may represent a meaning sequence of events in daily life. This motivates the task of Life Puzzle in ImageCLEFlifelog 2019. To solve this task, we propose a novel method to exploit personal habits in their daily routine activities. First we group images into several clusters based on the their visual similarity and extracted visual concepts. For each image cluster, we utilize our Bag-of-Visual-Words framework to query similar scenes from personal lifelog data to predict the possible time instant in a day of the cluster. Using our proposed method, we achieve the rst place at Multimedia retrieval in CLEF - \Solve my life puzzle" challenge with competitive Kendall's T score up to 0.4 and the nal score of 0:55.
Lifelog data is a valuable source of information to provide a better understanding of people's daily activities, habits, and behaviours. Everyday, each people may, intentionally or unintentionally, create a huge amount of useful lifelog data in various formats, such as photos, video clips, audio clips, text notes, or activities logs in computers or mobile devices. Among these data format, Visual data is one of the most important and interesting sources to analyze people's daily activities.
Sometimes, visual lifelog data are provided as a collection of photos without
any speci c temporal order. In this case, it is important to arrange the given
photos in some meaning meaningful order that re ects a reasonable sequence of
daily activities. This is the motivation for the \Solve my life puzzle" challenge in
the ImageCLEFlifelog 2019 [
In the ideal case, we aim to recover the original sequence of photos corresponding to the real sequence of events that actually happen in a day of a user. However, in practice, we may not nd exactly the original photo sequence but we may rearrange a collection of photos in di erent reasonable sequences, each of which can preserve the partial chronological order of images or image groups. It should be noticed that an image sequence which may be appropriate for one user may not be suitable for another user. One user may go shopping before going work, while another user may have a di erent order of these two activities/events. Therefore, we decide to exploit personal habits of a user to estimate the appropriate timespan of certain activity/event with reference to his or her personal history lifelog data. By looking into the habit of a user, i.e. the daily routines of activities that a user may perform repeatedly in many days, we may infer the most likely image sequences from di erent possible choices.
With the relatively discrete ow of concepts, it is challenging not only to machine learning-based models but also humans, to compare the chronological order of images. Acknowledging the nature of this shortcoming, we choose to divert our approach from processing images independently to a clustering approach. This would narrow down the discreteness to an acceptable continuity of concepts' ow, which allows groups to be sorted e ciently at the group level.
Our key idea is that two images with high similarity regarding visual and
metadata information are likely to occur in the same environment at the same
timespan within a day. Thus, we rst cluster images into multiple groups based
on the similarity of visual and metadata information. Currently, we do not
propose an approach to sort images in a single group and this will be left for
future improvements. After clustering all images into separate groups, information
regarding the temporal positions of those groups can be further exploited by
looking back to the user's history with the image similarity retrieval mechanism
using our Bag-of-Visual-Words retrieval framework [
We use our propose method to solve the Life Puzzle in ImageCLEFlifelog 2019. Our method achieves the best score in three criteri: Kendall's T , part of day, and nal score. However, our method still needs further improvements to evaluate the temporal relationship of photos in each single group and to subdivide a group of photos into multiple event instances.
The content of this paper is as follows. In section 2, information about Solve my Life Puzzle challenge as well as several possible approaches are discussed. Details about our approaches in this challenge are introduced in Section 3. Experimental results together with our discussions are introduced in Section 1. Sections 5 contains our conclusions and future works in order to enhance the performance of the proposed system. 2
To promote recent works in lifelogging problems from research groups across the
world, several challenges in lifelog data retrieval had been conducted in recent
years. ImageCLEFlifelog 2018 [
Solve my Life Puzzle challenge was rst introduced in ImageCLEFlifelog 2019
[
There are several promising approaches to arrange images in reasonable sequence based on their temporal order. We can exploit the order of action/subactivity in an event/activity to sort images in one event/activity. However, this approach may be useful when photos are taken at a small enough interval, thus photos are usually from a single event/activity.
Another approach is to exploit personal habits to estimate which timespans may be appropriate for a photo. This approach is motivated by the observation that people usually repeat to perform an activity in the same context, at the same place, and at the same timespan of the day or the same day in the week, etc. As the photos in each query of the Life Puzzle of ImageCLEFlifelog 2019 are sparsely sampled in one day (25 images in a day), we decide to follow the habit-based approach.
We adopt the Bag-of-Visual-Words retrieval framework [
In this section, rstly we list some of the main problems to sort images in a reasonable temporal sequence as follows: { (P1) Sorting images in one event/activity: it is necessary for the system to understand the meaning of that activity and the ordering of sub-actions in that activity. For instance, cooking should come after taking foods or drink out of a fridge and before eating breakfast. { (P2) Sorting images in the same scene: Images should be sorted according to the visual similarity ( ne grain clustering) or location information (coarse grain clustering). If a system can only use the location information, it is di cult to distinguish di erent scenes in the same geographical region. For example, if a user in di erent rooms in the same building, we cannot organize the photos simply based on GPS information. { (P3) Sorting multiple events and actions in chronological order: the problem is how can we determine the temporal order between events and actions. In some cases, it is possible to point out the topological relationship between several pairs of events and actions, but it is extremely di cult to have a complete relationship set between every events or activities.
To solve puzzle problem for images taken at a sparse sampling rate, i.e. images are note taken at short enough time interval to belong to the same activity/event, we focus our proposed solution on the two main problems (P2) and (P3).
Figure 1 illustrates the main components of our proposed system. By extracting visual features together with metadata of each query image, a clustering system can group the input images into several clusters that have the most similarity compared to others (c.f. Section 3.2). After this step, query images are processed at the group level only. A visual-based retrieval system can look up past scenarios in the history of the same lifelogger and return possible timeframes of each image. The possible timeframe of an image group is determined by the voting scheme from the possible timeframes of each image in the group (c.f. Section 3.3). 3.2
In our rst phase, with all the metadata set given for each image, we cluster all of them in a small number of image groups for each photo collection (25 images) in each query. Our assumption is that photos in an image group are usually belongs to the same scene and in the same timespan. In fact, this assumption may be violated if the lifelogger stay in the same environment (o ce, room, shop) multiple times, i.e. event instances, in a day. However, we do not have enough visual information and given metadata to better sub-divide a group of images based on the event instances in the same environment.
In our experiments, the number of clusters ranges from 4 to 7, which are
basically based on the similarities of the rough concepts. This work can be done
by utilizing special concepts and attributes from Places 365 (e.g., indoor lighting,
natural light, open area, working, eating, etc., together with GPS location home,
work, transport. We can also employ our visual clustering scheme [
After the rst phase, we already split images into multiple clusters, a.k.a. image
groups, each of which is expected to belong to the same environment/context. For
each image in a cluster, we use our Bag-of-Visual-Words (BoVW) retrieval
framework [
From the retrieved ranklist corresponding to an image, we can infer the score for each timespan when the activity in the image may be appropriate to occur. We use the voting scheme with all the scores for all possible timespans of images in a photo cluster to determine the most appropriate timespan for a cluster.
We can de ne a timespan to cover from 1 to 2 hours, then compare the temporal order between two clusters. Finally, we sort all clusters with reference to their temporal topological order. We also use this method with 4 pre-de ned timespans to determine the part of day for each cluster.
Figure 2 illustrates our method to determine the timespan for an image cluster with respect to a reference database of the history of the same user.
For the implementation of the BoVW retrieval framework, we use
HessianA ne detector [
The test dataset provided by organizers consists of 10 queries. There are 25 images taken at di erent times of day in each query. Metadata including GPS location, Scene CNN attributes, activities, etc., are provided corresponding to each image. No timestamps are given in the metadata. The challenge is how to sort all images in each query in chronological order with given information. Beside ordering, each image must be classi ed in 4 main categories regarding di erent times of the day, including morning (4h00 AM to 11h59 AM), afternoon (12h00 PM to 4h59 PM), evening (5h00 PM to 10h59 PM), night. 4.2
Several query cases with further explanations examples will be given in this section to illustrate our method.
Query ID 6: By using GPS location, Scene CNN attributes and visual features information, 25 images of the query number 6 are clustered into 4 main groups in Figure 3.
{ Obviously, images were taken indoor and have a home tag in their GPS label are placed in the rst group. { Images taken inside a car, with a representing of steering wheel belongs to the second group. { Images taken outdoor, with natural light belong to the third group { Images taken indoor, with o ce equipment like a laptop, papers, pen, etc.
are categorized in the last group Query ID 9: Similar to query ID 6, 25 images of the query number 9 are clustered in to 5 main groups in Figure 4 { Images with appearances of a car are placed into the rst group, due to the dominant area compared to other concepts represented in the respective images.
{ Images taken inside the o ce have a uniform distribution of detected o ce supplies' concepts { Images taken on the street are characterized with high contrast and clear boundary between dark road and bright sky, with buildings. { Some images contain a grill with its related concepts (i.e beef), are grouped into this group.
{ Remaining images are taken at logger's home position.
In order to place these image groups in a timeline, di erent approaches have been conducted. Together with BoW images retrieval, we observe and conclude that a normal person is likely to start a day in a bathroom or have breakfast inside a kitchen. That person then needs to travel by car to his or her o ce by a vehicle. After walking from the car-park and having co ee with colleges, that person then stays and have various activities inside an o ce building. Obviously, the person is likely to nish his day at home and watch television. Regarding the grill activities, likely scenarios have been found in the database, logger usually has this kind of activity in the afternoon.
Our system's heuristic is based on the main concepts and metadata of each image to separate them into clusters. Hence, we do not deal with images in intra-cluster order yet and focus on inter-cluster order. By looking at the main concepts of a given image, we nd out that it is extremely di cult to clarify the real position of each image in the timeline. For instance, with images in the second group of query 6, it is impossible to conclude the user was driving from home to the o ce or going back home after a working day. Similarly, we can only assume an image taken inside the logger's home may belong to the early morning or late at night. It is almost impossible to clarify unless the system can pay its attention to special details like the lighting from windows or arti cial lights are turned on inside the image given (Figure 5). 4.3
The o cial results from challenge's organizers are given in Table 1. For each
team, we present their run with the highest nal score. The details of submission
result can be found at [
Despite having competitive results in this challenge, the system de nitely needs improvements. Currently, images belong to the same group are return in ascending lename order due to the lack of taking in speci c metadata types. This leaves a potential headroom for improvements, as should more data be used for optimal retrieval instead of relying on visual information only. Future approaches can solve this remaining problem by further digest in other metadata.
Re-ordering lifelog images in chronological order is a challenging problem due to the various activities that can happen in the same place across multiple time frame and the limitation of visual information. Although we cannot propose a complete solution to sort every single image in this challenge, our team proposes a method which can place clustered images in order.
Our clustering method based on the similarity of visual features and metadata of given images. Sorting inter-cluster of images can be proceeded by exploiting information in the past by using Bag-of-Visual-Words with Root SIFT features which can nd the most similarity images in the image database. Our work achieved the rst place winner in this year challenge with this approach.
However, there are some aspects of future improvement. The overall performance of our system can be improved if the clustering system is more robust, at that time, we can try to increase the number of clusters in each query. Besides, additional metadata utilization also proves to be a potential headroom for improvement of intra-group information retrieval in similar systems.
We would like to thank AIOZ Pte Ltd for supporting our research team. This research is partially supported by the research funding for the Software Engineering Laboratory, University of Science, Vietnam National University - Ho Chi Minh City.