The development of technology has led to an increase in mobile devices' use to keep track of individual daily activities, as known as Lifelogging. Lifelogging has raised many research challenges, one of which is how to retrieve a specific moment in response to a user's information need. This paper presents an efficient interactive search engine for large multimodal lifelog data which is evaluated in the ImageCLEFlifelog2020 Lifelog Moment Retrieval task (LMRT). The system is the modified version of the Myscéal demonstrator used in the Lifelog Search Challenge 2020, with the addition of visual similarity and a new method of visualising results. In interactive experimentation, our system achieved an F1@10 score of 0.48 in the official submission but can be significantly improved by implementing a number of post-processing steps.
As defined in [
In this paper, we address this ad-hoc interactive retrieval challenge for lifelogs
by enhancing the performance of our pre-existing lifelog retrieval system Myscéal
[
There have been a number of interactive lifelog retrieval systems described in the
literature. One of the pioneers in this field, Doherty et. al.[
Our prior system at the LSC’20 [
The modified Myscéal retained the processing pipeline of the original version as
depicted in Figure 1, which follows a typical structure for a lifelog search engine
as introduced in [
In this version, besides the concepts detected from the descriptor, we employed the Microsoft Computer Vision API1 service to enrich the visual annotations further. To provide an optimized interactive retrieval system for LMRT, we introduced three updates to the previous system; visual similarity, user interface, and summary panel. We will now describe each of these components. 3.2
The Myscéal system was developed for the LSC’20 challenge, which required
participants to find any single relevant image for a given topic as fast as
possible. The ImageCLEFlifelog2020 LMRT task is different in that it seeks a list
of suitable images for a given query. This is a subtle difference that requires
a different retrieval engine. Firstly we implemented a visual similarity feature
to facilitate the user in finding all visually similar images to any given image.
We measured the similarities between images by using the cosine distance of
their visual features, which comprised SIFT [
The LSC’20 challenge included topics that had a definite temporal element (e.g. going to the cafe after work), which was not expected in the LMRT topics, hence 1 https://azure.microsoft.com/en-us/services/cognitive-services/ computer-vision/ we replaced the temporal query mechanism of Myscéal with a more conventional search box for users to enter the query. Whenever the user submits the query, the retrieval results are shown in the middle of the screen in a ranked list, supporting both vertical and horizontal scrolling. Each row contains concatenated grouped images, which are in decreasing order to the similarity to the query and grouped by day. Each group represents a moment or an activity in which there are many visually similar images. This structure not only reduces the visual and cognitive load on a user, but it also allows for a higher level of recall. This is because the user will have a clearer view with less identical images leading to have more time and high chances to spot out relevant photos in the ranked list.
Clicking on an image in the ranked list opens a detail window containing all images taken on the same day in an event-level view. This event view, as illustrated in 3, is arranged with three parts to show the hierarchy of event grouping. Visually similar images (in the first row) are grouped together and their thumbnail is shown as one item on the second row; the bottom row indicates the broadest level of grouping: each image in this row is a thumbnail of an activity that happens in the same location (walking in the airport, sitting inside the airplane). Using this, the user can browse the images through the day at a faster pace.
There is also a small visual similarity icon at the bottom of each image that allows users to open the similarity window listing all similar photos of the selected image. Every image in this window contains a button that opens the similarity panel.
On the top-right pane of the user interface, we show the "Saved scenes" section. Whenever users find a relevant image, they can save that photo quickly with the help of the saving icon appearing at the bottom of every image. The bottom-right map panel, which remains the same as the previous version, works as a location filter, or illustrates the location of an image.
Additionally, we introduced a new "Word List" panel appearing in the same area with the "Saved scenes" panel. This feature will list all concepts used in the retrieval system and their scores so that users can adjust (e.g., increase the scores, or remove the concepts) to have better results. 3.4
The word list and their corresponding scores comprise the query expansion
process, as introduced in [
In addition to the scoring word lists, we also provided a quick summary of the results by displaying the most frequently occurring visual concepts. This way, users can choose to remove some of the irrelevant contexts. For example, searching for kitchen might result in a distracting amount of images that show a window. Meanwhile, the user wants to look for something from another point of view. Using the summary feature, the user can remove the window to have a more focused result view. To evaluate the system’s performance, we designed a user experiment by asking three users, the expert system creator, the knowledgeable data owner, and a novice user, representing three use-cases. The developer would know clearly and could operate the system in the best way, while the data owner, or lifelogger, was expected to know the answers to all topics. In contrast, the novice user was not familiar with either the system or the dataset. All users had a maximum of 5 minutes to solve each query, which did not include the time for reading the topic. Prior to the experiment, all users were given the opportunity to familiarise themselves with the system by processing some sample topics under the guidance of the system creators. After finishing each query, users were required to record their searching time, displayed on the system, the distance of mouse movement on the screen (measured in meters), the number of mouse clicks, and the number of keystrokes (all gathered using a third-party tool). The three users were encouraged to utilize the entire 5 minutes to retrieve images from as many relevant events as possible.
Evaluation of system performance (user performance) is via three metrics
suggested by the LMRT organizers; Precision at 10 (P@10), Cluster Recall at 10
(CR@10), and the F1 Score at 10 (F1@10). Since the users were instructed to
focus on finding images from all relevant events instead of selecting all images in
a single event, the post-processing steps are necessary to improve the recall score
in LMRT. For each selected image, we expanded the result by finding all similar
photos within a given temporal window to augment the submitted results. After
performing the expansion for all images, we rearranged the result by moving the
user-selected images in each cluster to the top rank to ensure a higher P@10
score. All of the steps described above were processed automatically offline after
the users finished their search sessions. The F 1@10 scores [
The experimental results indicate that the lifelogger, who knows best about the dataset, got the highest overall score among all users in both the original and the refined answers. The authors of the system also obtained a comparable score and even achieved the same as the data owner after the post-processing step. The average F 1@10 score of the novice user was lower than the others with 0.21 and 0.39 for the raw and the modified result, respectively. One possible reason is this user was not successful in solving nearly half of the tasks in the challenge. The post-processing stage had a significant impact on the score as it improved the overall value by at least 30%. It could boost the original to the absolute value of 1 as two steps were expected to capture full precision and recall criterion. There were, however, some queries that remained the same after the second stage. Another interesting detail was that the system creator had four topics whose scores were higher than others, while that of the lifelogger was just three.
The detailed distribution of the precision and recall of each user is illustrated in Figure 4. There was a significant improvement in the precision score of the developer and the novice user. It was because they only submitted a few images within a cluster to spend more time discovering more groups leading to not enough answers for the evaluation, hence getting the low P@10 in the initial submission. The retrieving behaviors of these two users also allowed them to found out many distinct events and ranked them at the top places in the submitted results hence gained sufficient CR@10 scores. Therefore the refining steps almost had no impact on this metric and the CR@10 of both volunteers slightly remained. In contrast, the lifelogger tended to select all relevant images he saw on the screen making the answers from different clusters were not in the top 10 anymore. The post-processing algorithm could fix this issue by rearranging the results and improving the scores shown in Figure 4b. 5.2
We asked the users to keep track of their searching time, the distance of the mouse movement (in meters), number of mouse clicks, and the number of keystrokes after they finished a question. We illustrate these results in Figure 5.
All participants tended to utilize the entire five minutes for doing a query as they are suggested. It is apparent that the lifelogger needed a shorter time to finish some topics than two others because this user already had unique information about the collection and hence could form better queries and knew when to stop searching. The mouse’s distance moving on the screen, and the number of clicks reflected how users interacted with the user interface were also measured. The former measurement of the author and the novice were similar while the data owner had a slightly higher number due to the larger widescreen monitor used in the experiment (note that screen-size has an effect). This statistic indicated the simplicity of our interface design. All features were shown clearly on the screen, and even those not familiar with the system could operate effectively. The lifelogger and the author clicked much more than the novice user. It may come from the fact that they both had sufficient experience to carefully check each image’s full information prior to selecting it. In contrast, we observed that the novice user only checked the results on the main screen but not into the photo’s details. The number of keystrokes of the developer was lower than that of the novice as the author knew what were the suitable keywords to find the answer, while the latter user had tried many concepts to be able to get the result. It was noticeable that the data owner had used a keyboard less frequently compared to other users when he did the experiment with the least number of keystrokes.
(a) Retrieval Time (b) Number of keystrokes (c) Mouse movement distance (d) Number of clicks Fig. 5: The evaluation of how three users (U1: Developer, U2: Lifelogger, U3: Novice) interacted with our system.
The experiments showed that the expert user could achieve a similar result of the lifelogger who owns the dataset and also is the target of the system. Knowing the data well becomes the most significant advantage in the experiment. For instance, in the topic "Find the moment when u1 was looking at lifelog data on a large touchscreen on the wall", the lifelogger did it instantly by using his prior knowledge to search for the particular location where this event took place. Meanwhile, both the creator of the system and the novice user did not have such insights. However, this merit was not enough to secure getting the high score as sometimes the data owner missed some relevant events in a topic. This issue is reasonable when a lifelogger usually has a massive dataset, and it is onerous to remember precisely without losing any moment within it. The system, in this scenario, could help its user to solve this problem. The first topic "Find the moment when u1 was attending a praying rite with other people in the church" witnessed the developer and the novice user gain a higher score than the lifelogger. The user experiment implied that there was almost no difference in the system manipulation between users. However, we have a noticeable gap in the scores of novice users compared to others. Another notable point is that the lifelogger and the novice users rarely used the "Word list" panel but tried to search for other concepts. This is perhaps an indication that the panel was not intuitive for non-expert users and that the users could have benefited from more training on the use of the system.
Considering opportunities for improvement, firstly the location information
in the dataset seems to vary in accuracy, as stated by the lifelogger while testing.
This issue became a critical problem when users wanted to retrieve within the
specific area, such as a bus stop near their houses in topic 3 or churches in topic
1. Additionally, the detected concepts from Microsoft API service appeared with
many too specific terms leading to the decrease in the precision while searching.
There is a need for grouping these concepts to make the system more efficient.
Another thing is that our object descriptor cannot recognize the colors well in
the images, which is an essential feature in some cases like topic 10. In this work,
we used this attribute from the Microsoft service, but it can be improvable by
retraining our descriptor in some public datasets supporting these characteristics
[
In this paper, we aimed to solve the challenge of retrieving an exact moment in a lifelogger lifetime from the large scale multimodal dataset. Our system is modified from the previous version, which combined the retrained images descriptor with the query expansion mechanism, by updating the additional visual similarity functions and reorganizing the user interface to be suitable for the ImageCLEFlifelog2020 LMRT challenge. The user experiment revealed that our simple designed system could be operated easily by the novice user. The system, being utilized in the best way, can help the developer obtain equivalent results with the lifelogger after our post-processing stage.
This publication has emanated from research supported in party by research grants from Science Foundation Ireland under grant numbers SFI/12/RC/2289, SFI/13/RC/2106, 18/CRT/6223 and 18/CRT/6224. We acknowledge the support and input of the DCU ethics committee and the risk & compliance officer.