This paper describes the fourth edition of Lifelog challenges in ImageCLEF 2020. In this edition, the Lifelog challenges consist of two tasks which are Lifelog Moments Retrieval (LMRT) and Sport Performance Lifelog (SPLL). While the Lifelog Moments Retrieval challenge follows the same format of the previous edition, its data is a larger multimodal dataset based on the merger of three previous NTCIR Lifelog datasets, which contain approximately 191,439 images with corresponding visual concepts and other related metadata. The Sport Performance Lifelog, which is a brand new challenge, is composed of three subtasks that focus on predicting the expected performance of athletes who trained for a sport event. In summary, the ImageCLEF Lifelog 2020 receives 50 runs from six teams in total with competitive results.
Due to the widely use of wearable devices, digital sensors, and smart phones
which capture photos, biometric signals, and location information passively, a
huge amount of daily-life data is recorded by many people everyday. As a
result, there is an ever increasing reserach e ort into developing methodologies for
exploiting the potential of this data. Such lifelog data has been used for many
retrieval and analytics challenges since the inaugoral NTCIR-12 - Lifelog task [
There have been many reserch tasks addressed by these challenges, such as
lifelog retrieval, data segmentation, data enhancement/annotation and
interactive retrieval. Speci cally in the ImageCLEF lifelog challenge, we note a number
of di erent tasks, such as the Solve My Life Puzzle task in 2019 [
Details of the two challenges and the data employed are provided in section 2. In section 3, submissions and results are presented and discussed. In nal section 4, the paper is concluded with the discussion of nal remarks and future work. 2 2.1
Personal lifelog data is continually increasing in volume due to the popularity of personal wearable/portable devices for health monitoring and life recording such as smartphones, smart watches, tness bands, video cameras, biometric data devices and GPS or location devices. As a huge amount of data is created daily, there is a need for a systems that can analyse the data, index, categorize, summarize to gain deep insights from the data and support a user in some positive way.
Although many related workshops of lifelogging were held successfully for years such as three editions of NTCIR, annual Lifelog Search Challenge (LSC), and ImageCLEFlifelog 2019, we still aim to bring the attention of lifelogging to not only research groups but also to diverse audiences. Nevertheless, we continue to maintain the core task to encourage research groups to propose creative retrieval approaches to lifelog data, as well as nominating a new task to introduce a new challenge to the research community. 2.2
In this task, the participants were required to retrieve a number of speci c
moments in a lifelogger's life. Moments are de ned as "semantic events, or activities
that happened throughout the day" [
Description: Find the moment(s) when u1 built my personal computer from scratch. Narrative: To be considered relevant, u1 must be clearly in the o ce with the PC parts or uncompleted PCs on the table.
Description: Find the moment(s) when u1 was looking at items in a toyshop. Narrative: To be considered relevant, u1 must be clearly in a toyshop where various toys are being examined.
Description: Find the moment(s) when u1 was being recorded for a television show. Narrative: To be considered relevant, there must clearly be a television camera in front of u1. The moments the interviewer/cameramen is interviewing/recording u1 are also considered relevant.
Description: Find the moment(s) in 2015 and 2018 when u1 was using public transports in my home country (Ireland).
Narrative: Taking any form of public transport in Ireland is considered relevant, such as bus, taxi, train and boat.
Description: Find moment(s) in which u1 was walking by the sea taking photos or eating ice-cream.
Narrative: To be considered relevant, u1 must be taking a walk by the sea or eating ice-cream with the sea is clearly visible.
Description: Find moment(s) in 2016 and 2018 when u1 was doing grocery shopping on the weekends.
Narrative: To be considered relevant, u1 must be clearly buys or visibly interacts with products in a grocery store on the weekends.
Description: Find the moment(s) when u1 was taking a photo of a bridge. Narrative: Moments when u1 was walking on a street without stopping to take a photo of a bridge are not relevant. Any other moment showing a bridge when a photo was not being taken are also not considered to be relevant.
Description: Find the moment(s) when u1 was repairing his car in the garden. Narrative: Moments when u1 was repairing his car in the garden with the gloves on his hand. Sometimes, he also held the hammer and his phone and these moments are also considered relevant.
Description: Find the moment(s) when u1 was looking at an old clock, with owers visible, with a small monster watching u1.
Narrative: Moments when u1 was at home, looking at an old clock, with owers visible, with a lamp and two small monsters watching u1 are considered relevant.
Listing 1: Description of topics for the development set in LMRT.
Description: Find the moment when u1 was attending a praying rite with other people in the church.
Narrative: To be relevant, the moment must show u1 is currently inside the church, attending a praying rite with other people.
Description: Find the moment when u1 was looking at lifelog data on a large touchscreen on the wall.
Narrative: To be relevant, the moment must show u1 was looking at his lifelog data on the touchscreen wall (not desktop monitor).
Description: Find the moment when u1 was getting a bus to his o ce at Dublin City University or was going home by bus.
Narrative: To be relevant, u1 was on the bus (not waiting for the bus) and the destination was his home or his workplace.
Description: Find the moment when u1 was getting on a bus in the aircraft landing deck in the airport.
Narrative: To be relevant, u1 was walking out from the airplane to the bus parking in the aircraft landing deck with many airplanes visible.
Description: Find the moment when u1 was looking inside the medicine cabinet in the bathroom at home.
Narrative: To be considered relevant, u1 must be at home, looking inside the opening medicine cabinet beside a mirror in the bathroom.
Description: Find the moment when u1 was ordering fast food in the airport. Narrative: To be relevant, u1 must be at the airport and ordering food. The moments that u1 was queuing to order food are also considered relevant.
Description: Find moments when u1 was eating seafood in a restaurant in the evening time.
Narrative: The moments show u1 was eating seafood or seafood parts in any restaurant in the evening time are considered relevant.
Description: Find the moments when u1 was a round-table meeting with many people and there was pink (not red) name-cards for each person.
Narrative: The moments show u1 was at a roundtable and having a meeting with many people, and with pink name-cards visible are considered relevant.
Description: Find the moments when u1 was eating a pizza while talking to a man. Narrative: To be considered relevant, u1 must eat or hold a pizza with a man in the background.
Description: Find the moments when u1 was talking to a lady in a red top, standing directly in front of a poster hanging on a wall.
Narrative: To be relevant, u1 must be talking with a woman in red, who was standing right in front of a scienti c research poster.
Listing 2: Description of topics for the test set in LMRT. queries used in the challenge.
Given a dataset of 16 people who train for a 5km run for the sport event (e.g., daily sleeping patterns, daily heart rate, sport activities, and image logs of all food consumed during the training period), participants are required to predict the expected performance (e.g., estimated nishing time, average heart rate and calories consumption) of the trained athlete. 2.3
LMRT Task: The data is a combination of three previously released datasets
of NTCIR-Lifelog Tasks: NTCIR-12 [
Format of the metadata. The metadata was stored in a csv les, which was called the metadata table. The structure and meaning of each eld in the table are described in Table 3. Additionally, visual categories and concepts descriptors are also provided in the visual concepts table. The format of it could be found in Table 4.
SPLL Task: The data was gathered using three di erent approaches: wearable devices (Fitbtit Fitness Tracker (Fitbit Versa), Google Forms, and PMSYS. Biometric data of an individual (training athlete) is recorded using Fitbit Fitness Tracker, including 13 di erent elds of information such as daily heart rate, calories, daily sleeping patterns, sport activities, etc. Google Forms were used to collect information of meals, drinks, medications, etc. At the same time, information of subjective wellness, injuries, and training load was recorded by PMSYS system. In addition, image-logs of food consumed during the training period from at least 2 participants and self reported data like mode, stress, fatigue, readiness 2 Fitbit Fitness Tracker (FitBit Versa) - https://www. tbit.com
Calories Steps Distance Sleep Lightly active minutes Moderately active minutes Very active minutes Sedentary minutes Heart rate Time in heart rate zones Resting heart rate Exercise Sleep score Google Forms reporting Wellness Injury SRPE
Fitbit Fitbit Fitbit Fitbit Fitbit Fitbit Fitbit Fitbit Fitbit Fitbit Fitbit Fitbit Fibtit Google Form PMSYS PMSYS PMSYS
Per minute Per minute Per minute When it happens (usually daily) Per day Per day Per day Perday Per 5 seconds Per day Per day When it happens 100 entries per
le
When it happens
(usually daily)
Per day
Per day
Per day
Per day
3377529
1534705
1534705
2064
2244
2396
2396
2396
20991392
2178
1803
2440
1836
1569
1747
225
783
to train and other measurements also used for professional soccer teams [
Various cut o points are considered, e.g., X=5, 10, 20, 30, 40, 50. O cial ranking metrics are the F1-measure@10, which gives equal importance to diversity (via CR@10) and relevance (via P@10). In particular, the nal score to rank submissions of participants is the average F1-measure@10 of ten queries, which provides information of general performance of each interactive system for all 10 queries.
Participants were allowed to undertake the sub-tasks in an interactive or automatic manner. For interactive submissions, a maximum of ve minutes of search time was allowed per topic. In particular, methods that allowed interaction with real users (via Relevance Feedback (RF), for example), i.e., beside of the best performance, the way of interaction (like number of iterations using RF), or innovation level of the method (for example, new way to interact with real users) were encouraged.
SPLL: For this task, we employ two evaluation metrics to rank the submissions of participants. The primary score is to check how accurately the participants can predict whether it was an improvement or a deterioration after the training process by comparing the sign of the actual change value to the predicted one. The secondary score is the absolute di erence between the actual change and the predicted one. The primary score is ranked in descending order, and if there is a draw in the primary score, the secondary score is used to re-rank the teams. 2.5
LMRT Task. The development ground truth for the LMRT task was provided in two individual txt les: one le for the cluster ground truth and one le for the relevant image ground truth.
In the cluster ground-truth le, each line corresponded to a cluster where the rst value was the topic id, followed by the cluster id number. Lines were separated by an end-of-line character (carriage return). An example is presented below: { 1, 1 { 1, 2 { ... { 2, 1 { 2, 2 { ...
In the relevant ground-truth le, the rst value on each line was the topic id, followed by a unique photo id which is image name without the extension, and then followed by the cluster id number (that corresponded to the values in the cluster ground-truth le) separated by comma. Each line corresponded to the ground truth of one image and lines were separated by an end-of-line character (carriage return). An example is presented below: { 1, b00001216 21i6bq 20150306 174552e, 1 { 1, b00001217 21i6bq 20150306 174713e, 1 { 1, b00001218 21i6bq 20150306 174751e, 1 { 1, b00002953 21i6bq 20150316 203635e, 2 { 1, b00002954 21i6bq 20150316 203642e, 2 { ... { 2, b00000183 21i6bq 20150313 072410e, 1 { 2, b00000184 21i6bq 20150313 072443e, 1 { 2, b00000906 21i6bq 20150312 171852e, 2 { 2, b00000908 21i6bq 20150312 172005e, 2 { 2, b00000909 21i6bq 20150312 172040e, 2 { ...
SPLL Task. The ground truth was provided in one txt le. For each line in this le, the rst value was the id of the sub-task which is 1, 2 or 3 (since the SPLL task is split into three sub-tasks), followed by the id of individual (p01, p02, ..., p16), followed by the actual change in the status of the individual after the training period. Although the three sub-tasks has di erent requirements, their output format is the same, which is a number indicating the change before and after training with preceding '+' sign if the change is an increase, or '-' sign if the change is a decrease. If there is no change after the training process, a 0 value without a preceding sign is also allowed. Values in each line were separated by comma. Lines were separated by and end-of-line character (carriage return). An example is shown below: { 1, p01, +8 { 1, p10, +86 { ...
Run RUN1* RUN2* RUN3* RUN1 RUN2 RUN3 RUN4 RUN5 RUN6 RUN7 RUN1 RUN2
RUN3 Team
Run
This year, we obtained 50 valid submissions in both two tasks of ImageCLEFlifelog tasks from 6 teams, which is not as high as in previous year. However, the results of these submissions show a signi cant improvement in the nal scores compared to ImageCLEFlifelog 2019. In particular, there were 38 submissions in LMRT with 6 teams participating in the task, while only one non-organizer team submitted 10 runs in SPLL task. The submitted runs and their results are summarised in Tables 5 and 6. 3.2
In this section we provide a short description of all submitted approaches followed by the o cial result of the task.
The Organizer team continue to provided a baseline approach for the LMRT task with a web-based interactive search engine, which is an improved version
RUN1*
RUN2*
RUN1
RUN2
RUN3
RUN4
RUN5
RUN6
RUN7
RUN8
RUN9
RUN10
Notes: * submissions from the organizer teams are just for reference. The results in
this paper are o cial version of ImageCLEFlifelog 2020 tasks.
of LIFER 2.0 system [
The REGIM-Lab approaches the LMRT Task with the same strategies as
their work in ImageCLEFlifelog 2019 [
HCMUS focused on the LMRT task only this year. Their retrieval system
has three components which are query by caption, query by similar image, and
query by place and time from the metadata. For query by caption, they encoded
images using Faster R-CNN [
DCU-DDTeam interactive search engine is the improved version of their
Mysceal system in LSC'20 [24] and follows the same pipeline. The visual
concepts of each image are the combination of the given metadata and outputs from
DeepLabv3+ [
The BIDAL team is the only non-organizer team participating in both the
LMRT and SPLL tasks. For the LMRT task, the authors generated clusters by
employing a scene recognition model trained on the Google Fixmatch method
[
The UA.PT Bioinformatics team continued to employ the approaches from
last year challenge [
The o cial results are summarised in Tables 5 and 6. There are six teams participating in the LMRT task with the highest F1@10 score of 0.81 was achieved by HCMUS (Table 5). Most of the teams tried to enrich the visual concepts by deploying di erent CNNs for objects and places detection, then performing text analysis on the query and text matching. Some additional features were also added in most interactive systems such as searching for visually similar images, terms weighting for results re-ranking, context understanding before performing search, etc. The highest scoring approach by the HCMUS team, considered visual vector features extracted from CNNs when making the comparison between feature vectors to nd relative moments.
In the SPLL task, only one non-organizer team participated and they managed to achieve good scores. For the prediction of performance change, their approach gained 0.82 and 128.0 in terms of prediction accuracy and L1 distance between the prediction value and actual change respectively. 4
In the ImageCLEFlifelog 2020, most of the submitted results managed to gain high scores in both tasks. Although the set of topics for LMRT task is di erent from previous edition, participants managed to search for the relative moments on a large-scale dataset while still achieving higher scores than the results in previous edition. This proves that the proposed features and query mechanisms actually enhance the performance of their retrieval systems. Most of the teams enrich semantic visual concepts using many di erent CNNs pretrained on different datasets such as COCO, OpenImage, and Visual Genome before indexing and querying; retrieve relative images based on text matching and text retrieval algorithms; perform visual similar image search. We also note many interesting approaches from teams to enhance the a ordance and interaction of the retrieval systems, including integrating lter mechanism into free-text search, considering adding visual vector features into the nal encoded vector, clustering images into events, etc.
Regarding the number of teams and submitted runs, only 6 teams participated in the LMRT task, including an organizer team, which produced 50 submissions in total. Each team was allowed to submit up to 10 runs. For the LMRT task, among ve teams which participated in ImageCLEFlifelog 2019 (including the organizer team), four teams managed to obtain better results with the highest F1-score up to 0.81. The mean (SD) increase of nal F1-score from these ve teams is 0.25 (0.18). The new team from Dublin City University also managed to achieve the 4th rank with a 0.48 F1-score. For the SPLL task, as the task is new, only one team from The Big Data Analytics Laboratory submitted 10 runs. Their best submission achieves an accuracy of performance change and the absolute di erence between the prediction and actual change are 0.82 and 128 respectively, which is a good result. 5
This publication has emanated from research supported in party by research grants from Irish Research Council (IRC) under Grant Number GOIPG/2016/741 and Science Foundation Ireland under grant numbers SFI/12/RC/2289 and SFI/13/RC/2106. 24. Tran, L.D., Nguyen, M.D., Binh, N.T., Lee, H., Gurrin, C.: Mysceal: An experimental interactive lifelog retrieval system for lsc'20. Proceedings of the Third Annual Workshop on Lifelog Search Challenge (2020) 25. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017)