This paper describes ImageCLEFlifelog 2019, the third edition of the Lifelog task. In this edition, the task was composed of two subtasks (challenges): the Lifelog Moments Retrieval (LMRT) challenge that followed the same format as in the previous edition, and the Solve My Life Puzzle (Puzzle), a brand new task that focused on rearranging lifelog moments in temporal order. ImageCLEFlifelog 2019 received noticeably higher submissions than the previous editions, with ten teams participating resulting in a total number of 109 runs.
Since 2016 with the rst lifelog initiative, the NTCIR-12 - Lifelog task [
The ImageCLEFlifeLog2019 task at ImageCLEF 2019 [
Comparing to the previous editions, in this third edition we merged the previous two sub-tasks (challenges): Activities of Daily Living understanding (ADLT) and Lifelog Moment Retrieval (LMRT) into a single challenge (LMRT) and proposed a brand new one: Solve My Life Puzzle (Puzzle), which focused on the new ways of organising lifelog data, in particular in rearranging lifelog moments.
The details of this year two challenges will be provided in section 2. This includes also the descriptions of data and resources. For the rest of the paper, in Section 3, submissions and results are presented and discussed and in the nal section 4 the paper is concluded and nal remarks and future work are discussed. 2 2.1
An increasingly wide range of personal devices, such as smartphones, video cameras as well as wearable devices that allow capturing pictures, videos, and audio clips for every moment of our lives are becoming available. Considering the huge volume of data created, there is a need for systems that can automatically analyse the data in order to categorize, summarize and also query to retrieve the information the user may need.
Despite the increasing number of successful related workshops and panels,
lifelogging has seldom been the subject of a rigorous comparative benchmarking
exercise as, for example, the new lifelog evaluation task at NTCIR-13 [
In this task, the participants are required to retrieve a number of speci c moments in a lifeloggers life. Moments are de ned as semantic events, or activities that happened throughout the day. For example, a participant would have been required to nd and return relevant moments for the query \Find the moment(s) when the user1 is cooking in the kitchen". In this edition, particular attention was to be paid to the diversi cation of the selected moments with respect to the target scenario. The ground truth for this subtask was created using a manual annotation process. Figure 1 illustrates some examples of the moments when the lifelogger was having co ee with friends". In addition, listings 1 and 2 list all the queries used in the challenge.
T.002 Having Food in a Restaurant
food in a restaurant while away from home. Any kinds of dishes are relevant. Only
T.003 Watching Videos
T.004 Photograph of a Bridge
T.005 Grocery Shopping
T.006 Playing a Guitar
T.007 Cooking
T.008 Car Sales Showroom
T.009 Public Transportation
T.010 Paper or Book Reviewing
Listing 1: Description of topics for the development set in LMRT.
T.002 Driving home
T.003 Seeking Food in a Fridge
T.004 Watching Football
T.005 Co ee Time
T.006 Having Breakfast at Home
T.007 Having Co ee with Two Persons
T.008 Using a Smartphone Outside
T.009 Wearing a Red Plaid Shirt
T.010 Having a Meeting in China
Listing 2: Description of topics for the test set in LMRT.
Given a set of lifelog images with associated metadata (e.g., biometrics, location, etc.), but no timestamps, the participants needed to analyse these images and rearrange them in chronological order and predict the correct day (e.g. Monday or Sunday) and part of the day (e.g. morning, afternoon, or evening). Figure 2 illustrates an example of this challenge. 2.3
The data was a medium-sized collection of multimodal lifelog data over 42 days
by two lifeloggers. The contribution of this dataset over previously released
datasets was the inclusion of additional biometric data, a manual diet log and
the inclusion of conventional photos. In most cases the activities of the lifeloggers
were separate and they did not meet. However on a small number of occasions
the lifeloggers appeared in data of each other. The data consists of:
{ Multimedia Content. Wearable camera images captured at a rate of about
two images per minute and worn from breakfast to sleep. Accompanying this
image data was a time-stamped record of music listening activities sourced
from Last.FM1 and an archive of all conventional (active-capture) digital
photos taken by the lifelogger.
{ Biometrics Data. Using the FitBit tness trackers2, the lifeloggers
gathered 24 7 heart rate, calorie burn and steps. In addition, continuous blood
glucose monitoring captured readings every 15 minutes using the Freestyle
Libre wearable sensor3.
1 Last.FM Music Tracker and Recommender - https://www.last.fm/
2 Fitbit Fitness Tracker (FitBit Versa) - https://www. tbit.com
3 Freestyle Libre wearable glucose monitor - https://www.freestylelibre.ie/
(a) Evening-Monday
(b) Morning-Tuesday
(d) Afternoon-Wednesday
(e) Morning-Thursday
(f) Afternoon-Friday
(g) Afternoon-Saturday
(h) Morning-Sunday
(i) Afternoon-Sunday
{ Human Activity Data. The daily activities of the lifeloggers were captured
in terms of the semantic locations visited, physical activities (e.g. walking,
running, standing) from the Moves app4, along with a time-stamped diet-log
of all food and drink consumed.
{ Enhancements to the Data. The wearable camera images were annotated
with the outputs of a visual concept detector, which provided three types of
outputs (Attributes, Categories and Concepts). Two visual concepts which
include attributes and categories of the place in the image are extracted
using PlacesCNN [18]. The remaining one is detected object category and
its bounding box extracted by using Faster R-CNN [
Format of the metadata. The metadata was stored in a .csv les, which was called the minute-based table. The precise structured of it is described in Table 2. Additionally, extra metadata was included, such as visual categories and concepts descriptors. The format of the extra metadata could be found in Table 3. 4 Moves App for Android and iOS - http://www.moves-app.com/ LMRT For assessing performance, classic metrics were deployed. These metrics were: { Cluster Recall at X (CR@X) - a metric that assesses how many di erent clusters from the ground truth are represented among the top X results; { Precision at X (P@X) - measures the number of relevant photos among the top X results; { F1-measure at X (F1@X) - the harmonic mean of the previous two.
Various cut o points were considered, e.g., X=5, 10, 20, 30, 40, 50. O cial ranking metrics were the F1-measure@10, which gives equal importance to diversity (via CR@10) and relevance (via P@10).
Participants were allowed to undertake the sub-tasks in an interactive or automatic manner. For interactive submissions, a maximum of ve minutes of search time was allowed per topic. In particular, methods that allowed interaction with real users (via Relevance Feedback (RF), for example), i.e., beside of the best performance, the way of interaction (like number of iterations using RF), or innovation level of the method (for example, new way to interact with real users) were encouraged.
Puzzle For the Puzzle task, we used Kendall's Tau score to measure the similarity of the temporal order between the participant's temporal arrangement and ground-truth for each query. The formula of Kendall's Tau is as follows: = max 0;
C D C + D (1) where C and D are the number of concordant pairs and discordant pairs correspondingly between the participant's submission order and the one of groundtruth. In the original Kendall's Tau formula, the range of the formula is from [ 1; 1], however, we choose to narrow the range of the score to [0; 1]. It means that if the number of opposite ranking-direction pairs are greater than the quantity of the same ranking-direction pairs, participants would get nothing in Kendall's Tau score. The accuracy of part-of-day prediction was computed simply by dividing the number of correct predictions by the total number of predictions. time zone
Field name minute ID utc time local time lat lon name song steps calories historic glucose (mmol/L) scan glucose (mmol/L) heart rate distance img00 id to img 19 id cam00 id to cam14 id Finally, the primary score was computed as the average of Kendall's Tau score and accuracy of part-of-day prediction for all queries. 2.5
LMRT Task. The ground truth for the LMRT task was provided in two individual txt les: one le for the cluster ground truth and one le for the relevant image ground truth.
In the cluster ground-truth le, each line corresponded to a cluster where the rst value was the topic id, followed by cluster id number, followed by the cluster user tag separated by comma. Lines were separated by an end-of-line character (carriage return). An example is presented below: { 1, 1, Icecream by the Sea { 2, 1, DCU canteen { ... { 2, 8, Restaurant 5 { 2, 9, Restaurant 6 { ...
In the relevant ground-truth le, the rst value on each line was the topic id, followed by a unique photo id, and then followed by the cluster id number (that corresponded to the values in the cluster ground-truth le) separated by comma. Each line corresponded to the ground truth of one image and lines were separated by an end-of-line character (carriage return). An example is presented below: { 1, u1 20180528 1816 i00, 1 { 1, u1 20180528 1816 i02, 1 { 1, u1 20180528 1816 i01, 1 { 1, u1 20180528 1817 i01, 1 { 1, u1 20180528 1817 i00, 1 { 1, u1 20180528 1818 i02, 1 { ... { 2, u1 20180508 1110 i00, 1 { 2, u1 20180508 1110 i01, 1 { 2, u1 20180508 1111 i00, 1 { 2, u1 20180508 1111 i01, 1 { 2, u1 20180508 1112 i01, 1 { ...
Puzzle Task The ground truth was provided in only one individual .csv le. For each line in this le, rst value was the id of the query, followed by the id of image provided by the organisers, followed by the order that the image should be arranged in this query in temporal manner, followed by the part-of-day prediction. Values in each line were separated by comma. Lines were separated by and end-of-line character (carriage return). The values in ground-truth le were sorted by query id, and then image id column in ascending order. An example is shown below: { 1, 001.JPG, 17, 1 { 1, 002.JPG, 8, 1 { 1, 003.JPG, 9, 1 { 1, 004.JPG, 1, 1 { 1, 005.JPG, 2, 1 { ... { 8, 001.JPG, 12, 3 { 8, 002.JPG, 16, 3 { 8, 003.JPG, 13, 3 { 8, 004.JPG, 15, 3 { 8, 005.JPG, 10, 1 { ...
Run
Organiser [
RUN2*
ATS [
RUN2 RUN3 RUN4 RUN5 RUN6 RUN7 RUN8 RUN9 RUN11
RUN12
BIDAL [
RUN2
RUN3
HCMUS [
RUN2
REGIM [
RUN2 RUN3 RUN4 RUN5 RUN6 Notes: * submissions from the organizer teams are just for reference.
y submissions submitted after the o cial competition.
3.1
This year the number of participants as well as the number of submissions was considerably higher with respect to 2018: we received in total 50 valid submissions (46 o cial and 4 additional) for LMRT, and 21 (all are o cial) for Puzzle, from 10 teams representing over 10 countries. The submitted runs and their results are summarised in Tables 4 and 5. 3.2
In this section we provide a short description of all submitted approaches followed by the o cial result of the task.
The Organiser team [
Team
Organiser [
Notes: * submissions from the organizer teams are just for reference.
The REGIM-Lab [
The UPB team [
The UAPTBioinformatics (UAPT) team [
The TUC MI team [
The HCMUS team [
The BIDAL team [
The ZJUTCVR team [20] pre-processed the images with blur/cover lters to eliminate the blurred and occluded images. Then, they proposed three approaches to handle the remaining lifelog images: the two-class approach, the eleven-class approach, and the clustering approach. For two-class approach, the authors divided query topics into directories and ran a test on each directory with a ne-tuned CNN. After that, the results are classi ed into two classes based on the relevance to topic description. The eleven-class approach shared the same process with the previous method, but the results are split into 11 classes, where 10 classes are corresponding to 10 query topics and the 11th class contains irrelevant images to all 10 topics. With the clustering approach, the team inherited the procedure of two-class approach with a modi cation after the rst-round retrieval by clustering images with LVQ algorithm.
The ATS team [
The o cial results are summarised in Tables 4 and 5. For LMRT (Table 4),
eight teams participated and the highest F1@10 was submitted by HCMUS [
In the Puzzle task, four teams have participated and the highest score was obtained at the value of 0:55 by two runs (RUN03ME and RUN04ME), also from the HCMUS team. With the exception of the BIDAL team that utilised di erent functions and visual features to evaluate the similarity between each train-test pairs and proposed new grouping algorithm to decide query arrangement, most teams used BOVW to retrieve images from training data, in order to rearrange the test images based on the temporal order of the retrieved results. The highest score was achieved by applying BOVW with one million clusters, which would be a promising approach to conduct further research and improvement.
The submitted approaches in this year con rmed the trend from last year: all approaches are exploiting multi-modal instead of using only visual information. We also con rmed the importance of deep neural networks in solving these challenges: all ten participants are using directly tailored-built deep networks or exploiting the semantic concepts extracted by using deep learning methods. Unlike previous editions, we received more semi-automatic approaches, which combine human knowledge with state-of-the-art multi-modal information retrieval. Regarding the number of the signed-up teams and the submitted runs, the task keeps growing, with the highest number of registrations and participated teams. It is also a great successful that team retention rate is high with two third nonorganiser teams from last year continued to participate in this year. This again con rms how interesting and challenging lifelogging is. As next steps, we do not plan to enrich the dataset but rather provide richer and better concepts, improve the quality of the queries and narrow down the application of the challenges. 5
This publication has emanated from research supported in party by research grants from Irish Research Council (IRC) under Grant Number GOIPG/2016/741 and Science Foundation Ireland under grant numbers SFI/12/RC/2289 and SFI/13/RC/2106.
The authors thank to Thanh-An Nguyen, Trung-Hieu Hoang, and the annotation team of Software Engineering Laboratory (SELab), University of Science, VNUHCM for supporting in building the data collection as well as giving valuable discussions for the Lifelog moment retrieval (LMRT) task. 18. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and
19. Zhou, L., Hinbarji, Z., Dang-Nguyen, D.T., Gurrin, C.: Lifer: An interactive lifelog retrieval system. In: Proceedings of the 2018 ACM Workshop on The Lifelog Search
org/10.1145/3210539.3210542 20. Zhou, P., Bai, C., Xia, J.: ZJUTCVR Team at ImageCLEFlifelog2019 Lifelog Moment Retrieval Task. In: CLEF2019 Working Notes. CEUR Workshop Proceedings,