Benchmarking in Multimedia and Retrieval related research elds has a long tradition and important position within the community. Benchmarks such as the MediaEval Multimedia Benchmark or CLEF are well established and also served by the community. One major goal of these competitions beside of comparing di erent methods and approaches is also to create or promote new interesting research directions within multimedia. For example the Medico task at MediaEval with the goal of medical related multimedia analysis. Although lifelogging creates a lot of attention in the community which is shown by several workshops and special session hosted about the topic. Despite of that there exist also some lifelogging related benchmarks. For example the previous edition of the lifelogging task at ImageCLEF. The last years ImageCLEFlifelog task was well received but had some barriers that made it di cult for some researchers to participate (data size, multi modal features, etc.) The ImageCLEFlifelog 2018 tries to overcome these problems and make the task accessible for an even broader audience (e.g., pre-extracted features are provided). Furthermore, the task is divided into two subtasks (challenges). The two challenges are lifelog moment retrieval (LMRT) and the Activities of Daily Living understanding (ADLT). All in all seven teams participated with a total number of 41 runs which was an signi cant increase compared to the previous year.
Lifelogging is a research eld that gets more and more attention in the last years. This is not just due to the interesting challenges that this direction o ers (huge amount of data, complex patterns, multi-modal learning, etc.) but also because of the availability of devices. A great amount of people used devices such as smart watches and other type of sensors. These sensors in combination with smartphones that are an almost natural companion for a person nowadays enable powerful and more insightful lifelogging.
The data collected using these di erent devices is called lifelogs. A lifelog is a digital record of a persons daily routines. Such a lifelog can look di erent for di erent people depending on their habits and devices they use. Some people might record the whole day with videos others rely more on sensors. Nevertheless of the composition of such a lifelog it is clear that the collected data reaches huge dimensions for each speci c user. This calls for research with focus on systems that are able to analyze these huge amounts of data in a meaningful way. Such analysis can be manging fold and span from simple re- nding events task to summarization or information retrieval.
For example people that log their daily life want to recall certain things such as persons they saw during the day, products they found interesting in a shopping window while they were strolling trough the streets. Lifelogs can not only be used for the users need but hold also potential for other applications such as recommender systems. For example one could get recommendations based on items they focused on in a shopping window. Examples for events that a lifelogger might want to retrieve from their log can be seen in Figure 1.
The ImageCLEF2018lifeLog task is the second edition of it. The rst lifeLog
task was performed in 2017 [
This paper provides an overview of the second edition of the
ImageCLEFlifelog task which is again part of the overall benchmark campaign organized
every year by ImageCLEF [
The main goal of the task is to make use of lifelogging data and explore the possibilities that come with it. As discussed in the introduction there are several interesting and useful applications that can emerge from this data. To limit the scope for the 2018 version of the task two sub-tasks are proposed. This makes it easier for the participants to focus on a speci c outcome and participants were also allowed to submit only for one of the subtasks.
The two subtasks focus on two di erent topics. The rst one Lifelog moment retrieval (LMRT) asks the participants to retrieve a speci c moment in the daily life of a logger. Speci c queries are provided that should be answered. The second subtask, Daily Living understanding (ADLT), targets understanding of daily living over a period of time and for speci c concepts. In the following the two subtasks are described in more detail. 2.2
The participants have to retrieve a number of speci c moments in a lifeloggers life. We de ne moments as semantic events, or activities that happened throughout the day. For example, they should return the relevant moments for the query \Find the moment(s) when I was shopping for wine in the supermarket." Particular attention should be paid to the diversi cation of the selected moments with respect to the target scenario. The ground truth for this subtask was created using manual annotation.
Given a period of time, e.g., \From 13 August to 16 August" or \Every Saturday", the participants should analyse the lifelog data and provide a summarisation based on the selected concepts (provided by the task organizers) of Activities of Daily Living (ADL) and the environmental settings / contexts in which these activities take place. 1 http://www.clef-initiative.eu
Some examples of ADL concepts: \Commuting (to work or another common venue)", \Traveling (to a destination other than work, home or another common social event)", \Preparing meals (include making tea or co ee)", \Eating/drinking", and contexts: \In an o ce environment", \In a home", \In an open space". Appendix A provides the full ontology of the concepts and contexts. The summarisation should be described as the number of times and the spending time the queried event happened. For example: { ADL: \Eating/drinking: 6 times, 90 minutes", \Traveling: 1 time, 60 minutes" { Context: \In an o ce environment: 500 minutes", \In a church: 30 minutes" 2.3
The task will be split into two related subtasks using a completely new multimodal dataset which consists of 50 days of data from a lifelogger, namely: images (1,500-2,500 per day from wearable cameras), visual concepts (automatically extracted visual concepts with varying rates of accuracy), semantic content (semantic locations, semantic activities) based on sensor readings (via the Moves App) on mobile devices, biometrics information (heart rate, galvanic skin response, calorie burn, steps, etc.), music listening history. The dataset is built based on the data available for the NTCIR-13 - Lifelog 2 task. Table 1 summarises the data collection.
Format of the metadata. The metadata is stored in an .xml le, which is a simple aggregation of all users data. It is structured as follows:
The root node of the data is the USERS tag. Each user element contains all the data of that user (u1 or u2). Each user has a tag USER that contains the user ID as an attribute, example: [user id=u1]. For this year, only user u1 is considered. Inside the USER element, is his/her data:
Following that there is a tag DAYS, this tag contains the lifelogging information of that user organised per day, each day is included in a tag DAY that has the data (a tag DATA), the relative path to the directory that contains the images captured in that particular day (the tag IMAGES-DIRECTORY), then the minutes of of that day under a root tag called MINUTES.
At the start of each day there is a set of daily metatdata for that user. This data is of three forms; BIOMETRICS, ACTIVITIES & PERSONAL LOGS. The biometrics contains WEIGHT, FAT MASS, HEART RATE, SYSTOLIC blood pressure & DIASTOLIC blood pressure, which were readings taken after waking up each day. The activities contains summary activities: STEPS taken that day, DISTANCE walked in meters that day & ELEVATION climbed in meters that day. The personal logs contain HEALTH LOGS, including the TIME of reading, GLU Glucose levels in the blood, BP Blood Pressure, HR Heart Rate, MOOD manually logged every morning and sometimes a COMMENT, as well as DRINK LOGS and FOOD LOGS which were manually logged throughout the data.
Following that, the days data is organised into minutes. The MINUTES element, contains exactly 1440 child elements (called MINUTE), each child has an ID (example: [minute id=0], [minute id=1], [minute id=2] etc.), and it represent one minute in the day ordered from 0 = 12:00 AM, to 1439 = 23:59PM.
Each minute contains: 0 or 1 location information (LOCATION tag), 0 or one activity information (ACTIVITY tag), biometrics, 0 or more captured images (IMAGES tag with IMAGE child element (each element has has a relative path to the image and a unique image ID), and 0 or 1 MUSIC tag giving details of the music listened to at that point in time.
The location information is captured by Moves app (https://www.movesapp.com/), and they represent to semantic locations (Home, Work, DCU Computing building, GYM, Name of a Store, etc), or to landmark locations registered by Moves. This tag can contain information in several languages. For locations that are not (HOME) or (WORK), the GPS locations are provided. 2.4
Metrics LMRT. For assessing performance, classic metrics are deployed. These metrics are: { Cluster Recall at X (CR@X) - a metric that assesses how many di erent clusters from the ground truth are represented among the top X results; { Precision at X (P@X) - measures the number of relevant photos among the top X results; { F1-measure at X (F1@X) - the harmonic mean of the previous two.
Various cut o points are to be considered, e.g., X=5, 10, 20, 30, 40, 50. O cial ranking metrics is the F1-measure@10, which gives equal importance to diversity (via CR@10) and relevance (via P@10).
Participants are allowed to undertake the sub-tasks in an interactive or automatic manner. For interactive submissions, a maximum of ve minutes of search time is allowed per topic. In particular, the organizers would like to emphasize methods that allow interaction with real users (via Relevance Feedback (RF), for example), i.e., beside of the best performance, the way of interaction (like number of iterations using RF), or innovation level of the method (for example, new way to interact with real users) are encouraged.
Metrics ADLT. The nal score is computed as the percentage of similarity between the ground-truth and the submitted values, measured as average of the number of times and minutes di erences, as follows:
ADLscore = 21 max(0; 1 jnngntgtj ) + max(0; 1 jmmmgtgtj ) where n; ngt are the submitted and ground-truth values for how many times the events occurred, respectively, and m; mgt are the submitted and ground-truth values for how long (in minutes) the events happened, respectively. 2.5
Ground truth is provided in two individual txt les: one le for the cluster ground truth and one le for the relevant image ground truth.
In the cluster ground-truth le each line corresponds to a cluster where the rst value is the topic id, followed by cluster id number, followed by the cluster user tag separated by comma. Lines are separated by an end-of-line character (carriage return). An example is presented below: { 1, 1, Badger & Dodo Cafe { 1, 2, Costa co ee { { 2, 1, Airport Restaurant { 2, 2, Arnotts Department Store {
In the relevant ground-truth le the rst value on each line is the topic id, followed by a unique photo id, and then followed by the cluster id number (that corresponds to the values in the cluster ground-truth le) separated by comma. Each line corresponds to the ground truth of one image and lines are separated by an end-of-line character (carriage return). An example is presented below:
This year the number of participants was considerably higher with respect to 2017: we received in total 41 runs: 29 (21 o cial, 8 additional) for LMRT and 12 (8 o cial, 4 additional) for ADLT, from 7 teams from Brunei, Taiwan, Vietnam, Greece-Spain, Tunisia, Romania, and a multi-nation team from Ireland, Italy, Austria, and Norway. The received approaches range from fully automatic to fully manual, from using a single information source provided by the task to using all information as well as integrating additional resources, from traditional learning methods (e.g. SVMs) to deep learning and ad-hoc rules. Submitted runs and their results are summarized in Tables 2 and 3. 3.2
In this section we provide a short description of all submitted approaches followed by the o cial result of the task.
The organiser team participated in both tasks [
The CIE@UTB [
CAMPUS-UPB [
AILab-GTI [
The NLP-Lab [
HCMUS [
The Regim Lab [
As mentioned before, for the ADLT task, four teams have been participated:
CIE@UTB [
The o cial results are summarised in Table 2. The best run was submitted by CIE@UTB with a score of 0.556 which is also outperforming the organizers baseline approaches.
For LMRT, six teams have participated: AILab-GTI [
We learned that multi-modal data analysis has been explored and exploited this year, with the majority of the approaches combining visual, textual, location and other information to solve the task. This was quite di erent from last year when often only one type of data was analysed. Furthermore, we learned that many approaches are based on deep neural networks, from standard CNN to speci cally designed deep networks for lifelogging tasks. However, there are still rooms for improvement, since the best results are coming from the ne-tuned queries, which means we need more advanced techniques to bridging the gap between the abstract understanding of human needs and the multi-modal data. Furthermore, automatically \translate" the query into the retrieval criteria is still a challenge which requires further studies. Regarding the number of the signed-up teams and the submitted runs, we received a signi cant improvement compared to last year. This shows how interesting and challenging lifelog data is and that it holds much research potential. As next steps we do not plan to enrich the dataset but rather provide richer data and narrow down the application of the challenges (e.g., extend to health-care application). They are the activities/facets of daily life and the environmental settings / contexts in which these activities take place. For the ADLT task, we selected a subset of ten of these activities and settings for evaluation.
A.1
Activities / Facets of Life Activity
Settings/Context (the physical environment)
ADL: commuting Span: 2016-08-15 to 2016-08-19 Description: Find how many times and how long the user commuted from 15th Aug to 19th Aug
ADL: Eating/Drinking Span: 2016-08-29 to 2016-09-04 Description: Find how many times and how long the user was eating or drinking from 29th Aug to 4th Sep
ADL: Watching (TV) Span: 2016-09-26 to 2016-10-02 Description: Find how many times and how long the user was watching the TV from 26th Sept to 2nd Oct
ADL: Shopping Context: Grocery shops Span: 2016-09-19 to 2016-09-25, 16: 00 to 20: 00 Description: Find how many times and how long the user was shopping in the grocery shops from 19th Sept to 25 Sept
ADL: Socialising Context: Public place Span: Weekends Description: Find how many times and how long the user was socialising with his friends in the public place (co ee shop, bar, restaurant, etc.) on weekend days.
ADL: Using laptop or desktop computer Description: Find how many times and how long the user was using his laptop on weekend days.
T.007 In O ce ADL: Exclude meetings Context: In o ce environment Span: 2016-09-05 - 2016-09-09 Description: Find how many times and how long the user was working (exclude meetings) in o ce environment from 5th Sept to 9th Sept. No matter which o ce the user was working in.
Description: Find the moments when I was taking public transportation Narrative: Moments in which the user was was taking any public transportation.
Description: Find the moments when I was eating lunch from 11: 00am to 3: 00pm Narrative: Moments in which the user was eating lunch are relevant regardless of where the lunch is eaten. Time is relevant T.003 Co ee Description: Find the moment(s) when I was drinking co ee in a cafe. Narrative: Moments that show the user consuming co ee or tea in a cafe (outside of home or o ce) are considered relevant. The co ee can be hot in a cup or paper cup, or cold co ee in a plastic or paper cup.
Description: Find the moments when I was outside at sunset and sunrise. Narrative: To be considered relevant, the moment must show the sun setting or rising. This can be at night time and morning time, or can be when the sun is disappearing and appearing behind a mountain in the evening.
Description: Find the moments when I was lecturing to a group of people in a classroom environment.
Narrative: A lecture can be in any classroom environment and must contain more than one person in the audience, who are sitting down. A classroom environment has desks and chairs clearly visible. Discussion or lecture encounters in which the audience are standing up, or outside of a classroom environment are not considered relevant.
Description: Find all the moments when I was grocery shopping. Narrative: Any moment when the user was in a grocery store and visibly interacting with products is considered relevant.
Description: Find the moments when I was cooking at home.
Narrative: Cooking at home includes preparation of ingredients and cooking of the food. To be considered relevant the user must be seen to be preparing food. Eating food at home is not considered relevant.
Description: Find the moment when I had beer in a bar or in a restaurant. Narrative: To be considered relevant, the user must be clearly in a bar and having more than one drink. Black and light beers were consumed in this moment.
Description: Find the moments in which I was working in a co ee shop. Narrative: To be considered relevant the user must be seen working with a laptop in a co ee shop. Relevant moments must show co ee on the table beside the laptop. Working in any place besides a co ee shop is not considered relevant. Socialising or relaxing in a co ee shop is not considered relevant if there is no laptop being used.
Description: Find the moments when I was eating Pasta.
Narrative: The user was eating pasta, either sitting at a table, an o ce desk or in a corridor outside an o ce. Sometimes pasta eating occurred with another person, sometimes it was in solitude.
Description: Find the moments when I was preparing salad.
Narrative: To be considered relevant, the moments must show the lifelogger preparing a salad, in a kitchen or in an o ce environment. Eating salad is not considered relevant. Preparing other types of food is not considered relevant.
Description: Find the moments when I was doing Virtual Reality (VR) experiments or seeing someone else doing VR experiments.
Narrative: To be considered relevant, the moments must show a VR device in use by the lifelogger.
Description: Find the moments when I was giving a presentation to a large group of people.
Narrative: To be considered relevant, the moments must show more than 15 people in the audience. Such moments may be giving a public lecture or a lecture in the university.
Description: Find all the moments when I was interviewed by TV presenter. Narrative: The moment must show the cameras or cameramen in front of the lifelogger. The interviews can occur at the lifelogger's home or in the o ce environment.
Description: Find the moments when I was having dinner at home. Narrative: Moments in which the user was having dinner at home are relevant. Dinner in any other location is not relevant. Dinner usually occurs in the evening time
Description: Find the moments when I was assembling a piece of furniture. Narrative: To be considered relevant, the moments must show some parts of the furniture being assembled.
Description: Find the moments when I was taking a road vehicle in foreign countries.
Narrative: To be considered relevant, the user must be taking road transport in a di erent country (i.e. not Ireland). Taking airplane, train or boat is not considered relevant.
Description: Find the moments when I was with friends in Costa co ee. Narrative: To be considered relevant, the moment must show at least a person together with the lifelogger in any Costa Co ee shop. Moments that show the user alone are not considered relevant.
Description: Find the moments in which I was using my mobile phone or tablets in a vehicle.
Narrative: To be considered relevant the user must be seen using with a mobile phone or a tablet in a vehicle, as a driver or as a passenger.
Description: Find the moments when I was at a graveyard.
Narrative: The user must be in a graveyard or inside a church inside a graveyard. Passing or standing outside of the graveyard is not considered relevant.