This paper describes the participation of the Organizer Team in the ImageCLEFlifelog 2018 Daily Living Understanding and Lifelog Moment Retrieval. In this paper, we propose how to exploit LIFER, an interactive lifelog search engine to solve the two tasks: Lifelog Moment Retrieval and Activities of Daily Living Understanding. We propose approaches for both baseline, which aim to provide a reference system for other approaches, and human-in-the-loop, which advance the baseline results.
A new trend in multimedia research is the generation of personalized archives
that store rich detail of your life experience using various modalities such as
videos, images, text or sensor data. These logs are commonly refereed to as
lifelogs [
Lifelogging poses many challenging research questions [
In this paper we describe our solution to the 2018 Image CLEF [
Utilizing time and concepts comes with the advantage that boundaries
between events are easily identi able. This saves both processing time and
computation power [
The paper is organized as follows: Firstly, we provide an overview of related work in the eld. After that we give a detailed description of LIFER, the interactive system that our approach is based on, which is followed by a methodology of how to exploit the system. We then show the results obtained from the o cial competition and nally, we discuss the solutions, the results, and conclude the paper. 2
In general there exits an ever increasing body of research aimed at solving di
erent aspects of the overall lifelogging information access challenge, ranging from
computer vision [
In context of our approach, several related works exist. A common practice
for image segmentation based on time data is heuristic splitting [
As with other elds of research, information retrieval currently relies heavily
on deep learning approaches [
LIFER: An Interactive Lifelog Search Engine
Our proposed solutions for the two tasks is to exploit the baseline system,
LIFER [
LIFER uses the core search engine of [
LIFER is built based on the six sources of information which was extracted from ImageCLEF dataset o ered.
{ Time. The most basic unit of data in the dataset, time gave us the
possibility of including more semantic concepts, such as days of the week,
weekday/weekend, times of the day, etc. In the LIFER system, we consider the
unit of time as minute, i.e., each image is attached to a minute. These time
is extracted (and linked to the image) directly from the provided data.
{ Locations. Semantic location were provided in the dataset which provided
localised names for all locations visited. For example `The Helix', `Dunnes
stores', `Dublin City University' and so on.
{ Visual Concepts. visual concepts extracted by Microsoft API[
These included the Galvanic Skin Response (stressed/excited, relaxed) which can be considered to be a correlate of stress or excitement levels, and the level of physical activity (exertion / resting) as identi ed from the heart rate. { Music. A log of the music listing history of the lifelogger was included in the collection and we considered that it could be an important aspect of some topics. The song name and song artist are two options which are used to search results.
These six sources of information are instantiated in the user interface as facets of a user query, as shown in Figure 1.
The Interface of LIFER is shown in Figure 2. The upper section of the interface is the query-panel in which the faceted queries are created. Below that is the main part of the interface which is where the selected lifelog images are displayed in temporal sequence.
In the query-panel, the search facets are shown. The facets are directly related to the indexed data (see the six sources of information above). Upon submission of a faceted query, the system returns a temporally organised listing of potentially relevant images. In the rst version of LIFER, the query facets are combined
Visual Music Concept Activity Location Time Biometrics Raw Data
Feature Vectors Indexed Database
API/Interface Images
A set of criteria
User in an AND boolean manner. This can be changed on a per-topic basis, but does not form part of the interface at present.
fully automatic without ranking fully automatic with ranking
The temporally organised listing of relevant images is displayed in the lower part of the screen (the result-display panel). Each relevant image is listed with an overview metadata as a form of context. This metadata is con gurable to display various sources of information, as required. Figure 2 shows a basic form of such metadata. 4
Exploiting LIFER for ImageCLEFlifelog2018 tasks As mentioned, we exploited LIFER for ImageCLEFlifelog2018 LMRT and ADLT tasks. Firstly, based on the topic description, the search criteria are determined (by both automatically considering all the words in the queried topic as concepts, or alternatively allowing concepts to be `determined by the user). Secondly, we improved the interface of LIFER to allow user manually select multiple relevant images for the submission or taking all of them as relevant.
In term of ranking, we determine to use the default option of LIFER by chronological order.
In the next section, we present the o cial results on the test set exploiting the LIFER system. We submitted 10 runs (5 for each task) in total, summarized in Table 1 and 2. For ADLT, the best result is made by searching the relevant keywords and using human-in-loop to lter the irrelevant results. The remaining runs are created by using the relevant query terms and human ltering. For LMRT, the rst two runs are automatic while the rest three adopted the same approaches used in ADLT. Shown in Tables 3 and 4 are the search criteria for the best run of each task. 6
In this paper we introduce di erent baseline approaches, from fully automatic to fully manual approaches, by exploiting LIFER, an interactive lifelog retrieval
system to tackle the ImageCLEFlifelog 2018 task, as a participant of the Lifelog Moment Retrieval and Activities of Daily Living Understanding tasks. These approaches, that require di erent levels of involvement from the users, exploit only the information provided by the organizers along with the collection of images, e.g., the description of the semantic locations and the physical activities. With the human in the loop, we obtained the highest score for ADLT task (ADLT Run 1, *please notice that we are not ranked since we are the task organiser). However, without the manually input, the results can be close to random (as in the result of LMRT Run 1). This shows that the key challenge is how to translate the query to the search criteria, with requires further study.