Organizer Team at ImageCLEFlifelog 2017: Baseline Approaches for Lifelog Retrieval and Summarization

Abstract

This paper describes the participation of Organizer Team in the ImageCLEFlifelog 2017 Retrieval and Summarization subtasks. In this paper, we propose some baseline approaches, using only the provided information, which require di erent involvement levels from the users. With these baselines we target at providing references for other approaches that aim to solve the problems of lifelog retrieval and summarization.

-

Personalized multimedia archives that contain a large amount of data collected using various personal devices, such as smart phones, cameras, wearable devices and so on are getting more and more common nowadays. In these archives, every moment and aspect of our lives are stored. They can contain information about our daily routines, consumed food but also about our health status, etc. These data logs of a human lives, also commonly referred to as lifelogs, are more and more interesting for the research community but also companies. Collecting and storing the data is one challenge but getting insights from the collected data and nd new information by connecting di erent types of data requires a lot of researches for analyzing, categorizing and querying these huge amounts of data in a e cient way.

In this paper we present our approach to tackle the Image CLEF 2017 [ 11 ] Lifelog Task [ 6 ], which aims at solving the problems of lifelog retrieval and summarization. Lifelogs are usually chronologically organized and moments that belong to the same activity or the same event are normally very similar. This can be exploited to reduce processing time by grouping moments that are similar based on the time when they happened and the belonging concepts. This transforms the image retrieval challenge into a image segments retrieval challenge. This has the advantage that boundaries between moments or activities are automatically segmented based on time and concepts [ 7 ]. To remove non-relevant images ltering is recommended. In our case, we remove images that seem to be sparse on information (blurry, only big objects, etc.) Retrieved images then can be diversi ed into clusters which then can be further used for summarization, which can be done automatically or via relevance feedback by follow the methods described in [ 5 ].

The remainder of this paper is organized as follows, rst we present related work in the eld. This is followed by a detailed description of our approach. After that we present the experimental results which is followed by a discussion and conclusion. 2

Related Work

In this section we discuss brie y recent studies on lifelog segmentation and the retrieval problem in term of relevant and diversity. In addition, many novel techniques are proposed and evaluated, to accurately retrieve the similar events from lifelog dataset using contextual data.

Typically, chronological images segmentation is done by heuristic split based on long interval with no capture [14] or by thresholding the distances between the frames (or images) based on the content [ 3 ]. Doherty et al. in [ 8 ], to determine the similarity between adjacent block of images, proposed to use Hearst's Text Tiling Algorithm [ 9 ] on edge histogram, which is extracted by using Canny edge detection. For egocentric photo streams (from wearable cameras), a typical segmentation is based on unsupervised hierarchical agglomerative clustering to extract the key-frame summary [ 1 ].

Current works in multimedia retrieval have considered relevance and diversity as two core criteria. Relevance was commonly estimated based on textual information, e.g., from the photo tags, and many of current search engines are still mainly based on this information. Diversity is usually improved by applying clustering algorithms which rely on textual or/and visual properties [12]. Recently, in social image retrieval, some methods have exploited the participation of humans by collecting the feedbacks of the results to improve the diversi cation [ 2 ]. To reduce the number of images to be returned to the user, some papers in the past years proposed the use of image clustering techniques [15]. These approaches exploit the hierarchical indexing structure of the clusters to re ne the number of images to consider [13]. More recently di erent type of relevance feedback has been used to expand the query to improve both relevant and diversity as well as reduce the number of iterations [ 5 ]. 3

The proposed approaches

The proposed approaches follow the schema as illustrated in Figure 1. Since lifelogs are chronologically organized and moments in the same activity or the same Query Segmentation

Retrieval Based on: - Concepts - Location - Activity - Time - # People

Based on: - Time - Concepts event are normally very similar to each other, in order to reduce the processing time, we group similar moments together based on time and concepts. By applying this chronological-based segmentation, we turn the problem of images retrieval into image segments retrieval, in which the boundary between activities such as having breakfast, working in front of a computer, and so on [ 7 ], are automatically decided based on the time and concepts. Starting from a topic query, it is transformed into small inquiries, where each of them is asking for a single piece of information of concepts, location, activity, and time. The moments that matched all of those requirements are returned as the retrieval results. In order to remove the non-relevant images, a ltering step is applied on the retrieved images, by removing blurred and images that covered mainly by huge object or by the arms of the user. Finally, the images are diversi ed into clusters and the top images that close to center are selected for the summarization, which can be done automatically or using relevance feedback by follow the methods in [ 5 ]. These steps are described as follows: For the segmentation we applied a simple chronological-based segmentation as follow: For each pair of two continuous images It and It+1 at the time t, the distance d(It; It+1) between them is computed as: d(It; It+1) = jjCt

Ct+1jj where jj jj is the normalized Euclidean distance, and C is the concept vector of each image provided from the task. If d(It; It+1) < , where is a threshold, the two images are set belong to the same segment, otherwise they are set in di erent segments. If is too small, an activity should be split into small activities, while larger value of tau should grouped di erent activities into the same one. Since jj jj is normalized, when = 0, the images are grouped into di erent segments, and when = 1, all images are belong to a single segment.

Segmenting the activities is not simply an incident of identifying the exact event boundaries; it also concerns with keeping track of the ne-grained group of events together into extended meaningful units, and thus deciding the right value of is not trivial. In the proposed approaches, we try di erent values of tau for di erent runs, which will be explained in Section 4.

After this step, each segment is represented by the rst image (of that segment) with these basic information: location, activity, time segment, number of people and the list of the concepts. If any of these information is missing from the rst image, we take it from the second image and so on. 3.2

Parsing the Query and Retrieval

Converting a topic into precise criteria for retrieval is the key question for both sub-tasks. It can be automatically done by considering any word in the topic as the queried concepts and then searching for all segments that contain those concepts; by applying natural language processing techniques; or by ne-tunning by a human in the loop, i.e., the user will read the topic and \translate" it into the search criteria. For example, with the topic: { Topic: Using laptop out of office { Query: Find the moment(s) in which user u1 was using his laptop outside the working places. { Description: To be consider to relevant, the user should use his laptop, for work or for entertainment out of his working place.

Can be \translated into": { User: +u1 { Concepts: +laptop { Activities: +working { Time: -{ Location: -work

where +/- means the retrieval images has to contain/not contain this information, respectively, and -- means any.

In the proposed approaches, we use the automatic and the human-in-the-loop methods. Our \translation" will be shown in Section 4. 3.3

Filtering

An image can be considered as blurred based on it focus level. In the proposed approaches, we estimate the focus by computing the absolute sum of the wavelet coe cients and comparing it to a threshold, by exploiting the method in [ 10 ]. The return of this method is a scalar number in [0; 99] which the bigger value the sharper image. From our observation, for values below 30, most of the images are blurred, and thus we set this threshold to 30.

In order to remove images that covered by large objects, we apply an heuristic method as follows: Step 1 Convert the image to binary images by applying thresholding with several thresholds.

Step 2 Extract connected components and calculate their centers.

Step 3 Group centers based on their coordinates, and then close them to form the corresponds blob.

Step 4 Take the biggest blob and its size (in pixels).

If the size is over 50% of the whole area, the image is considered as covered. This whole method is implemented by calling the function SimpleBlobDetector from OpenCV5.

After this step, all remain images are considered as relevant to the topic. Please notice that the images are still kept inside the segment. 3.4

Diversi cation

In this step, for automatic approach, we use a hierarchical agglomerative clustering algorithm (see in [ 4 ]) to group similar segments into the same cluster based on the concepts. The clusters are then sorted based on the number of segments, decreasingly. Finally, we produce the summary for the queried by selecting representative images from the clusters based by selecting the images closest to the center of each cluster.

We also propose a human-in-the-loop approach in this step by using the usual dichotomous Relevance Feedback paradigm (more details can be seen in [ 5 ]), that asks the user to assign the labels Relevant n Non-relevant to the retrieved images. The system asks the user to label the representative images of the top N results returned by the automatic diversi cation procedure (as mentioned above), and the number of images that have been labeled as being Relevant n Non-relevant for each cluster is computed. Then, the clusters are sorted as follows: { Clusters that have a large number of relevant counts are sorted higher. { Clusters that have the same number of relevant counts are sorted based on the number of non-relevant counts (i.e., a cluster that contains a larger number of `non-relevant' images should be selected later). { Clusters that have the same number of Relevant n Non-relevant counts are sorted on the basis of the number of segments.

For each cluster, the images that are selected to represent the topic are chosen in the same way as in the automatic diversi cation. 4 4.1

Experimental results

Submitted Runs

We submitted 3 runs on the Retrieval sub-task and 5 runs on the Summarization sub-task, summarized in Table 1.

As for the retrieval task, the rst run is exploiting only time and the concepts information. We consider every single image as the basic unit and the retrieval just returns all images that contains the concepts extracted from the topics. We named this run is the `baseline' with the purpose that any other approaches should obtain better performance than this. T008 T009 T010 T011 T012 T013 T014 T015 T016 T017 T018 -Walking, -Running, -Transport +Transport, -Walking, -Running -Walking, -Running, -Transport -Transport, +Running, +Walking -Waking, -Running, -Transport -Waking, -Running, -Transport -Waking, -Running, -Transport -Waking, -Running, -Transport +Walking, -Running, -Transport -Walking, -Running, -Transport -Walking, -Running, -Transport +Walking, -Running, -Transport +MinuteID:

400-1400 +MinuteID: -Home, -Work, -Television, 400-540, 660-840, +Science Gallery -laptop, -Commic 1080-1190 Caf, +DCU book, -Notebook

Restaurant, -Work, -Home +Laptop

+Guitar +MinuteID: 590- -Work, +Home 1400(workday),

5401240(weekend) +MinuteID: 960-1190 +DCU, -Home, +Running shoes

-Work +MinuteID: 540 +Work, +Home Apple, +Banana, -1240 + Orange, +

Strawberry +Home, -Work +Washbasin +MinuteID: 400-540, 1290-1400 +MinuteID:

400-1400 +MinuteID: 400-540, 660-840, 1080-1190 +MinuteID:

540-1080 +MinuteID:

540-1138 +MinuteID:

400-1400 +MinuteID: 650-1080 T020 T002 T003 T004 T005 T006 T007 u3 u1 T009 T010 u3 u3 +Transport, -Walking, -Running -Transport, -Walking, -Running -Transport, +Walking, -Running +Home, -Work

The same strategy is applied on the summarization subtask, in which the rst three runs were ran to test the automatic approach with the increasing level of the `criteria', while the last two runs are used to test the ne tuning and the relevance feedback approaches. For the relevance feedback approach, we ran a simulation by exploiting the ground-truth annotated data. Shown in Tables 4 and 5 are the results of the runs on the retrieval and summarization sub-tasks, respectively. The results con rm that applying segmentation improved both retrieval and summarization performance. It is quite clear that applying ne-tunning signi cantly improved the performance. The big gaps in results between the automatic approach with the ne-tunning and between the ne-tunning with the human-in-theloop (relevance feedback) approaches, shown that we need better natural language processing as well as machine learning studies for these problems. In this paper we introduced di erent baseline approaches, that came from fully automatic to fully manual paradigm, proposed by the Organizer Team of the ImageCLEFlifelog 2017 task as participant of the Retrieval and Summarization subtasks. These approaches, that require di erent level of involvement of the users, exploit only the information provided by the organizers along with the collection of images, i.e., the description of the semantic locations and the physical activities. From the obtained results it appears clear that deeper analysis of the methods should be considered as well as the use of extra information. LST Run 1 Baseline LST Run 2 Segmentation LST Run 3 Filtering LST Run 4 Fine Tuning LST Run 5 Relevance Feedback 12. van Leuken, R.H., Garcia, L., Olivares, X., van Zwol, R.: Visual diversi cation of image search results. In: Proceedings of the 18th International Conference on World Wide Web. pp. 341{350. WWW '09, ACM, New York, NY, USA (2009) 13. Mironica, I., Ionescu, B., Vertan, C.: Hierarchical clustering relevance feedback for content-based image retrieval. In: IEEE International Workshop on Content-Based Multimedia Indexing. pp. 1{6 (2012) 14. Peitgen, H., Jurgens, H., Saupe, D.: Chaos and fractals - new frontiers of science (2. ed.). Springer (2004) 15. Thomee, B., Lew, M.S.: Interactive search in image retrieval: a survey. International Journal of Multimedia Information Retrieval 1(1), 71{86 (2012)

References

1. Bolanos , M. , Mestre , R. , Talavera , E. , Nieto , X.G. , Radeva , P. : Senseseer mobilecloud-based lifelogging framework. Visual summary of egocentric photostreams by representative keyframes pp. 1 { 6 ( July 2015 )

2. Boteanu , B. , Mironic , I. , Ionescu , B. : A relevance feedback perspective to image search result diversi cation . In: 2014 IEEE 10th International Conference on Intelligent Computer Communication and Processing (ICCP) . pp. 47 { 54 (Sept 2014 )

3. Byrne , D. , Lavelle , B. , Doherty , A.R. , Jones , G.J. , Smeaton , A.F. : Using bluetooth and gps metadata to measure event similarity in sensecam images . 5th International Conference on Intelligent Multimedia and Ambient Intelligence (July 2007 )

4. Dang-Nguyen , D.T. , Piras , L. , Giacinto , G. , Boato , G. , De Natale , F.G. : A hybrid approach for retrieving diverse social images of landmarks . In: 2015 IEEE International Conference on Multimedia and Expo (ICME) . pp. 1 { 6 ( 2015 )

5. Dang-Nguyen , D.T. , Piras , L. , Giacinto , G. , Boato , G. , De Natale , F.G. : Multimodal retrieval with diversi cation and relevance feedback for tourist attraction images . ACM Transactions on Multimedia Computing , Communications, and Applications ( 2017 ), accepted

6. Dang-Nguyen , D.T. , Piras , L. , Riegler , M. , Boato , G. , Zhou , L. , Gurrin , C. : Overview of ImageCLEFlifelog 2017: Lifelog Retrieval and Summarization . In: CLEF 2017 Labs Working Notes. CEUR Workshop Proceedings , CEUR-WS.org <http://ceur-ws. org> , Dublin, Ireland (September 11 -14 2017 )

7. Doherty , A.R. , Smeaton , A.F. : Automatically segmenting lifelog data into events . 9th International Workshop on Image Analysis for Multimedia Interactive Services (30 June 2008 ), http://doras.dcu.ie/4651/

8. Doherty , A.R. , Smeaton , A.F. , Lee , K. , Ellis , D.P. : Multimodal segmentation of lifelog data. Large Scale Semantic Access to Content (Text, Image, Video, and Sound) pp. 21 { 38 ( June 2007 ), http://dl.acm.org/citation.cfm?id= 1931393

9. Hearst , M.A. : Texttiling: A quantitative approach to discourse segmentation . Technical Report UCB:S2K-93-24 ( 1993 )

10. Huang , J.T. , Shen , C.H. , Phoong , S.M. , Chen , H.: Robust measure of image focus in the wavelet domain . In: Intelligent Signal Processing and Communication Systems . pp. 157 { 160 ( 2005 )

11. Ionescu , B. , Muller, H., Villegas , M. , Arenas , H. , Boato , G. , Dang-Nguyen , D.T. , Dicente Cid , Y. , Eickho , C. , Garcia Seco de Herrera , A. , Gurrin , C. , Islam , B. , Kovalev , V. , Liauchuk , V. , Mothe , J. , Piras , L. , Riegler , M. , Schwall , I. : Overview of ImageCLEF 2017: Information extraction from images . In: Experimental IR Meets Multilinguality, Multimodality, and Interaction 8th International Conference of the CLEF Association, CLEF 2017. Lecture Notes in Computer Science , vol. 10456 . Springer, Dublin, Ireland (September 11 -14 2017 )