This paper describes the participation of Organizer Team in the ImageCLEFlifelog 2017 Retrieval and Summarization subtasks. In this paper, we propose some baseline approaches, using only the provided information, which require different involvement levels from the users. With these baselines we target at providing references for other approaches that aim to solve the problems of lifelog retrieval and summarization.
Personalized multimedia archives that contain a large amount of data collected using various personal devices, such as smart phones, cameras, wearable devices and so on are getting more and more common nowadays. In these archives, every moment and aspect of our lives are stored. They can contain information about our daily routines, consumed food but also about our health status, etc. These data logs of a human lives, also commonly referred to as lifelogs, are more and more interesting for the research community but also companies. Collecting and storing the data is one challenge but getting insights from the collected data and find new information by connecting different types of data requires a lot of researches for analyzing, categorizing and querying these huge amounts of data in a efficient way.
In this paper we present our approach to tackle the Image CLEF 2017 [11] Lifelog Task [6], which aims at solving the problems of lifelog retrieval and summarization. Lifelogs are usually chronologically organized and moments that belong to the same activity or the same event are normally very similar. This can be exploited to reduce processing time by grouping moments that are similar based on the time when they happened and the belonging concepts. This transforms the image retrieval challenge into a image segments retrieval challenge. This has the advantage that boundaries between moments or activities are automatically segmented based on time and concepts [7]. To remove non-relevant images filtering is recommended. In our case, we remove images that seem to be sparse on information (blurry, only big objects, etc.) Retrieved images then can be diversified into clusters which then can be further used for summarization, which can be done automatically or via relevance feedback by follow the methods described in [5].
The remainder of this paper is organized as follows, first we present related work in the field. This is followed by a detailed description of our approach. After that we present the experimental results which is followed by a discussion and conclusion.
In this section we discuss briefly recent studies on lifelog segmentation and the retrieval problem in term of relevant and diversity. In addition, many novel techniques are proposed and evaluated, to accurately retrieve the similar events from lifelog dataset using contextual data.
Typically, chronological images segmentation is done by heuristic split based on long interval with no capture [14] or by thresholding the distances between the frames (or images) based on the content [3]. Doherty et al. in [8], to determine the similarity between adjacent block of images, proposed to use Hearst's Text Tiling Algorithm [9] on edge histogram, which is extracted by using Canny edge detection. For egocentric photo streams (from wearable cameras), a typical segmentation is based on unsupervised hierarchical agglomerative clustering to extract the key-frame summary [1].
Current works in multimedia retrieval have considered relevance and diversity as two core criteria. Relevance was commonly estimated based on textual information, e.g., from the photo tags, and many of current search engines are still mainly based on this information. Diversity is usually improved by applying clustering algorithms which rely on textual or/and visual properties [12]. Recently, in social image retrieval, some methods have exploited the participation of humans by collecting the feedbacks of the results to improve the diversification [2]. To reduce the number of images to be returned to the user, some papers in the past years proposed the use of image clustering techniques [15]. These approaches exploit the hierarchical indexing structure of the clusters to refine the number of images to consider [13]. More recently different type of relevance feedback has been used to expand the query to improve both relevant and diversity as well as reduce the number of iterations [5].
The proposed approaches follow the schema as illustrated in Figure 1. Since lifelogs are chronologically organized and moments in the same activity or the same -NLP -Manual
Fig. 1. Schema of the proposed methods.
event are normally very similar to each other, in order to reduce the processing time, we group similar moments together based on time and concepts. By applying this chronological-based segmentation, we turn the problem of images retrieval into image segments retrieval, in which the boundary between activities such as having breakfast, working in front of a computer, and so on [7], are automatically decided based on the time and concepts. Starting from a topic query, it is transformed into small inquiries, where each of them is asking for a single piece of information of concepts, location, activity, and time. The moments that matched all of those requirements are returned as the retrieval results. In order to remove the non-relevant images, a filtering step is applied on the retrieved images, by removing blurred and images that covered mainly by huge object or by the arms of the user. Finally, the images are diversified into clusters and the top images that close to center are selected for the summarization, which can be done automatically or using relevance feedback by follow the methods in [5]. These steps are described as follows:
For the segmentation we applied a simple chronological-based segmentation as follow: For each pair of two continuous images I t and I t+1 at the time t, the distance d(I t , I t+1 ) between them is computed as:
where || • || is the normalized Euclidean distance, and C is the concept vector of each image provided from the task. If d(I t , I t+1 ) < τ , where τ is a threshold, the two images are set belong to the same segment, otherwise they are set in different segments. If τ is too small, an activity should be split into small activities, while larger value of tau should grouped different activities into the same one. Since || • || is normalized, when τ = 0, the images are grouped into different segments, and when τ = 1, all images are belong to a single segment.
Segmenting the activities is not simply an incident of identifying the exact event boundaries; it also concerns with keeping track of the fine-grained group of events together into extended meaningful units, and thus deciding the right value of τ is not trivial. In the proposed approaches, we try different values of tau for different runs, which will be explained in Section 4.
After this step, each segment is represented by the first image (of that segment) with these basic information: location, activity, time segment, number of people and the list of the concepts. If any of these information is missing from the first image, we take it from the second image and so on.
Converting a topic into precise criteria for retrieval is the key question for both sub-tasks. It can be automatically done by considering any word in the topic as the queried concepts and then searching for all segments that contain those concepts; by applying natural language processing techniques; or by fine-tunning by a human in the loop, i.e., the user will read the topic and "translate" it into the search criteria. For example, with the topic:
-Topic: Using laptop out of office -Query: Find the moment(s) in which user u1 was using his laptop outside the working places. -Description: To be consider to relevant, the user should use his laptop, for work or for entertainment out of his working place.
Can be "translated into":
-User: +u1 -Concepts: +laptop -Activities: +working -Time: ---Location: -work
where +/-means the retrieval images has to contain/not contain this information, respectively, and --means any.
In the proposed approaches, we use the automatic and the human-in-the-loop methods. Our "translation" will be shown in Section 4.
An image can be considered as blurred based on it focus level. In the proposed approaches, we estimate the focus by computing the absolute sum of the wavelet coefficients and comparing it to a threshold, by exploiting the method in [10]. The return of this method is a scalar number in [0, 99] which the bigger value the sharper image. From our observation, for values below 30, most of the images are blurred, and thus we set this threshold to 30.
In order to remove images that covered by large objects, we apply an heuristic method as follows:
Step 1 Convert the image to binary images by applying thresholding with several thresholds.
Step 2 Extract connected components and calculate their centers.
Step 3 Group centers based on their coordinates, and then close them to form the corresponds blob.
Step 4 Take the biggest blob and its size (in pixels).
If the size is over 50% of the whole area, the image is considered as covered. This whole method is implemented by calling the function SimpleBlobDetector from OpenCV 5 .
After this step, all remain images are considered as relevant to the topic. Please notice that the images still kept inside the segment.
In this step, for automatic approach, we use a hierarchical agglomerative clustering algorithm (see in [4]) to group similar segments into the same cluster based on the concepts. The clusters are then sorted based on the number of segments, decreasingly. Finally, we produce the summary for the queried by selecting representative images from the clusters based by selecting the images closest to the center of each cluster.
We also propose a human-in-the-loop approach in this step by using the usual dichotomous Relevance Feedback paradigm (more details can be seen in [5]), that asks the user to assign the labels Relevant \ Non-relevant to the retrieved images. The system asks the user to label the representative images of the top N results returned by the automatic diversification procedure (as mentioned above), and the number of images that have been labeled as being Relevant \ Non-relevant for each cluster is computed. Then, the clusters are sorted as follows:
-Clusters that have a large number of relevant counts are sorted higher.
-Clusters that have the same number of relevant counts are sorted based on the number of non-relevant counts (i.e., a cluster that contains a larger number of 'non-relevant' images should be selected later). -Clusters that have the same number of Relevant \ Non-relevant counts are sorted on the basis of the number of segments.
For each cluster, the images that are selected to represent the topic are chosen in the same way as in the automatic diversification.
4 Experimental results
We submitted 3 runs on the Retrieval sub-task and 5 runs on the Summarization sub-task, summarized in Table 1.
As for the retrieval task, the first run is exploiting only time and the concepts information. We consider every single image as the basic unit and the retrieval just returns all images that contains the concepts extracted from the topics. We named this run is the 'baseline' with the purpose that any other approaches should obtain better performance than this. With the second run, we applied the optimized value for τ (optimized from the devset) to do the segmentation. So in this run, the only difference is the basic unit of retrieval now is the segment, not image.
For the Fine-tunning runs, the "translation" is applied as in Tables2, and 3. The same strategy is applied on the summarization subtask, in which the first three runs were ran to test the automatic approach with the increasing level of the 'criteria', while the last two runs are used to test the fine tuning and the relevance feedback approaches. For the relevance feedback approach, we ran a simulation by exploiting the ground-truth annotated data.
Shown in Tables 4 and 5 are the results of the runs on the retrieval and summarization sub-tasks, respectively. The results confirm that applying segmentation improved both retrieval and summarization performance. It is quite clear that applying fine-tunning significantly improved the performance. The big gaps in results between the automatic approach with the fine-tunning and between the fine-tunning with the human-in-theloop (relevance feedback) approaches, shown that we need better natural language processing as well as machine learning studies for these problems.
In this paper we introduced different baseline approaches, that came from fully automatic to fully manual paradigm, proposed by the Organizer Team of the ImageCLE-Flifelog 2017 task as participant of the Retrieval and Summarization subtasks. These approaches, that require different level of involvement of the users, exploit only the information provided by the organizers along with the collection of images, i.e., the description of the semantic locations and the physical activities. From the obtained results it appears clear that deeper analysis of the methods should be considered as well as the use of extra information.