This article describes the RobotVision@ImageCLEF 2012 challenge, which addresses the problem of multimodal place classi cation. Participants of the challenge were asked to classify rooms on the basis of image sequences captured by cameras mounted on a mobile robot. The proposals of the participants had to answer the question \where are you?" (I am in the elevator, in the toilet, etc) when presented with a test sequence, acquired within the same building and oor but with di erent lighting conditions than the training sequence. The 2012 edition of the challenge introduced the use of depth images in addition to visual images. Moreover, several techniques for feature extraction and cue integration were also proposed. As in previous editions, two di erent tasks were proposed: task 1 and task 2. In task 1 (mandatory) participants were asked to classify the frames separately, while the temporal continuity of the image sequence could only be exploited in task 2 (optional). Eight di erent groups participated to the 2012 edition of the Robot Vision challenge. The winner in both tasks was the Centro de Investigacion en Informatica para la Ingenier a (CIII), from the Universidad Tecnologica Nacional, Argentina (CIII UTN FRC). This participant obtained an overall score of 2071 (84.70% of the maximum score) in task 1 and 3930 (96.35% of the maximum score) in task 2.
The ImageCLEF 2012 Robot Vision challenge has been the fourth edition of a
competition [
The 2009@ImageCLEF edition of the task [
In 2010, two editions of the challenge took place. The second edition of the
task, 2010@ICPR [
Several changes have been proposed for the ImageCLEF 2012 Robot Vision task. Firstly, stereo images have been replaced by images acquired using two types of camera: a perspective camera for visual images and a depth camera (the Microsoft Kinect sensor) for range images. Therefore, each frame consists of two types of images and the challenge is focused on the problem of multimodal place classi cation. In addition to the use of depth images, the optional task contains kidnappings and no unknown rooms appear in the test sequences. Moreover, several techniques for features extraction and cue integration have been proposed to the participants.
We received a total of 23 runs from 8 di erent groups. 18 runs were submitted to the task 1 (mandatory) and 5 to the task 2 (optional). The best result in both tasks was obtained by the Centro de Investigacion en Informatica para la Ingenier a (CIII), from the Universidad Tecnologica Nacional, Argentina (CIII UTN FRC).
The rest of the paper details the challenge and is organized as follows: Section 2 describes the 2012 ImageCLEF edition of the RobotVision task. Section 3 presents all the participants groups, while the results are reported in Section 4. Finally, in Section 5, conclusions are drawn and future work is outlined.
This section describes the details concerning the setup of the ImageCLEF 2012 Robot Vision task. Section 2 gives a description of the task. Section 2.1 describes all the sequences of frames provided for training and test while the two subtasks are explained in Section 2.2. The performance evaluation is detailed in Section 2.3 and nally, Section 2.4 describes the information provided by the organizers. 2.1
The fourth edition of the Robot Vision challenge was focused on the problem of multi-modal place classi cation. Participants were asked to classify functional areas on the basis of image sequences, captured by a perspective camera and a Kinect mounted on a mobile robot (see Fig. 1) within an o ce environment.
Participants had available visual images and range images that could be used to generate 3D point cloud les. The di erence between visual images, range images and 3D point cloud les can be observed in Figure 2. Training and test sequences were acquired within the same building and oor but with some variations in the lighting conditions or the acquisition procedure (clockwise and counter clockwise).
Two di erent tasks were considered in the Robot Vision challenge: task 1 and task 2. For both tasks, participants should be able to answer the question \where are you?" when presented with a test sequence imaging a room category already seen during training. The di erence between both tasks was the presence
Visual Image
Range Image
3D point cloud le (or lack) of kidnappings in the nal test sequence, and the availability on the use of the temporal continuity of the sequence.
The kidnapping (only task 2) is a ected by the robot changing room. Room changes in sequences without kidnappings were usually represented by a small number of images showing a smooth transition. On the other side, room changes with kidnappings were represented by a drastic change for frames, as can be observed in Figure 3. 2.2
Training and validation sequences consisted of a set of the Robot Vision VIDA dataset. VIDA is a dataset with images acquired within an indoor environment using a robot platform in the IDIAP research building. This dataset contains sequences of several rooms belonging to di erent room categories such as \Corridor" or \Toilet". Sequences were acquired using two cameras: a perspective visual camera and a 3D range laser sensor Kinect camera. Therefore, there are two di erent types of images: RGB images and depth images.
Three di erent sequences of frames were provided for training and two additional ones for the nal experiment. All training frames were labelled with the name of the room they were acquired from. There were 9 di erent categories of rooms, and the distribution for the training sequences can be observed in Table 1
The di erence between all the room categories can be observed in Figure 4, where an exemplar visual image for each one of the 9 room categories is shown.
Elevator Area
Corridor
Toilet Lounge Area
Technical Room
Professor O ce Student O ce
Visio Conference
Printer Room
All the participants of the ImageCLEF 2012 Robot Vision task were allowed to submit their runs to two di erent subtasks: task1 and task2.
Task 1 This task was mandatory and the test sequence had to be classi ed without using the temporal continuity of the sequence. Therefore, the order of the test frames cannot be taken into account. Moreover, there were not kidnappings in the nal test sequence.
Task 2 This task was optional and participants could take advantage of the temporal continuity of the test sequence. There were kidnapping in the nal test sequences that allowed participants to obtain additional points when they were managed correctly 2.4
The proposals of the participants were compared using the score obtained by their submissions. These submissions were the classes or room categories assigned to the frames of the test sequences, and the score was computed using the rules that are shown in Table 2. Due to wrong classi cations obtaining negative points, participants were allowed to not classify test frames.
Each correctly classi ed frame +1 points Each misclassi ed frame -1 points Each frame that was not classi ed +0 points (Task 2) All the 4 frames correctly classi ed after a kidnapping +1 points (additional) 2.5
We proposed the use of several techniques for features extraction (PHOG and
NARF) and cue integration (OBSCURE). Thanks to the use of these techniques,
participants could focus on the development of new features while using the
proposed method for cue integration or vice versa. We also provided information
as the point cloud library [
Depth Features NARF features is a novel descriptor technique that has been
included in the point cloud library [
In 2012, 43 groups registered to the Robot Vision task but only 8 submitted, at least, one run, namely: { CIII UTN FRC: Universidad Tecnologica Nacional, Cordoba, Argentina. { NUDT: National University of Defense Technology, Changsha, China. { UAIC2012: Alexandru Ioan Cuza University, Iasi, Romania. { USUroom409: Ural Federal University, Yekaterinburg, Russian Federation. { SKB Kontur Labs: Kontur Labs, Yekaterinburg, Russian Federation. { CBIRITU: Istanbul Technical University, Istanbul, Turkey. { SIMD: University of Castilla-La Mancha, Albacete, Spain. { Bu aloVision: University at Bu alo, New York, United States.
A total of 23 runs were submitted, with 18 runs submitted to the task 1 (mandatory) and 5 runs submitted to the task 2 (optional). The limit to the number of runs that could be submitted was 3. 4
This section presents the results of the Robot Vision task of ImageCLEF 2012 for the two subtasks: task1 and task 2. Eight di erent groups submitted runs for the task 1 as can be observed in Table 3. The maximum score that could be achieved was 2445 and the winner (CII UTN FRC) obtained 2071 points. CII UTN FRC and NUDT teams ranked rst and second respectively and their score was higher than 70% of the maximum score.
The score obtained by the SIMD/IDIAP team could be considered as a baseline result that all the groups were expected to improve. Such score was obtained using the techniques provided by the organizers without new contributions. As it was expected, 6 out of 7 teams obtained higher scores. The results are summarized in Figure 5, where only the best run for each team has been considered. For the optional task, the maximum score was 4079 and only 4 groups submitted runs. The winner for the task 2 was the CIII UTN FRC group with 3930 points, only 71 more than the NUDT group, which ranked second. All the results can be seen in Table 4.
In view of these results, it should be remarked the high quality of the participant proposals, due to the score obtained by CIII UTN FRC, NUDT and CBIRITU groups was higher than 75% of the maximum score. All the results for task 2 are summarized in Figure 6, where only the best run for each team has been considered. 5
We have described in this article the fourth edition of the Robot Vision task at ImageCLEF 2012, which attracted a considerable attention with 8 groups submitting runs. There are 2 main conclusions that can be drawn from the proposals: (i) despite of two types of images were provided (visual and range), depth features were not commonly used, and (ii) most of the proposals were based on the use of SVMs. We plan to continue the task in the next years with new challenges related to place categorization. Concretely, we have plans to introduce object categorization in the following editions.