This article describes the RobotVision@ImageCLEF 2014 challenge, which addresses two problems: place classi cation and object recognition. Participants of the challenge were asked to classify rooms on the basis of visual and depth images captured by a Kinect sensor mounted on a mobile robot. They were also asked to detect the appearance or lack of several objects. The proposals of the participants had to answer two questions: \where are you?" (I am in the corridor, in the kitchen, etc.), and \list the objects that you can see?", from a prede ned list (I can see a table and a chair but not a computer) when presented with a test frame (a visual and a depth image). The number of times a speci c object appears in a frame is not relevant. Two di erent sequences of frames were provided for training and validation purposes, respectively. The nal test sequence included images acquired in a similar but di erent indoor o ce environment, which is considered the main novelty of the 2014 edition of the task. In contrast to previous editions of the task, sequences do not represent the temporal continuity in the acquisition procedure and therefore, test frames have to be processed sparsely. The winner of the 2014 edition of the Robot Vision task was the NUDT group, from China.
This paper describes the ImageCLEF 2014 Robot Vision challenge [
The 2009@ImageCLEF edition of the task [
In 2010, two editions of the challenge took place. The second edition of the
task, 2010@ICPR [
In 2012 [
In 2013 [
For the ImageCLEF 2014 Robot Vision challenge, we have introduced two main changes. Firstly, the temporal continuity from the image acquisition has been completely removed in the training, validation and test sequences. That is, consecutive frames in the provided sequences do not represent consecutive frames during the acquisition procedure. The second change is the inclusion of validation and test frames acquired in a di erent environment. Namely, we acquired new frames in a di erent building that contains the same type of rooms and objects imaged in the training and part of the validation sequence. Regarding the participation, in this edition, we received a total of 17 runs, from 4 di erent participant groups. The best result was obtained by the NUDT research group from the National University of Defense Technology, Changsha, China.
The rest of the paper details the challenge and is organized as follows: Section 2 describes the 2014 ImageCLEF edition of the RobotVision task. Section 3 presents all the participants groups, while the results are reported in Section 4. Finally, in Section 5, the main conclusions are drawn and some ideas for future editions are outlined. 2
This section describes the details concerning the setup of the ImageCLEF 2014 Robot Vision task. In Section 2.1 a description of training, validation and test sequences is provided. In Section 2.2 the performance evaluation criteria is detailed. Finally, in Section 2.3 a brief description of the baseline visual place classi cation system provided by the organizers, as well as other relevant details concerning the task are presented. 2.1
The sixth edition of the Robot Vision task is focused on the use of multimodal information (visual and depth images) with application to semantic localization and object recognition. The main objective of this edition is to address the problem of robot localization in parallel to object recognition from a semantic point of view, with a special focus on generalization. Both problems are inherently related: the objects present in a scene can help to determine the room category and vice versa. Solutions presented should be as general as possible while speci c proposals are not desired. In addition to the use of visual data, a 3D point cloud representation of the scene acquired from a Microsoft Kinect device was used, which has shown as a de facto standard in the use of depth images. In this new edition of the task, we introduced strong variations between training and test scenarios with the aim to solve the object recognition and localization problems in parallel and for a great variety of di erent scenarios.
Participants were given visual images and depth images in Point Cloud Data (PCD) format. Fig. 2 shows the same scene represented in a visual image and a point cloud data le. Training, validation and test sequence were acquired within two di erent buildings presenting a similar structure but with some variations in the objects distribution. All the room and object categories included in the test sequence were previously seen during training and validation.
As for the 2013 edition of the challenge, no sub-tasks were de ned and all participants have to prepare their submissions using the same test sequence.
In the 2014 edition of the RobotVision challenge, a new version of the unreleased Robot Vision dataset was created. This speci c dataset consists of three sequences (training, validation and test) of depth and visual images acquired within the following indoor environment: two department buildings at the University of Alicante, in Spain. Visual images were stored in PNG format and depth ones in PCD. Every image in the dataset was manually labelled with its corresponding room category/class and with a list of eight di erent objects to appear or not within it. The 10 di erent room categories are: Corridor, Hall, ProfessorO ce, StudentO ce, TechnicalRoom, Toilet, Secretary, VisioConference, Warehouse and ElevatorArea. The 8 di erent objects are: Extinguisher, Phone, Chair, Printer, Urinal, Bookself, Trash and Fridge. The dataset used in the task includes two labelled sequences used for training and validation with 5000 and 1500 images respectively. The unlabeled sequence used for test consists of 3000 di erent images. The frequency distribution for room categories in the training, validation and test sequences are depicted in Table 1. Regarding the building used in the acquisition, all the 5000 training images were acquired in the building A, the same used for the 2013 edition dataset. The validation sequence included 1000 images from building A but 500 new images from building B. Finally, all 3000 test images were acquired in building B.
It can be observed that in all sequences, Corridor is the class with higher number of frames. This is because most of the space of the University of Alicante building, suitable for robot navigation, belongs to several corridors. This situation makes it easier the classi cation of test frames as Corridor while other classes as VisioConference or ElevatorArea are more challenging. The frequency distribution for rooms in the di erent sequences is graphically presented in Fig. 3.
The distribution for object categories in the training, validation and test sequences is depicted in Table 2, while frequencies are presented in Fig. 4. Despite of small variations, it can be observed how classes and objects frequencies are maintained along training, validation and testing sequences.
The di erences between all the room categories can be observed in Fig. 5, where a single visual image for each of the 10 room categories is shown. Moreover,
Training 4% 6%
17% 27% 10% 3% 17% 14%
2% 7% 28%
19% 8% 2% 19% 16%
Test 4% 6%
17% 27% 10% 3% 17% Extinguisher
Chair
Printer
Bookshelf
Urinal
Trash
Phone The runs submitted for each participant were compared and sorted according the score assigned to each submission. Every submission consisted of the room category assigned to each test image and the corresponding list of the 8 detected/nondetected objects within that image. As we already mentioned above, the number of times a speci c object appears in an image was not relevant to compute the score. The score was computed using the rules shown in Table 3. For a better understanding of the score computation, an example of three di erent hypothetical user decisions for a speci c test image is shown in Table 4. Due to the fact that wrong room classi cations account negatively to the score, participants were allowed to not providing such information, in which case the score is not a ected. The nal score was computed as the sum of the score obtained for each individual test frame. According to the test set released the maximum score to be obtained was 7004 points.
In addition to all the image sequences, we created a Matlab script to be used as template for participants proposals. This script performs all the steps for generating solutions for the Robot Vision challenge: features generation, training, classi cation and performance evaluation. Basic features are generated for both visual and depth images (histograms) and training and classi cation is performed
using an On-line Independent Support Vector Machines [
In 2014, 28 participants registered to the Robot Vision task but only 4 submitted, at least, one run accounting for a total of 17 di erent runs. These participants were: { NUDT: National University of Defense Technology, Changsha, China. { UFMS CPPP: Federal University of Mato Grosso do Sul, Ponta Pora, Brazil { AEGEAN: University of the Aegean Karlovassi, Greece { SIMD: University of Castilla-La Mancha, Albacete, Spain.
Out of competition organizers contribution using the techniques included in the MATLAB proposed script. It can be considered as a baseline result. 4
4.1
This section presents the results of the Robot Vision task of ImageCLEF 2014. The scores obtained by all the submitted runs are shown in Table 5. The maximum score that could be achieved was 7004 and the winner (NUDT) obtained a score of 4430,25 points. This maximum score is the addition of the maximum score computed from rooms classi cation (3000) and object recognition (4004).
SIMD organizers submission was out-of-competition submission, and it was
provided to be considered a baseline score. For this submission, just the
techniques proposed in the webpage of the challenge6 were used. Concretely, it was
generated an image descriptor by concatenating both depth and visual
histograms. These descriptors were then used as input to train an Online Support
Vector Machine [
A detailed view of the obtained results for the best submissions of the 4 participants is show in Fig. 7. This gure graphically presents how participants submission performed notoriously better for object recognition than for room classi cation. For example, the winner of the task achieved 83,78% of the maximum score (3354.75 out of 4004 points) for the object recognition problem, while they just obtained 1075,5 out of 3000 points (35,85%) for the scene classi cation problem.
5000 4000 3000 2000 1000
0 -1000 -2000 Objects Rooms NUDT
UFMS
SIMD
AEGEAN
In relation to the participant approaches, NUDT and UFMS groups
submitted a working note with internal details of their submissions. The NUDT
proposal [
In view of the obtained results, we can conclude that room classi cation remains as an open problem when generalization is requested. That is, current approaches (as shown in previous task editions) perform well when the test environment has been previously imaged during training, but they present problems to classify frames acquired in new environments. On the other hand, we should point out the high performance of the submissions when facing the object recognition problem. This can be explained because object recognition does not rely on the scene generalization as for the room classi cations. Namely, phones or chairs will always be recognized as their type (a phone or a chair, respectively) independently from the scene where they are placed. 5
In this paper, the overview of the 2014 edition of the Robot Vision task at ImageCLEF has been presented. We have described the task, which had slightly variations from previous editions, and a detailed analysis of the results obtained for the participants proposals.
As a novelty for this edition, we have introduced physical changes in the environment where the test sequence has been acquired. That provides an additional component to the classical place classi cation problem, empathizing in the generalization. According to the obtained results, this novelty has resulted in a notorious decrease on the room classi cation performance: none of the submission achieved more than 40% of the maximum score. The inclusion of depth images in the participants proposals could have increased the performance of the room classi ers. With respect to the object recognition, it was properly managed by the NUDT group that ranked rst.
As future work, we plan to manage both room classi cation and object recognition problems jointly. All the participants solutions are based on using the same technique to classify the room and to recognize objects. Both problems are solved without any type of correlation, a di erent way as humans do. Therefore, future work will focus on making participants classify rooms using as input the list of objects recognized in the scene.