This article describes the RobotVision@ImageCLEF 2013 challenge, which addresses two problems: place classi cation and object recognition. Participants of the challenge were asked to classify rooms on the basis of image sequences captured by cameras mounted on a mobile robot. They were also asked to detect the appearance or lack of several objects. The proposals of the participants had to answer two questions: \where are you?" (I am in the elevator, in the toilet, etc.) and \which objects can you see?" (I can see a table and a chair but not a computer) when presented with a test sequence. The number of times a speci c set of object appears in a frame was not considered but if they appeared or not in it. The test sequence was acquired within the same environment but with di erent lighting conditions than the training sequences. The main novelty of the 2013 edition of the task is the object recognition problem. For both problems: place classi cation and object recognition, depth and visual images were provided. Moreover, participants were allowed to take advantage from the temporal continuity of the test sequence. The winner of the 2013 edition of the Robot Vision task was the MIAR ICT group, from China.
This paper describes the ImageCLEF 2013 Robot Vision challenge [
The 2009@ImageCLEF edition of the task [
In 2010, two editions of the challenge took place. The second edition of the
task, 2010@ICPR [
In 2012, stereo images were replaced by images acquired using two types of camera: a perspective camera for visual images and a depth camera (the Microsoft Kinect sensor) for range images. Therefore, each frame consisted of two types of images and the challenge become a problem of multimodal (place) classi cation. In addition to the use of depth images, the optional task contained kidnappings and unknown rooms (not previously seen in training sequences) not appeared in the test sequences. Moreover, several techniques for features extraction and cue integration were proposed to the participants.
For the ImageCLEF 2013 Robot Vision challenge we changed the visual data, providing the traditional RGB images and its corresponding point cloud information. The main di erence from 2012 edition was that no depth image was provided but the point cloud itself. The purpose of that was to encourage the participants to make use of 3D image processing techniques, in addition to visual ones, with the aim to obtain better classi cation results. Furthermore, for some speci c rooms, we provided completely dark images for which the use of the 3D information had to be used in order to classify such a room.
Regarding the participation, in this edition, we received a total of 16 runs, from 6 di erent participant groups. The best result was obtained by the MIART ICT research group from Beijing, China.
The rest of the paper details the challenge and is organized as follows: Section 2 describes the 2013 ImageCLEF edition of the RobotVision task. Section 3
presents all the participants groups, while the results are reported in Section 4. Finally, in Section 5, the main conclusions are drawn and some ideas for future editions are outlined. 2
This section describes the details concerning the setup of the ImageCLEF 2013 Robot Vision task. In Section 2.1 a description of training, validation and test sequences is provided. In Section 2.2 the performance evaluation criteria is detailed. Finally, in Section 2.3 a brief description of the baseline visual place classi cation system provided by the organizers, as well as other relevant details concerning the task are presented. 2.1
The fth edition of the Robot Vision challenge was focused on the problem of multi-modal information retrieval from indoor scenes. Participants had to detect, for each test image, the presence or lack of a set of objects. They also had to determine the kind of room where the image was acquired from. All the images were captured by a perspective camera (visual images) and a Kinect device (depth images) mounted on a mobile robot (see Fig. 1) within an o ce environment.
Participants had available visual images and depth images in Point Cloud Data (PCD) format. Fig. 2 shows the same scene represented in a visual image and a point cloud data le. Training, validation and test sequences were acquired within the same building at two di erent oors but with some variations in the lighting conditions and in the acquisition procedure (clockwise and counter clockwise, ground oor rst or ground oor last).
Visual Image
Point Cloud File
In one hand, and as opposite to previous editions of the challenge, a single task was considered this time, therefore no sub-tasks were de ned. All the room and object categories included in the test sequence were previously seen during training. On the other hand, the use of the temporal continuity of the test sequence was allowed. 2.2
In the 2013 edition of the RobotVision challenge the O-VIDA Robot Vision dataset was used. This dataset consists of di erent training, validation and test sequences of depth and visual images acquired within an indoor environment: a department building at the University of Alicante. Visual images were stored in PNG format and depth ones in PCD. Every image in the dataset was manually labelled with its corresponding room category/class and with a list of eight di erent objects to appear or not within it. The 10 di erent room categories are: corridor, hall, professorO ce, studentO ce, technicalRoom, toilet, secretary, visioconference, elevator area and warehouse. The 8 di erent objects are: extinguisher, computer, chair, printer, urinal, screen, trash and fridge. From this dataset two di erent labelled sequences were selected for training, one labelled sequence for validation, and one unlabelled sequence for testing. The frequency distribution for room categories in the training, validation and test sequences are depicted in Table 1.
It can be observed that in all sequences, Corridor is the class with higher number of frames. This is because most of the space of the University of Alicante building, suitable for robot navigation, belongs to several corridors. This situation makes it easier the classi cation of test frames as Corridor while other classes as Warehouse or Toilet are more challenging. The validation sequence was released for providing participants an additional sequence for testing their preliminary proposals. It was also released for preventing the extreme lighting conditions present in the test sequence. The validation sequence was acquired just in the rst oor of the building and it does not contains any frame for three rooms: Hall, VisioConference and Warehouse. The frequency distribution for object categories in the training, validation and test sequences are depicted in Table 2, where can be observed that there are no presence of Screens in the validation sequence. The di erences between all the room categories can be observed in Figure 3, where a single visual image for each of the 10 room categories is shown.
Corridor
Hall
ProfessorO ce
TechnicalRoom TechnicalRoom
Toilet
Secretary
VisioConference ElevatorArea
Warehouse
Fig. 4 shows four examples of visual images for each of the 8 di erent objects appearing in the dataset.
Exting. Computer
Chair
Printer
Urinal
Screen
Trash
Fridge
The runs submitted for each participant were compared and sorted according the score assigned to each submission. Every submission consisted of the room category assigned to each test image and the corresponding list of the 8 detected/nondetected objects within that image. As we already mentioned above, the number of times a speci c object appears in an image was not relevant to compute the score. The score was computed using the rules shown in Table 3. Due to the fact that wrong room classi cations and/or wrong object detections account negatively to the score, participants were allowed to not providing such information, in which case the score is not a ected. The nal score was computed as the sum of the score obtained for each individual test frame. According to the test set released the maximum score to be obtained was 7030 points.
Class / Room Category Room class/category correctly classi ed +1.0 points Room class/category wrongly classi ed -0.5 points Room class/category not classi ed +0.0 points
Object Recognition For each correctly classi ed object whitin the frame +0.125 points For each misclassi ed object whitin the frame -0.125 points
For each not classi ed object whitin the frame +0.000 points 2.4
As in the previous edition [
(PHOW) [
Depth Features NARF features is a novel descriptor technique that has been
included in the point cloud library [
In 2013, 39 participants registered to the Robot Vision task but only 6 submitted, at least, one run accounting for a total of 16 di erent runs. These participants were: { NUDT: National University of Defense Technology, Changsha, China. { MIAR ICT: Beijing, China. { MICA: Hanoi university of Science and Technology, Hanoi, Vietnam { REGIM: University of Sfax National School of Engineers, Tunisia { GRAM:University of Alcala de Henares, Spain { SIMD: University of Castilla-La Mancha, Albacete, Spain.
Out of competition organizers contribution using proposed techniques 4
4.1
This section presents the results of the Robot Vision task of ImageCLEF 2013.
The scores obtained by all the submitted runs are shown in Table 4. The
maximum score that could be achieved was 7030 and the winner (MIAR ICT)
obtained a score of 6033.5 points. NUDT and SIMD teams ranked second and third
respectively and their score was higher than 71% of the maximum score (the one
obtained with the baseline system, SIMD result in the table).
* SIMD organizers submission was out-of-competition, it was provided to
be considered a baseline score. The organizers only used the techniques
proposed in the webpage of the challenge 4. Concretely, PHOW features were
extracted from visual images and then, a Support Vector Machine was trained
using DOGMA [
Here we present a deeper analysis of the best submission for each participant group. We have computed separately the score for the class classi cation and the recognition sub-problems. All these results can be seen in Table 5 and Fig. 5. As it can be observed, in one hand, that MIAR ICT and NUDT groups obtained similar scores, with better results for room classi cation than for object recognition. On the other hand, REGIM and MICA proposals ranked better for object recognition than for room classi cation.
We have also analysed the speci c performance for the di erent room categories and objects. For each room class and object considered, we have computed the percentage of right and wrong classi cations. We also have computed the percentage of times of not providing information for room classes or objects. All these data can be observed in Fig. 6 and Fig. 7 for room classes and objects, respectively.
From the reported results, we can state that Hall and Elevator Area are the most challenging rooms, while Corridor is the easiest one. The number of training frames containing these classes can be one of the most important reasons for this
fact. That is, the number of frames containing Corridor is about one order of magnitude higher than the ones containing Hall or Elevator Area.
It can be noticed that all the objects are managed properly by the participant proposals. Urinal was the object that obtained the highest percentage of right detections, while Trash obtained the lowest one. It should be pointed out that, for all the objects (see Table 2), the appearance ratio in less than 30%. Classifying all test frames as \there are no objects in the scene" would obtain a high positive score, especially for Urinal, Fridge or Screen. Chair and Trash could be considered the most challenging test objects because their appearance ratio is higher than for the rest of the objects. There are two possible reasons explaining that participants obtained better results for Chair than for Trash: (1) trashes are considerably smaller than chairs, and (2) trashes can appear in most of the room categories while chairs are only present in 6 rooms (TechnicalRoom, ProfessorO ce, StudentO ce, Secretary, VisioConference and Warehouse). 5
In this paper the overview of the 2013 edition of the Robot Vision task at ImageCLEF has been presented. We have described the task, which had slightly variations from previous editions, and a detailed analysis of the results obtained for each run submitted by the participants.
As a novelty for this edition, we have introduced the additional problem to recognize speci c objects that can appear within an image. That provides and additional component to the classical place classi cation problem, turning it into a multimodal classi cation problem.
According to the obtained results we can conclude that the introduction of the object recognition task was not as challenging as we expected: most of the participants were able to identify those objects properly. With respect to the scores obtained by the di erent runs, almost half of them improved the baseline results provided by the organizers, obtaining score higher than the 80% of the maximum score.
For future editions we plan to continue in the direction of including new challenging variations to the problem of scene classi cation problem. In particular, as the next step forward we will focus on providing the number of occurrences a speci c object appears in an image.