Gualzru’s path to the Advertisement World F. Fernández, M. Martı́nez I. Garcı́a-Varea, J. Martı́nez-Gómez J.M. Pérez-Lorenzo, R. Viciana Planning and Learning Group SIMD M2P University Carlos III Madrid University Castilla-La Mancha University of Jaén Email: {ffernand}@inf.uc3m.es Email: {ismael.garcia,jesus.martinez}@uclm.es Email: {jmperez, rviciana}@ujaen.es P. Bustos, L.J. Manso, L. Calderita, A. Bandera, A. Romero-Garcés, M. Gutiérrez, P. Núñez J.P. Bandera, R. Marfil RoboLab Ingenierı́a de Sistemas Integrados University of Extremadura University of Málaga Email: {pbustos, lmanso, lvcalderita, marcog, pnuntru}@unex.es Email: {ajbandera, argarces, jpbandera, rebeca}@uma.es Abstract—This paper describes the genesis of Gualzru, a robot first ’robotic’ consortium was composed by the Universities of commissioned by a large Spanish technological company to Málaga, Extremadura and Carlos III of Madrid. From 2010, provide advertisement services in open public spaces. Gualzru the first two groups were working together on the definition has to stand by at an interactive panel observing the people passing by and, at some point, select a promising candidate and of a software framework for robotics, which could be used for approach her to initiate a conversation. After a small verbal the project. The University of Extremadura would also build interaction, the robot is supposed to convince the passerby to the platform and would be the responsible of endowing the walk back to the panel, leaving the rest of the selling task to robot with the abilities for autonomous navigation and facial an interactive software embedded in it. The whole design and emotion detection. The University of Málaga would address building process took less than three years of team composed of five groups at different geographical locations. We describe here the rest of vision-based problems (e.g. use facial descriptors the lessons learned during this period of time, from different to estimate the gender or age) and help with the navigation points of view including the hardware, software, architectural modules. Finally, the University of Carlos III of Madrid would decisions and team collaboration issues. be in charge of the high-level planning and learning modules. I. I NTRODUCTION Everyone agreed in using this project as a test for the initial proposal from the University of Extremadura: to organize Gualzru is a social robot built as an advertisement tool for the whole software architecture around a centralized internal a consortium of technological and digital media companies model of the outer world. Such representation is accessed by within the ADAPTA1 project. The core of this project is an all software components to keep them informed about the interactive panel able to provide personalized advertisement current world state. They can also update it as the result of according to the preferences of the user. To achieve this goal, processing the data from the sensors. The ADAPTA project the consortium includes advertising companies, media asset will provide a controlled but realistic scenario for testing the management, software developers, technological consultants idea. and software infrastructure providers, coordinated by the Soft- One major requirement of the proposal that the robot ware Labs group from the Spanish Indra company. The idea initially lacked was the ability to dialogue with people. To of using a robot as a more personal way of bringing people’s solve this problem the SIMD group from the University of attention was suggested in order to endow the panel with Castilla-La Mancha, summing a large expertise on automatic the ability to recognize the emotional state of the user and speech recognition and natural language processing, joined the to classify her according to the estimated age and gender. Consortium. Furthermore, the human-robot interaction ability After agreeing on creating a new social robot, it was decided was strengthen with the incorporation of the researchers from that Gualzru, would team up with the interactive panel and the University of Jaén. Six researchers from all groups were boost the advertisement potential of the platform. The project contracted during different periods of time to work on the started in May 2012 and this paper describes our experiences, project, however a larger group of researchers was always successes and failures, during the three-year process. involved on the project. II. T HE TEAM A. Team coordination and sharing of resources From the early analysis of the problem we knew that the project needed the expertise from different research groups. The coordination of this large group was supported by the So, once a solid group of complementary researchers was use of collaborative tools. However, we soon understood that agreed, we accepted to join the project’s consortium. Our the only way to make a steady progress in the development of a large and complex project like ADAPTA, was by sharing 1 See Acknowledgments section at the end. a common code base and by scheduling periodic hackathons FinE-R 2015 Page 55 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 of the project. It was a relatively new process for us with many steps that were out of our direct control. Going in Table I from top to botton and from left to right, we can rapidly summarize the manufacturing steps: 1) Gualzru’s initial 3D design. To come up with a nice robot image we set up a public design contest among all Spanish universities and people and companies in the design business. One person from Cádiz, Spain, was selected among more than 30 proposals with a poll among a selected resolution committee. 2) The 3D drawings were sent to a company specialized Fig. 1. Gualzru the robot in manufacturing expanded polystyrene molds using industrial CNCs. We learned that the choice of prices and qualities here are apparently important, since the in which the members of all the teams could seat together final quality of the surface of the cover and the number for a week and fight a specific, common battle. This strategy of hours spent by the sculptor in fixing the small naturally led to the division of the project in well defined imperfections generated in the machining process were milestones, consisting on system features to be integrated and closely related. It is important to assure the final quality tested during the one-week period. Coordination was therefore level in this early stage. subtended on: 3) An external coating over the mold is necessary to • A unique robotic prototype, available from month zero in facilitate the unmolding process. The mold is split in a robotics simulator. two halves. • A common programming framework, RoboComp [17], 4) A thick silicone layer is manually applied on the mold used in several previous projects. All the software de- with additives to avoid sagging. This layer is called veloped for the project had to qualify as a RoboComp negative. component, meeting the established quality standards, 5) On top of the silicone a resin with fiber glass layer is and had to be uploaded to a common git repository. applied to create a rigid external cover called mother. • A common cognitive architecture, RoboCog, available for 6) Both layers are unmolded. all researchers and where individual modules could be 7) The silicone mold after being separated. inserted and tested minimizing the knowledge required 8) A positive mold is finally built by applying resin and about the rest of the architecture fiber glass inside the negative. After drying, the cover • The organization of several intensive working weeks - is unmolded from the silicone and both parts are glued hackathons- coinciding with the project milestones. These together. A final polish work is done to obtain a nice meetings were intense and dedicated to integrate and texture. debug specific target functionalities. 9) A solid and reliable differential base is built as the To maintain the global view of the project and of the specific mechanical core of the robot. requirements, all the members should meet for each milestone 10) The cover is fit on the base. Additional holes and slits and have always access to an open document storing this in- have to be carved to allow for laser, camera, fastening, formation. The document was edited online by all researchers etc. and also served as a battlefield to discuss technical issues. We 11) Sensors are incorporated to the robot. The tactile screen did not always coincide about how to do things but we agreed is placed after a final coating is ordered to a car painting that the digital arena was the right place to fight. workshop. III. G UALZRU 12) Gualzru at the University of Málaga in a public event with the University’s Provost. Gualzru, a phonetic transcription of the English phrasal verb ”walk through” pronounced by a native speaker of As a summary of the experience it is evident that the process Extremadura, is a 1.60m. tall robot with an external cover is slow, expensive in working hours and almost impossible built of resin and fiber glass, and a differential base with to rectify if a new idea comes by. The whole process took two powered wheels and two casters. It includes gel lead us many more months than expected and we had to use a batteries that provide an autonomy of three hours and all the replacement Nomad 200 robot while the robot was being built. necessary power electronics, recharging and power supplies In summary, it is a valid solution to the cover problem but for the sensors and processors. The complete fabrication of with the arrival of 3D printing technology, all chances are that the robot was custom made by the groups of the consortium. future robot covers will be divided in pieces small enough Table I shows the complex handcrafting process of to be printed in a modern 3D printer, and then assembled Gualzru’s external cover. This step was one of the most together. There are also new small companies starting to offer exasperating and time-consuming in the overall development these kind of services. FinE-R 2015 Page 56 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 TABLE I A SERIES OF SNAPSHOTS OF THE BUILDING PROCESS OF G UALZRU . S EE TEXT FOR DETAILS ON EACH STEP AND THE CONCLUSIONS OBTAINED AFTER IT WAS FINISHED . IV. ROBO C OMP • RoboComp’s component model has been evolving since its beginning and has the necessary complexity for our As commented in section II-A, one of the few things that needs. Not more. were already clear when the project started, was the need of • The current communications middleware, Ice by ZeroC a common code base. [14], is extremely robust. No complaints and a big thank A big part of the group had been already working in you to an excellent open source project. • New middlewares could appear in the future with some previous projects together and sometimes with other partners. From these works we learned that one of the main causes game-braking features. In that case, if you control the that prevented the formation of a cohesive, long lasting group framework you can define a reduced set of communica- with a common goal was the fact that each one was coding tion primitives, like the ones proposed by Schlegel in his their own programs on different frameworks or without one PhD thesis [1] and a set of data types, and write some at all. There are many robotics labs around, still unable to interface code that makes you framework middleware organize and create a coherent code base that grows from the independent. • A code generator is mandatory so the generic part of the accumulated work of dozens of researchers. After some tough negotiations involving the different frameworks that the groups components is always the same, compiles without errors were using or planning to use, we agreed to use RoboComp. and keeps the required quality levels. Code re-generation We believe there are several reasons that, in the hindsight, might be trickier but there are several techniques. Robo- justify this decision: Comp splits the working part of the component in two using inheritance. The inherited part is always generated • We keep the control of the core and thus, we decide when and the part that inherits is generated only the first time. to change and when to hold. It looks like a contradiction A lesson learned here is that it is easier if all tools and but when some complex open source software is very technology in the framework use the same development soon used by thousands of people, its evolution freezes language and environment. Better if it is the one that most or slows down almost immediately. The reason is that of the users are familiar with. Otherwise the no common the core decisions made at the very beginning cannot be specific tool becomes a bottle-neck that might delay and easily changed without generating compatibility problems affect other parts of the framework. and versions nightmares. As an example you can look at Our initial code generator was created with the Eclipse the widely expanded Microsoft’s operating system (Win- ecosystem using the existing tools it provides for DSL dows) and the relatively slow addition of new features designs. This tool turned out not to be easily adjustable by with each release (mainly nothing on the core changes). developers (since they mostly develop C++ and Python) This does not mean that good software cannot be used by and a heavy environment that would not exactly match the many people, but that complex software that deals with team needs. Therefore, we ended up rewriting a lighter new, changing, not very well defined sort of things, takes code generator in Python using pyparse and COG so its time to settle down. FinE-R 2015 Page 57 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 everybody could collaborate in the natural evolution of This now familiar scheme was synthesized by Erann Gat as the tool. Now, RoboComp’s code generator generates the three-layered architectures [2], probably the most extended also Python components, that are becoming more a more approach to build deliberative-reactive agent control systems popular due to their simplicity. today. However, when making decisions that directly involve • We have developed all the tools we needed, although human users, the domain of HRI, these architectures present there are always tools that we would like to have but we some limitations. The most important one, from our point of have not had time to code them. RCIS deserves a special view, is the need of a shared representation among all agents mention, RoboComp’s simulator, that it has been there including metric and symbolic information, making each of almost since the beginning of the framework. By the time them more aware of what was going on in the rest of the RoboComp started the only existing open source simula- agents. For example, if a navigation module is driving the tor was Gazebo and it was in its early versions. If you are robot to a target place, and a person appears somewhere building a robotics framework and have already decided close to the planned path, how does the navigation agent on the communications middleware, the chances are that differentiate between and obstacle and the person, so different you want a simulator that speaks the same language avoiding (social) behaviors can be elicited? Or how does a as the components of the framework. Only doing so, conversational module knows that the person the robot is the simulator would behave like a component or several talking to, is not paying attention anymore, and thus a change components with all the advantages that come with that. in the discourse is advisable? Therefore we wrote RCIS using Open Scene Graph [20] To us, it looks like that the good engineering practice of de- and an initial scene specification language that we named coupling the problem in parts of infinite impedance, took away InnerModel. Later on, we discovered with great joy that a crucial element, context. It is sometimes argued that context having our own simulator would immediately provide us is somehow coded in the interactions between agents [3] or in with an emulator. That is, a simulator that could be run the dynamics of coupled differential equations [4]. We decided inside the architecture computing in super real-time future to take here the more classical path of building an explicit courses of action and predictions. That is now part of our shared representation for the context and face the problems to new architecture CORTEX, which is still in development. come. To start, we already had a representation of the robot V. ROBO C OG and its close environment in the form of a scene-graph, called InnerModel. This simple DSL served to initialize our Initially, we addressed the ADAPTA project from a very RoboComp’s 3D simulator, RCIS. Therefore, RoboComp’s specific point-of-view. That is, giving the use case, we trans- InnerModel was the perfect starting point to develop the idea lated it to a finite state machine and assigned tasks to software of a shared representation of the robot, the environment and components or groups of them that we call agents. The the people in it. The initial scene graph specification language idea of using a finite state machine to manage the whole was gradually extended to include more types of objects. Also use case was soon unbearable. The number of states and a C++ class was written to hold in memory the graph and allow transitions grew with every bit of reality added to scenario. an easy and safe access to all the handy functionalities that Even modern hierarchical and concurrent formalizations of this structure provided, such as coordinate transformations, state-machines [8] and ready to go implementations such as measuring, insertion, modification and removing of nodes, the Qt StateMachine Framework [9] did not offer enough perspective changing, frustum reachability, etc. flexibility and maintainability to risk a project with many A basic scene-graph is essentially a kinematic tree with potential implications for our future. some add-ons. We had to incorporate all the symbolic in- We thus decided to take the hard way to a fully fledged formation needed by the deliberative elements of the ar- symbolic planning system, in charge of the automatic gen- chitecture. The requirements were that the perception-action eration of those huge state machines. The Planning and related agents could update a fixed set of symbolic attributes Learning Group at the University Carlos III of Madrid had and predicates, and that the selected representation could be a very long trajectory in these disciplines and was the perfect efficiently translated to PDDL, so a specialized planning, match to provide the needed technology. The use case was executing and monitoring framework like PELEA [5] could be translated into a PDDL domain specification [18] and several used. Our election on how to proceed in that situation was not planning algorithms were tested for that domain. A separated exactly a mistake but it was certainly close to it. We decided interface was clearly defined between high and low level to build a second graph, this time a graph rather than a tree, to domains. High-level being the domain of logic attributes and hold this symbolic data and leave to a less stressed moment predicates, and low-level the domain of behavior agents that the problem of how to integrate both structures. It was not receive parametrized calls to act and provide metric values a mistake in the sense that the solution worked well and the for relevant variables of the world state. The interface layer robot managed to complete the use case. It was a mistake translates between high and low level, so both worlds are kept in the sense that now, months after the end of the project, communicated. Of course, it is also the main cause of the so we are hurrying to finish the integration of both structures called, symbol grounding problem [11]. because the separation is already generating many problems. FinE-R 2015 Page 58 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 The current solution we are working on is the embedding of may be some limited capability of planning or sequencing sub the kinematic tree inside the symbolic graph, and the code tasks, e.g., maintain the interest of the person through dialog, necessary to efficiently extract and insert the tree in a format monitor the correct execution of a gesture, recognize her facial that can be used by the many components that were written emotions, etc. The most basic components are in charge of before the integration. It is hard to evaluate if the other choice sensor motor loops and normally execute their commands would have permitted us to finish the robot on time, saving the without interaction with other components. posterior integration step. Software developing time is really VI. T HE USE CASE hard to estimate, specially when robots are in the loop. The new graph was named AGM, for Active Grammar- We now describe the use case that constituted the main based Model [6], and besides fulfilling both requirements, it goal of the project. It was defined in the ADAPTA’s kick-off was also an experiment on planning with a variable number meeting. The first version of this use case is depicted in Figure of symbols. In HRI, the perception component is getting 3. It can be textually described as, more and more important. When interacting with a human Gualzru is waiting in the Waiting area. It is now in domestic or service environments, there are references to ready to start one of its tedious working days. The objects and places that may not be known beforehand. Of Waiting area is at the middle of an uncluttered course, planning in an open world takes you out of the comfort corridor in a large shopping center. People usually zone of algorithms, where you know that the program will enter this side of the mall from the left side of the finish. In open worlds, there is always the possibility of adding corridor, crossing in front of the Panel area before a new symbol if the solution does not arrive. entering the shops. People going out the mall also AGM and InnerModel where independent structures and cross in front of the Panel area, but walking toward their coordination was managed by ad-hoc procedures, but the left part of the corridor. The objective of Gualzru AGM was finished on time and will maintain a complemen- is to offer products and services to all these people. tary, symbolic representation of the robot and the objects in In fact, its aim is to drive potential consumers to the scene. For example, a human in front of the robot being an advertising panel, in which these products and detected and represented as a skeleton inside InnerModel, services will be displayed. As there are products for could now be tagged happy, focused or woman in AGM. With everybody, it can choose any person in the corridor. AGM we had the missing part of the architecture and all the When it chooses a target, it moves from the Waiting groups could start to meet in hackatons and reach, one by one, area following an intersecting trajectory with the the urgent remaining milestones of the project. In summary, person’s heading direction. This displacement is very the main purpose of this dual representation was to provide short (2-3 meters maximum) and allows Gualzru to both, a local description that could be updated and used wait for the person in a static pose, facing her at by the different agents for their computations, and a shared comfortable social distance (1,5-2 meters minimum). context that is propagated among them to carry information Therefore, Gualzru can say ’hello’ to the person that otherwise would remain hidden. without scaring her even if she is not very used to As a result of the graphs occupying their places with the interact with a moving robot. agents, the overall idea of RoboCog started to change and If the person engages with him in this first contact, we started to move from the three-tier original model to a Gualzru will classify her into a group -using gender non-hierarchical disposition in which all agents gather around and age parameters- and will choose a product these shared graphs and interact among them by reading, topic to offer. Product topics provide Gualzru an writing and propagating the changes. The abstraction axis is specific theme of conversation before inviting her hidden inside the agents and defines what parts of the graph to walk back to the Waiting area. During this short are accessible by its internal components. This is discussed conversation, Gualzru will be always ready to say in recent works by the group [7], [12]. Figure 2 shows a goodbye to the user if she shows the intention of schema of the RoboCog architecture by the time of the final leaving the conversation or if the presented product demonstrations of the project. topic is not interesting to her. On the other hand, When a mission is assigned to the robot it is internally Gualzru must also check its batteries level to say re-coded as a desired state in an AGM graph, which could goodbye and move to the Charging area if this level include the whole world or just the symbols needed to satisfy is under a minimum value. The Charging area is the mission. The Executive module is the one in charge close to the Waiting area. In fact, when Gualzru of achieving it. The steps needed to transform the current arrives to the Waiting area, he will home itself to AGM graph are provided to the Executive by the Task-based the Charging area. If the person agrees on going Planning and Monitoring module PELEA as a sequence of with Gualzru to the Waiting area, both move there tasks that are injected back in AGM. At this point, AGM holds and the robot says goodbye to her. Then, it returns the current belief about the world and the current desire about to the Waiting area and waits for some time, which how the world should be. Agents scan the graph and find is the expected time that the person is going to be tasks that can be performed by them. Inside each agent there at the panel, before starting the process to select a FinE-R 2015 Page 59 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Fig. 2. An overview of the RoboCog architecture (from [16]) new target. As before, if batteries level are under a in the way. Hackatons have turned out to be an effective way certain value, Gualzru moves to the Charging area to code, debug, test, share, make progress and build a team for a reload. spirit. A. May 2012. Kick-off The project initiated with a kick-off meeting at Málaga where the overall strategy was discussed and the periodicity of the meetings was set. B. December 2012. The ”WORST” workshop at Cáceres The main objective of the first hackathon was to explain and establish RoboComp as the common code base. All groups on the consortium had certain degree of knowledge about RoboComp, but it was considered mandatory to organize a workshop where simple examples could be programmed by all researchers under the supervision of experts from the Universities of Extremadura and Málaga. Fifteen people from all research groups and some more and some from Indra Software Labs met at Cáceres. C. May 2013. First public demonstration at Málaga Fig. 3. The ADAPTA use case For the first public demonstration of the whole project we had a simple prototype of Gualzru (Figure 4). Two autonomous VII. A N ANNOTATED DIARY behaviors were tested: the reactive navigation and a face As was mentioned before, the coordination of the project detector. The AGM graph and the kinematic tree were able to was based on periodic hackathons. We think this decision internalize the perceived information. The seed of the architec- was a real success. We have already noticed the difficulty to ture was planted. Obviously, not everything worked properly. integrate complex software and reach milestones without a real The algorithms underneath both behaviors were changed in motivation from the people working in different labs. Many the final version, for example. But this fact was rather usual times, the global objective or the potential implications of the during the project. Other issues were more time consuming work are not correctly perceived. Other, personal relations get as expected. During 2013 we tried to replace the laser by FinE-R 2015 Page 60 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Fig. 5. Playing with the ’empty-boxes’ architecture. The dashboard shows different panels: the one on the right shows the graph models that encodes (a) the goal to achieve -target model plan, and (b) the symbolic view of the outer world. It also includes the current action of the plan (’approachperson’ in this Fig. 4. The initial internal skeleton of Gualzru figure). The panel on the left shows a visualization of the kinematic tree -up- and of the virtual world -down. We did not endowed the virtual robot with virtual sensors. The person is automatically detected if she is in front of the robot. an array of RGBD sensors, arranged in a configuration that provided a wide field of view and theoretically gave good 3D coverage. Researchers from the Universities of Extremadura interaction stage. The RGBD sensor was used to detect the face and Málaga were involved on achieving this goal. The sensor of a real person during the conversation. It was an intensive had limitations to perceive at short distances and we had to integration task. connect them to embedded computers like the Raspberry Pi of One of the major successes of this architecture was the the time to liberate the USB ports in the main computers of development of the triangle, high-level decision maker - exec- the robot, Intel’s NUC. We could not make the Asus’ Xtion to utive - symbolic graph model. We were now able to translate run reliably with the available Raspbian. A few months after to PDDL the information stored and updated in the symbolic the end of the project, we succeeded with another board, the graph (AGM) to the PELEA framework at the deliberative ODroid C1 [13], and now we can create hard-components that level. Furthermore, the Executive module was able to publish are cheap and provide real-time performance. We spent a few the graph to all software components on the architecture months trying to make that device work because we though it when a change was introduced. These components were ar- should work. Clearly, it was not the time. You have to choose ranged on networks, connected to the Executive through one the right battles. distinguished component, the so-called agents. These agents were the responsible of maintaining the coherence of the D. November 2013. First evaluation of the architecture at information stored in the inner model, since they update the Albacete graph-model and, simultaneously, the geometric information One of the major goals achieved after the meeting at Málaga of the kinematic tree. This second route was not supervised by was the development of a complete architecture able to work the Executive. For the first time, the new definition of agent, with a simulated robot in a virtual environment. The so-called included formally in the RoboComp component model and ’empty boxes’ architecture took this name from the fact that code generator, allowed all participants to share the graphs it included a complete version of the architecture RoboCog, using the same interface. Our shared global representation on although some of the components only had the public interface the state was now real and working. -IDL file- and the structure inside. Nevertheless, it included We were able to launch more than fifteen software com- the two inner graphs -the symbolic and the geometric-, the ponents. From this point of view, we were able to modify or conversational module, reactive navigation, person detector add new components over a full-integrated architecture. Each and high-level planning, executing and monitoring. To play successive meeting would imply a refinement of the previous with it we did not need the physical robot, but a computer proposal. While waiting for the robot Gualzru, see Section III with a RGBD sensor, speakers and a joystick. The simulated for reasons explaining the long wait, we set up an old Nomad robot operated as an autonomous agent and we were able to 200 robot with the RoboCog architecture and organized a new move a virtual person in the simulator using the joystick. When meeting at Albacete. See Figure 6. robot and person were at interaction distance, the robot tried At Albacete we evaluated for the first time the robot’s to convince the person to accompany it to the advertisement behavior through questionnaires filled by the people interacting panel using his conversational skills. The speakers and the with the robot. The questionnaire is designed as a Likert microphones on the RGBD sensor were used to support this scale, although it uses six levels, from 0 to 5, to remove the FinE-R 2015 Page 61 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 TABLE II Q UESTIONNAIRE RESULTS (50 TESTS ) Question x̄ σ 1.1 Do you feel safe when the robot approaches you? 4.31 0.95 1.2 Does the robot invade your personal space? 0.96 1.37 1.3 Do you think robot movements are natural? 2.62 1.23 1.4 Have you stepped away from the robot? 0.96 1.46 2.1 Have you understood the robot? 3.57 1.28 2.2 Has the robot understood you? 2.70 1.30 2.3 Was the conversation coherent? 2.96 1.38 2.4 Do you like the voice of the robot? 3.13 1.29 3.1 Did the robot get blocked? 1.39 1.72 3.2 Was the interaction natural? 3.11 1.11 3.3 Was the conversation fluent? 2.85 1.22 3.4 Did the robot seem to be tele-operated? 0.87 1.44 4.1 Did you enjoy the experiment? 4.31 0.88 4.2 Do you think the exp. was not interesting? 0.70 1.32 4.3 Would you like to repeat? 4.28 1.32 4.4 Would you recommend it to other people? 4.52 0.86 Fig. 6. The old Nomad performing through the use case at Albacete did this integration without time to test the whole system. Also, we put the emphasis on collecting a larger collection of questionnaires. And to worsen things even more, the meeting neutral option -middle point. It is similar to that employed was not set as the other previous hackathons: the goal did not by Joosse et al. [15] to generate the database BEHAVE-II. focus on solving technical problems that were really there, Its main difference is that it has been created not from the but on showing a prototype that, at the end of the day, we point of view of the person observing the behavior of the user should have known that it would not do the job. And the against the presence of the robot, but from the point of view results were not good. During the demonstration the robot was of the same user that interacts with the robot. In this sense, able to interact with a person but it showed its limitations: the we can consider that it collects influences of questionnaires odometry alone was not able to correctly solve the localization of the Almere original model for man-machine interaction. In problem, speech recognition had problems to work on crowed particular, the questionnaire includes a collection of questions environments, the person classification module blocked the full arranged in four blocks: navigation, conversation, interaction use case until a good image of a face was taken, and so on. and general sensations. The user fills the questionnaire giving Another significant lesson was learned and never forgotten: a value for each response between 5 (completely agree) and you can not say that your robot will succeed in one trial until 0 (completely disagree). These questions are listed in Table you have tested it for at least hundreds of times. There is a II. From that point on we would use that tool to evaluate the saying that can be applied here: ”let’s rehearsal so hard that evolution of the project. Although at Albacete we were able the show looks like a rest”. to run the software in a real robot, we only could finish 12 use cases with different users. The number of questionnaires F. June 2014. Hackathon within the Workshop on Physical were also reduced to take hard decisions based on it, however Agents the results were promising. In June 2014, all groups had talks within the Workshop on Physical Agents (WAF2014) to be held in León (Spain). E. May 2014. Second public demonstration at Málaga We asked the local organizers to facilitate us a working space The Nomad 200 was moved from Albacete to Málaga and to set up another hackathon during the week before. Vicente we continued testing specific problems related to the naviga- Matellán, the conference director gave us a cordial welcome tion module, speech generation and recognition, and person and provided an excellent place for testing. classification based on age and gender. The communication Before the hackathon, we discussed and organized the prob- among components was forbidden and all information was lems to solve there and when we arrived to León everybody transmitted through the inner representation. Things still did knew what to do and joined in groups for a long week. not work as we needed and the causes were not clear. Then, The result was a real success: the dialog module was largely the robot Gualzru arrived and all efforts were translated to improved and tested, the localization problem was solved using getting it ready. AprilTags landmarks, and so on. The use case was repeated For the second public demonstration Gualzru was already and repeated, and for the first time we detected real bugs and running RoboCog. Probably, this was not a mistake, but we problems to deal with. After several days of intense work, FinE-R 2015 Page 62 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Gualzru was able to do its job relentlessly until the battery was the robot. This situation is more common than expected due off! The robot spoke with all of us in the Lab and accompanied to the interest the robot produces. Additional issues such as us to the panel. We had the impression that all ours problems different accents, voice volumes, etc. add more difficulties were solved. We were happy for the moment... to the scenario. Despite these limited conversational skills, But we were not going to enjoy the success for a long Gualzru achieved its main objective, to capture the attention of time. After discovering that the learning module would make people. Most of them enjoyed the experiment and also would the robot avoid people that always answered: ’No, I do not recommend the experience to friends or would like to repeat want to go with you to the panel’, our host asked us to move it. Comparing these results with the ones collected in the first Gualzru to the large hall where the conference was about to experiments, revealed that successive updates in the robot have start and to have it welcoming the assistants. It looked like a made it more robust and its conversational abilities, while still good scenario for our use case. We accepted. constrained, have been significantly improved. The moment we moved to the hall new problems appeared. The robot was unable to talk to people because nobody, not H. Last stage: refining the HRI even humans, could hear what the other was saying. The space The conversational abilities represented a severe drawback. was wide open and we could not find a good place for the Despite our efforts, only 50 % of the people that interacted AprilTags landmarks. Light conditions were changing and the with the robot in these real scenarios thought that it was able algorithms in charge of the RGBD camera did not always to maintain a coherent conversation. This was not enough for run correctly. For our younger researchers the experience was a robust, useful robot. But if you cannot solve a problem, really hard, as they passed from the complete success to a perhaps is a good option to totally change the way to solve glaring failure in a short time. Nevertheless, we were now in it. The speech recognition issue is hard to solve in noisy and the final scenario. A spacious environment where the robot crowded environments, where even humans find difficulties must interact with a specific person while other people are in understanding each other. Therefore, our idea was to look speaking and moving around. The failure had an aftertaste for alternative methods to allow people communicate with the of an approaching victory. We still were able to close some robot. Speech recognition was reinforced with the incorpora- use cases in this challenging scenario and a new time for tion of a tactile screen installed on the chest of the robot. The improvements had started. verbalized phrases were now displayed on this screen and it was possible for the person to answer the robot by touching G. December 2014. Large evaluation test at Málaga it. This way, Gualzru retained its conversational abilities but After a new demonstration at Ingenia (Málaga), in an the new interfaces increased its robustness and reliability. environment very similar to the hall at León where we could Following this modification, a new set of questionnaires were capture new questionnaires, we returned to the Lab. The array collected on the same scenario at the University of Málaga. of microphones of the Kinect sensor was intensively tested These questionnaires showed us that the mean values related and, finally, we decided to change it for a shotgun microphone. to questions 2.1 and 2.2 (Table II) improved from 3.57 to 4.27 As it is described with more detail in [12], other minor issues and from 2.7 to 3.72, respectively. Additional changes on the were also solved. whole architecture allowed the robot to successfully close 93 On December, 2014, the current version of Gualzru was % of the started use cases (on December 2014, this rate was 81 tested in a real working scenario. The system was deployed %). Furthermore, the unfinished use cases were always caused in the hall of the Escuela de Ingenierı́as at the University of by the large amount of people in the place that would prevent Málaga. The area where the robot was operating was about 70 Gualzru from reaching the panel. square meters. Fixed obstacles included a column and some tables, but most of the area was free for the robot to move. VIII. C ONCLUSIONS The hall was populated by students and the trials lasted two In this paper we have presented the long process of creation half-days. The robot worked without human intervention and of Gualzru, the salesman robot built for the ADAPTA research engaged with people passing nearby. These people had no a project. Looking at the starting requirements, we can firstly priori knowledge about the robot, nor its functionality. We conclude that the final version of the robot conforms with collected a large set of questionnaires. The results are shown the goals and expectations that we and the companies in on Table II. the consortium initially had. But it is also true that even This data showed that the conversational system remained more rewarding than Ursus has been the whole process of as the weak point of the robot. Some people did not correctly collaboration and the knowledge distilled during these years. understood the robot due to the environmental noise and the It is not that common that basic research is taken close voice of the robot was perceived as not particularly pleasant. to the production line while all the intermediate steps are But the most important issue was related to the understanding registered and analyzed as a means to improve both, the capabilities of Gualzru. Even when using the shotgun micro- forthcoming research and the methods and ways to generate phone these capabilities were strongly limited. The system is reliable technology, given a limited amount of resources. For too sensitive to environmental noise and echos and it gets us, it has been a productive experience, both personally and also confused when there are several people speaking around professionally, and the capacity of the group to approach FinE-R 2015 Page 63 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 new technological challenges has increased notably. We have to be very important to improve the people’s attitude towards learned something useful in every step of the project, from the robot, as well as to reveal the most urgent weaknesses the handcraft manufacturing materials and steps, to the way in preliminary stages. It is a valuable lesson to be kept humans are starting to look at the (social) robots. that periodic tests and surveys are an important part of HRI From the point of view of the technology that has been research, although they are often seen by roboticits as a dull, created and used in the project, we reaffirm the initial idea questionable use of the scarce human resources available. of the need for a common code base that brings together the Another important source of feedback are the robotic contests work of all researchers. We still need some adjustments in the like RoboCup@Home2 or RoCKin3 , that put all teams in the protocols and some refinements in the technology, and even track of a common goal, and where real performances are more conviction by some doubters, but at the end of the day we evaluated in front of expert judges. might well be in the right track. The cost of maintaining and The ADAPTA project officially finished on May 2015 with improving a framework like RoboComp is compensated by the a final public demonstration in Málaga. There Gualzru was flexibility of adapting it to your needs. Making good choices able to interact with many people and successfully closed in this field, where Robotics meets Software Engineering, is several difficult use cases. All partners in the consortium were not easy at all but once the software reaches a certain point satisfied and the robot will be serving from now on at the of maturity, the leverage is undeniable. In the near future, we headquarters of Indra in Madrid. The research groups are now believe that these frameworks will play a crucial role in the involved in more collaborative projects that hopefully will fund evolution of intelligent robots. A role much more important the construction of new social robots. We hope that the next that it is given today. It is needed the confluence of interested generation will be capable of providing a better service to people from Software Engineering to gradually introduce new humans. advances in DSLs, meta-models, model-based design and com- munication middlewares. From the recent evolution of social ACKNOWLEDGMENTS robot software, it looks like to us that the near future will bring This paper has been partially supported by the Spanish larger and finer-grained networks of components, hundreds Ministerio de Economı́a y Competitividad TIN2012-38079 and within the next years, that will demand more efficient software FEDER funds, and by the Innterconecta Programme 2011 communications, self-diagnosis and repair, and sophisticated project ITC-20111030 ADAPTA. monitoring and deployment systems. Maybe classic, coarse- grained architectures will meet fine-grained ones at a point R EFERENCES where interaction dynamics play a relevant role. [1] C. Schlegel, “Communication patterns as key towards component-based The cognitive robotics architecture, RoboCog, is a much robotics,” Advanced Robotic, vol. 3, no. 1, pp. 049–054, 2006. more experimental and uncertain piece of handcraft. We [2] E. Gat, “On three-layer architectures,” in Artificial Intelligence and Mobile Robots, D. Kortenkamp, R. P. Bonnaso, and R. Murphy, Eds. started with a standard three-tier schema and managed to AAAI Press, 1998. integrate symbolic planning with a fair amount of perception [3] R. Brooks, “Intelligence without representation,” Artificial intelligence, and action. To get there we re-introduced the idea of a shared vol. 47, no. September 1987, 1991. [4] C. ELIASMITH, “The third contender: a critical examination of the dy- representation among modules playing the role of an explicit namicist theory of cognition,” in Mind Readings: Introductory Selections context. It was implemented as two graphs, one geometric and on Cognitive Science, P. Thagard, Ed. MIT Press, 1999. one symbolic, and it proved enough for the required adver- [5] C. Guzman, V. Alcazar, D. Prior, E. Onaidia, D. Borrajo, J. Fdez- Olivares, and E. Quintero, “Pelea: a domain-independent architecture for tisement scenarios. Also, the introduction from the beginning planning, execution and learning,” Proceedings of ICAPS’12 Scheduling of symbolic planning and learning technology in the project and Planning Applications woRKshop (SPARK), pp. 38–45, 2012. has proven a huge success. The initial idea of a using a flat [6] L. J. Manso, “Perception as stochastic sampling on dynamic graph spaces,” Ph.D. dissertation, 2013. PDDL description of the domain with a standard planner has [7] R. Marfil, L. V. Calderita, J. P. Bandera, L. J. Manso, and A. Bandera, evolved now into HRI specialized schemes, where hierarchical “Toward social cognition in robotics : Extracting and internalizing mean- planners take care of quotidian, repetitive tasks and flat ones of ing from perception,” Waf2014, no. June, pp. 1–12, 2014. the fine details and contingencies that might occur [19]. But, [8] http://www.w3.org/TR/scxml/ [9] http://www.qt.io/ each solution takes to the next problem and before the end [10] http://www.ros.org/ of the project, we were already working on integrating both [11] Harnad, S. The Symbol Grounding Problem. Physica D 42: 335-346. graphs, reordering the classical hierarchies into more versatile 1990 [12] A. Romero, L. Calderita, J.P. Bandera, J. Martnez, R. Marfil, L.J. Manso, organizations, infiltrate lifelong learning into all crevices of the P. Bustos, A. Bandera. The Cognitive Architecture of a Robotic Salesman, system or use domain specific symbolic planning in classical in review. low-level modules like navigation or object recognition. This [13] http://www.hardkernel.com/main/products/prdt info.php [14] http://www.zeroc.com issue, dealing with the overall organization of robotic intel- [15] M. Joosse and A. Sardar and M. Lohse and V. Evers, BEHAVE-II: ligence, is undoubtedly the hardest one but projects like this The revised set of measures to assess users’ attitudinal and behavioral motivate, and ultimately enforce, the search for new theoretical responses to a social robot, International Journal of Social Robotics 5(3): 379-388. 2013 perspectives. Other crucial part of the global Gualzru experience has been 2 http://www.robocup.org the use of evaluation metrics. User questionnaires turned out 3 http://rockinrobotchallenge.eu/ FinE-R 2015 Page 64 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 [16] Calderita LV, Manso LJ, Bustos P, Surez-Mejas C, Fernndez F, Bandera A. THERAPIST: Towards an Autonomous Socially Interactive Robot for Motor and Neurorehabilitation Therapies for Children JMIR Rehabil Assist Technol 2014;1(1):e1 [17] Gutirrez, Marco A., A. Romero-Garcs, P. Bustos, and J. Martnez. ”Recent Advances in RoboComp.” Journal of Physical Agents 7.1 (2013): 38-47. [18] McDermott, D. (2000). The 1998 AI planning systems competition. AI Magazine,21 (2). [19] J. C. Pulido, J. C. Gonzlez, A. Gonzlez, J. Garca, F. Fernndez, A. Bandera, P. Bustos, and C. Surez, Goal-directed Generation of Exercise Sets for Upper-Limb Rehabilitation, in Proceedings of the 5th Workshop on Knowledge Engineering for Planning and Scheduling (KEPS), ICAPS conference, 2014, pp. 3845. [20] http://www.openscenegraph.com FinE-R 2015 Page 65 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015