2015 The Path to Success: Failures in Real Robots (FinE-R) Proceedings for FinE-R Workshop Hamburg, Germany - October 2, 2015 https://finer-iros2015.appspot.com/ The Path to Success: Failures in Real Robots (FinE-R) FinE-R 2015 The Path to Success: Failures in rEal Robots Main goals of the workshop Along the history, there are many important discoveries that resulted from long trials and error processes (e.g. the electric light bulb from Edison) or from analysing 'failed' results (e.g. the Michelson-Morley experiment). In each case, the key contributor for the final success was the willingness to learn from previous mistakes and to share the gained experience with the research community. The path to progress in the field of robotics is not free of failures and caveats. These failures provide valuable lessons and insights on future approaches by analysing errors and finding methods to avoid them. As such, the robotics community could benefit from the experience of those who had faced and overcome similar failures before. The objective of this workshop is to provide a forum for researchers to share their personal experiences on their "failure to success" stories, to present what they have learnt, what others should avoid while experimenting in similar context, providing tips for better research practices and for creating more successful robots that meet people's expectations. Topics of interest (but not limited to)  Analysis of failures when participating in robotic challenges  Design of robust human-computer interfaces for robots  When failure is not an option: creating an outstanding robot from HW to SW  The search for errors: benchmarking and tools for testing robots  Avoiding common but frequently seen errors when deploying robots for industrial or general public environments  Advanced techniques for failure recovery and troubleshooting  Matching the expectations and needs of industries and consumers with the current technology  Alternatives to techniques and algorithms that are prone to fail  The keys for successful research projects and proposals on robotics  Analysis of failed results and projects when using smart algorithms, well-established techniques or brilliant designs Page I The Path to Success: Failures in Real Robots (FinE-R) INVITED KEYNOTE SPEAKERS  Ryad Chellali Nanjing Robotics Institute- CEECS Nanjing Tech University, Nanjing, China rchellali@njtech.edu.cn Lessons Learned in Human Movements and Behavior Analysis Human Robots Interactions (HRI) is a hybrid filed mixing many domains including engineering, physics, social sciences, neurosciences, etc. Accordingly, research in HRI combines exact models and experimental approaches towards developing interaction frameworks and friendly robots. The inherent heterogeneity leads in many cases to ill-posed problems with oversimplification on one side (the engineering side) and forced (thus inexact) models on the other side. Indeed and for the "R" part in HRI, measurements and procedures are known to be exact and objective. Models in these fields are enough known allowing quantitative accurate observations that can be measured repeatedly supporting the original models. In psychology and social sciences the situation is different: the object of studies, namely humans, is much less known. Scientists in these areas are lacking in terms of accurate models compared physicists and engineers. Indeed, humans can be seen as high dimension multivariate systems, with complex dynamics, preventing from having complete explanatory models. This leads to qualitative approaches, where only isolated aspects (and most of the time related indirectly to the object of investigations) are considered. This situation is even worst when experiments are performed in real life conditions. Indeed, to obtain realistic observations, experiments in real world are needed; unfortunately, the control of experimental conditions is almost impossible out of laboratories, leading to higher difficulty and complexity in analysis and understanding. Through our previous works, we found out that one should consider carefully modelling human behaviour. In analysing human gestures for instance, our system used to work well in lab conditions but failed completely in real situations. We imposed a model that cannot handle the variability both intra-personal and inter-personal. Likewise, in studying relationships between body movements and human physiology, we found out a new phenomenon that was not considered in our original model. Our conclusion (which is obvious a posteriori) is that model driven approaches are useless in human behaviour analysis. Instead, data driven techniques (unsupervised, latent modelling, etc.) seem to be more effective. Indeed, in both previous cases we achieved better results by removing modelling constraints and by using statistical tools extracting weakly hypothesized regularity. Short Bio Ryad Chellali is a distinguished professor at Nanjing Tech University, Nanjing China since 2015. From 2005 to 2015, he was senior research scientist at the Italian Institute of Technology. He created and leaded the Page II The Path to Success: Failures in Real Robots (FinE-R) Human Robots Mediated Interactions Group (2006-2011) and then joined the Department of Pattern Analysis and Computer Vision (PAVIS, 2011-2015). From 1995 to 2006, he was with Ecole des Mines de Nantes/CNRS (France), heading the automatic control chair. From 1993 to 1995 he was assistant professor at University of Paris. He served as Junior Researcher in 1992 at the French Institute of Transports (INRETS). He obtained his Ph.D. in Robotics from University of Paris in 1993 and his Dr. Sc from University of Nantes (France) in 2005. His main research interests include robotics, human-robot interactions, human behavior analysis (social signal processing and affective computing). Telepresence, virtual and augmented realities are also keywords of his activity. Ryad Chellali co-authored more than 100 papers. In 2000 and 2005 the French Government awarded him for the creation of innovative technologies companies. Laurence DeVillers LIMSI-CNRS, Sorbonne University, France laurence.devillers@limsi.fr The communication accommodation of the machine to deal with errors in Human - Robot Spoken Interaction Talk during social interactions naturally involves the exchange of propositional content but also and perhaps more importantly the expression of interpersonal relationships, as well as displays of emotion, affect, interest, etc. Such social interaction requires that the robot has the ability to detect, interpret the social language and represent some complex human social behaviour. Cognitive decisions will be used for reasoning on the strategy of the dialog and deciding social behaviours (humour, compassion, white lies, etc.) taking into account the user profile and contextual information. The research challenges also include the evaluation of such systems and the various metrics that could be used like the measure of social engagement with the user. Engagement in dialog with a machine is not only linked to the error rates. We argue that the communication accommodation theory is a promising paradigm to globally consider the errors in the convergence or divergence dimensions. Short Bio Laurence DeVillers is a Professor of Affective Computing at Paris-Sorbonne University and she leads a team of research on "Affective and Social Dimensions of Spoken Interactions" at the CNRS. Her current research addresses the problem of sensing and understanding human non-verbal interactive language and intentions. Her background is on machine learning, speech recognition, spoken dialog system and evaluation. She participates in BPI ROMEO2 project, which has the main goal of building a social humanoid robot for elderly people. She leads the European CHIST-ERA project JOKER: JOKe and Empathy of a Robot. She is member of the working group on the ethics of the research in robotics (CERNA). She is also heading the "Human- Page III The Path to Success: Failures in Real Robots (FinE-R) machine co-evolution" research group at the Numerical Society Institute (France). She has (co-) authored more than 140 publications. She is a member of AAAC (board), IEEE, ACL, ISCA, WACAI and AFCP. She is also involved in the Eurobotics Topic Groups: "Natural Interaction with Social Robots" and "Socially intelligent 1 robots". (Video-demo ). Short institution presentation The Computer Sciences Laboratory for Mechanics and Engineering Sciences (LIMSI) is one of France's largest research laboratories of the CNRS working on language technologies. The team on "Affective and Social Dimensions of Spoken Interactions" (Head: L. Devillers) is working on affective computing and robotics 2 applications . 1 https://www.youtube.com/watch?v=p1ID-gvUnWs 2 https://www.limsi.fr/en/research/tlp/topics/topic2 Page IV The Path to Success: Failures in Real Robots (FinE-R) ORGANIZING COMMITTEE Luis Fernando D'Haro Chair SERC Robotics Program, A*STAR Institute for Infocomm Research, Singapore luisdhe@i2r.a-star.edu.sg Andreea Ioana Niculescu Co-Chair SERC Robotics Program, A*STAR Institute for Infocomm Research, Singapore Aravindkumar Vijayalingam Technical Program TUM CREATE, Singapore Advisory Committee  Rafael E. Banchs (I2R-HLT, Singapore)  Suraj Nair (Technische Universität München, Germany)  Marco Antonio Gutierrez (Universidad de Extremadura, Spain) Program Committee (in alphabetical order)  Aamir Ahmad, Max Planck Institute for Biological Cybernetics, Tübingen, Germany  Marcelo Ang, National University of Singapore, Singapore Page V The Path to Success: Failures in Real Robots (FinE-R)  Rafael E. Banchs, Human Language Technologies - A*STAR, Singapore  Antonio Bandera, University of Malaga, Spain  Pablo Bustos, Universidad de Extremadura, Spain  John-John Cabibihan, Qatar University, Qatar  Raffaella Carloni, University of Twente, Netherlands  Chih-Hong Cheng, ABB, Germany  Luis Fernando D'Haro, SERC Robotics Program - A*STAR, Singapore  Gamini Dissanayake, University of Technology, Sydney (Australia)  Ismael García-Varea, University of Castilla-La Mancha, Spain  Marco Antonio Gutierrez Giraldo, Robolab, Universidad de extremadura, Spain  Martin Hägele, Head of Department Robotics and Assistive Systems, Fraunhofer IPA, Germany  Haizhou Li, Human Language Technologies - A*STAR, Singapore  Dilip Kumar Limbu, Institute for Infocomm Research, Singapore  Luis J. Manso, University of Extremadura, Spain  Jose Moreno, University of Extremadura, Spain  Omar Mubin, University of Western Sydney, HCI & Robotics  Suraj Nair, TUM, Germany  Andreea Ioana Niculescu, SERC Robotics Program - A*STAR, Singapore  Pedro Nuñez, University of Extremadura, Spain  Helmuth Radrich, KUKA, Augsburg, Germany  Eloy Retamino, TUM CREATE, Singapore  Markus Rickert, Fortiss, Germany  Sean Sabastian, SIMTech - A*STAR, Singapore  Eduardo B. Sandoval, University of Canterbuy HIT Lab NZ  Allison Sauppé, University of Wisconsin-La Crosse, USA  Yeow Kee Tan, Institute for Infocomm Research, Singapore  Keng Peng Tee, Institute for Infocomm Research, Singapore  Aravindkumar Vijayalingam, TUM CREATE, Singapore  Erik Wilhelm, Singapore University of Technology and Design, Singapore  Martin Wojtczyk, Baylabs, California (USA)  Thilo Zimmermann, GPS Stuttgart, Germany Page VI The Path to Success: Failures in Real Robots (FinE-R) SUPPORTING ORGANIZATIONS Page VII The Path to Success: Failures in Real Robots (FinE-R) LIST OF ACCEPTED CONTRIBUTIONS Page Luis Fernando D'Haro, Andreea Niculescu, Aravindkumar Vijayalingam, Marco Antonio 1 Gutierrez Giraldo, Suraj Nair, and Rafael Banchs. “The path to success: Failures in rEal Robots (FinE-R)” Anders Billesø Beck, Anders Due Schwartz, Andreas R. Fugl, Martin Naumann, and Björn 5 Kahl. “Skill-based Exception Handling and Error Recovery for Collaborative Industrial Robots” Richard Wang, Manuela Veloso, and Srinivasan Seshan. “Using Autonomous Robots to 11 Diagnose Wireless Connectivity” Hong Tuan Teo, and John-John Cabibihan. “Toward Soft, Robust Robots for Children with 15 Autism Spectrum Disorder” Thommen Karimpanal George, Mohammadreza Chamanbaz, Abhishek Gupta, Wen Lizheng, 20 Timothy Jeruzalski, and Erik Wilhelm. “Adapting Low-Cost Platforms for Robotics Research” Felipe Cid Burgos, Pedro Núñez Trujillo and Luis J. Manso. “Improvements and 27 considerations related to human-robot interaction in the design of a new version of the robotic head Muecas” Joel Stephen Short, Aun Neow Poo, Chow Yin Lai, Pey Yuen Tao, and Marcelo H Ang Jr. 36 “Lessons from the Design and Testing of a Novel Spring Powered Passive Robot Joint” Zheng Ma, Aun-Neow Poo, Marcelo Ang, Geok-Soon Hong and Feng Huo. “Design, 42 Simulation and Implementation of a 3-PUU Parallel Mechanism for a Macro/mini Manipulator” Roger Bostelman, Tsai Hong, and Elena Messina. “Intelligence Level Performance Standards 48 Research for Autonomous Vehicles” Fernando Fernández, Moisés Martínez, Ismael García-Varea, Jesús Martínez-Gómez, Jose 55 Pérez-Lorenzo, Raquel Viciana, Pablo Bustos, Luis Manso, Luis Calderita, Marco Gutiérrez, Pedro Núñez, Antonio Bandera, Adrián Romero-Garcés, Juan Bandera and Rebeca Marfil. “Gualzru's path to the Advertisement World” Page VIII The Path to Success: Failures in Real Robots (FinE-R) SCHEDULE Session Title 8:45 – 9:00 Opening Ceremony 9:00 – 10:00 Keynote: Lessons Learned in Human Movements and Behavior Analysis Ryad Chellali (Nanjing Robotics Institute- CEECS, Nanjing Tech University, Nanjing, China) 10:00 – 10:30 Coffee Break 10:30 – 10:50 “Intelligence Level Performance Standards Research for Autonomous Vehicles” - Roger Bostelman, Tsai Hong, and Elena Messina. 10:50 – 11:10 “Lessons from the Design and Testing of a Novel Spring Powered Passive Robot Joint” - Joel Stephen Short, Aun Neow Poo, Chow Yin Lai, Pey Yuen Tao, and Marcelo H Ang Jr. 11:10 – 11:30 “Improvements and considerations related to human-robot interaction in the design of a new version of the robotic head Muecas” - Felipe Cid Burgos, Pedro Núñez Trujillo and Luis J. Manso. 11:30 – 11:50 “Adapting Low-Cost Platforms for Robotics Research” - Thommen Karimpanal George, Mohammadreza Chamanbaz, Abhishek Gupta, Wen Lizheng, Timothy Jeruzalski, and Erik Wilhelm. 11:50 – 12:10 “Using Autonomous Robots to Diagnose Wireless Connectivity” - Richard Wang, Manuela Veloso, and Srinivasan Seshan 12:10 – 14:10 Lunch time 14:10 – 15:10 Keynote: The communication accommodation of the machine to deal with errors in Human-Robot Spoken Interaction Laurence DeVillers (LIMSI-CNRS, Sorbonne University, France) 15:10 – 15:30 “Toward Soft, Robust Robots for Children with Autism Spectrum Disorder” - Hong Tuan Teo, and John-John Cabibihan. 15:30 – 16:00 Coffee Break 16:00 – 16:20 “Gualzru's path to the Advertisement World” - Fernando Fernández, Moisés Martínez, Ismael García-Varea, Jesús Martínez-Gómez, Jose Pérez-Lorenzo, Raquel Viciana, Pablo Bustos, Luis Manso, Luis Calderita, Marco Gutiérrez, Pedro Núñez, Antonio Bandera, Adrián Romero-Garcés, Juan Bandera and Rebeca Marfil. 16:20 – 16:40 “Design, Simulation and Implementation of a 3-PUU Parallel Mechanism for a Page IX The Path to Success: Failures in Real Robots (FinE-R) Macro/mini Manipulator” - Zheng Ma, Aun-Neow Poo, Marcelo Ang, Geok-Soon Hong and Feng Huo. 16:40 – 17:00 “Skill-based Exception Handling and Error Recovery for Collaborative Industrial Robots”- Anders Billesø Beck, Anders Due Schwartz, Andreas R. Fugl, Martin Naumann, and Björn Kahl. 17:00 – 17:10 Closing Ceremony Page X The path to success: Failures in rEal Robots (FinE-R) Luis Fernando D’Haro1, Member, IEEE, Andreea I. Niculescu1, Aravindkumar Vijayalingam2, Marco A. Gutierrez1,3 Member, IEEE, Suraj Nair2, Rafael E. Banchs1, Member IEEE  Abstract— This paper presents our motivation for organizing As many other sciences, the path to progress in the field of the FinE-R workshop at IROS 2015, as well as a summary of all robotics is not free of failures and caveats. These failures accepted papers. The main workshop goal is to provide an open provide valuable lessons and insights on future approaches by exchange forum to the robotic community where participants analyzing errors and finding methods to avoid them. As such, can share their personal “failure to success” stories. We believe the robotics community could benefit from the experience of that such exchanges are of tremendous importance for the those who had faced and overcome similar failures before. community as they provide a rich source of knowledge on how to avoid future mistakes with possible high impact. On the other The objective of this workshop is then to provide an hand, the papers accepted in the workshop give a good overview international forum for researchers in robotics and its related of different types of errors encountered in the robotic fields. fields, where they can share their personal experiences on Through deep analysis and clear description of failures, the their "failure to success" stories, to present what they have authors of these papers contribute to a learning process by learnt, what others should avoid while experimenting in extracting positive experiences and conclusions from negative similar context, and providing tips for better research results leading ultimately to success. practices and for creating more successful robots that meet people's expectations. Keywords: Workshop goals, summary of accepted papers, failure analysis. II. MOTIVATION FOR THE WORKSHOP Nowadays, in the scientific community only successful I. INTRODUCTION theories and positive results have a chance of being regarded as true, and then published in prestigious publications, Along the history there have been many important discarding odd and unexpected findings. However, the discoveries that resulted from long trials and error processes, success of these theories does not warrant that they are truth like the ones done for the creation of the electric light bulb by neither prove their adequacy to realism. Unfortunately, the Tomas A. Edison [1] (who is believed to have made current scientific publishing system privileges “successful” thousands of experiments before successfully creating the results as it is expected that their research findings will be in incandescent lamp). Similarly, other important discoveries alignment with well-established literature or with expected came out from analyzing 'failed' results as, for instance, the outcomes. However, as pointed by [3], research is a “voyage famous Michelson-Morley experiment in the late 1880’s, of discovery”, which is subject to unpredictability and designed to enhance the accuracy of the prevalent Aether theory. In this case, their efforts to advance the theory led to a fallibility, therefore science evolves according to testability, continual rejection of their research hypotheses. However, which might result in refutations or confirmations, as well on their null results were published in [2] and later played an the absence of anticipated correlations or in failed results, but important role in inspiring new experiments and paradigms, in any case, it should be clear that both kind of results like the special theory of relativity proposed by Albert contribute to the advance of the science. Einstein in 1905. In each case, the key point for the final However, ignoring the huge amount of information that success and contribution to the science was the willingness of negative results can provide (which, according to [4], are the researchers to learn from previous mistakes and to share statistically more trustworthy than positive data) is the gained experience with the scientific community. troublesome. Firstly, because by doing so, an important bias in the scientific publications is created since only certain pieces of information are provided. Regrettably, this tendency is yearly increased as pointed by [5], whom after analyzing 1 Luis Fernando D’Haro, Andreea I. Niculescu and Rafael Banchs work over 4,600 papers published in different disciplines between at the Human Language Techonology Group in the Institute for Infocomm 1990 and 2007 found that the proportion of published Research (I2R - A*STAR). 1 Fusionopolis Way, #21-01 Connexis (South negative results dropped from 30% to 14% between 1990 and Tower), Singapore 138632. (emails: {luisdhe, aandrea-n, 2007, and with significant differences between disciplines and rembanchs}@i2r.a-star.edu.sg). 2 Aravindkumar Vijayalingam and Suraj Nair work at TUM-Create, 1 countries. Secondly, this tendency of omitting information Create Way, 8th Floor, Singapore 138602. (emails: {aravind.v, can cause a huge waste of time and resources, as other suraj.nair}@tum-create.edu.sg) scientists considering similar questions may perform the same 3 Marco Antonio Gutierrez is PhD student at the Robotics Laboratory experiments; besides, this can also delay the development of (Robolab), Computer and Communication Technology Dept in the new ideas inspired on the ‘unsuccessful’ results. Finally, as University of Extremadura, Spain. Polytechnic School, University of pointed by [6], this problem is increased by the misconception Extremadura Avda. de la Universidad s/n 10003 Cáceres-Spain. (email: marcog@unex.es). The author conducted this work as part of his A*STAR that publishing negative results might harm scientists’ Research Attachment Programme (ARAP) at the Human Language reputations or, furthermore, it might give the perception that a Technology Department of Institute for Infocomm Research, Singapore. project was poorly designed and the researchers were either FinE-R 2015 Page 1 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 unknowledgeable about the subject or incapable of tailoring  Description of alternatives to techniques and algorithms more robust research hypotheses. To make things worse, that are prone to fail. some scientists will not report negative results just to avoid their papers to be rejected by the peer-reviewers, who could  Presentation of keys for successful research projects and give priority to other studies with “successful” results or that proposals on robotics. follow a more popular theory or approach.  Analysis of failed results and projects when using smart Fortunately, the scientific community is becoming aware algorithms, well-established techniques or brilliant that negative results are not meaningless and that there is a designs. potential value in sharing also negative results and discussing These proposed topics not only were in line with the idea the lessons learnt after analyzing the failures, as well as in of learning from failures, that is central to our workshop, but explaining what were the keys to avoid problems and achieve also allowed to differentiate FinE-R from other workshops successful results. Some examples of this tendency can be that are mainly centered on specific and vertical topics or seem in the New Negatives in Plant journal4 that according to areas of research. With FinE-R we aim at providing a space their scope is “an open access, peer reviewed, online journal for sharing practices and experiences of robot design and that publishes hypothesis-driven, scientifically sound studies construction across multiple disciplines, therefore making the describing unexpected, controversial, dissenting, and/or null workshop more interesting and open to a wider audience. (negative) results in basic plant sciences. The journal also consider studies that validate controversial results or results Finally, it is worth mentioning that it was gratifying for us that cannot reproduce previously published data”, or in the to read comments from reviewers of the Workshop proposal new approach supported by the Wealth Health Organization about the appropriateness and timelines of an initiative such (WHO)5 that has a new policy of publishing, in their peer as FinE-R. Some examples of these are: reviewed journal, results of clinical trials that include also “This is a very interesting proposal as learning from negative findings. failure in real-world applications is an important and Following such examples and taking into account that the essential capability for robots. This is not a topic not well scientist community working in the robotics field can also addressed so far. It is very good to see a group of people benefit of following a similar approach, we decided to discussing this” propose FinE-R (Failure in Real Robots), a workshop in the “This workshop will provide such a unique opportunity context of IROS6 (IEEE/RSJ International Conference on that we can learn from not only our own failure but also Intelligent Robots and Systems) conference. For this, we others. We surely need such a workshop. Topics cover wide decided not only to focus on presenting the negative results ranges. Speakers are from well-known organizations. Suggest obtained while working on real robots, but also on how the leaving more time for discussions.” researchers were able to extract meaningful lessons from their failures and what kind of solutions they proposed to finally III. SUMMARY OF CONTRIBUTIONS overcome their problems. Then, we made the FinE-R’s call for papers targeting at the following topics: In this section we summarize the accepted contributions to the first edition of FinE-R. All submissions went through a  Analysis of failures when participating in robotic single blind review process. In average, all papers received challenges. three reviews.  Design of robust human-computer interfaces for robots. A. Skill-based Exception Handling and Error Recovery for Collaborative Industrial Robots  Description of problems and solutions faced when failure is not an option, therefore there is the need of creating an Written by Billesø et al [7], this paper discusses the outstanding robot from hardware to software. problem of error handling and recovery in the context of open human workspaces. The authors propose a skill-based  Description of benchmarking and tools for testing and exception handling and error recovery approach that allows creating robust robots. non-robot expert users to operate a robotic system in open environment where other human co-workers are present. The  Description of techniques to avoid common but paper presents the skill-based execution model and describes frequently seen errors when deploying robots for the situation assessment module which learns and monitors industrial or general public environments. the skill execution. Further, the authors show in details how  Description of advanced techniques for failure recovery their exception handler model based on a hierarchical four and troubleshooting. layered Bayesian network works. Non-expert users can accept or reject a solution of an error handling strategy using a  Matching the expectations and needs of industries and simple GUI. The user preference is learned by the system for consumers with the current technology. future re-use. B. Using Autonomous Robots to Diagnose Wireless Connectivity 4 http://www.journals.elsevier.com/new-negatives-in-plant- This paper, written by Wang et al [8], presents a science method/system for diagnosis of wireless connectivity issues 5 http://www.who.int/ictrp/results/reporting/en/ through the use of autonomous robots within the author's 6 http://www.iros2015.org building infrastructure. The proposed study and solution is of FinE-R 2015 Page 2 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 interest for most robotic laboratories when dealing with G. Design, Simulation and Implementation of a 3-PUU wireless connectivity problems. The authors claim that using Parallel Mechanism for a Macro/mini Manipulator this method they were able to improve the diagnosis of In [13], Zheng et al. present the design of a 3-PUU wireless connectivity issues as compared to manual methods. parallel mechanism which is used as a mini manipulator in a C. Soft, Robust Robots for Children with Autism Spectrum macro/mini manipulator configuration. The mechanism is Disorder suitable for applications requiring precision force control. The paper describes the shortcomings in the initial attempt to In this paper, Hong Tuan and Cabibihan [9] describe an design the system and further discusses new methods and experimental comparison between polyester resin and silicone strategies adopted by the authors to overcome these rubber as casing materials for protecting robotic circuitry and deficiencies. The mechanism is a parallel kinematic servomechanism. The main motivation of the work is on mechanism for pure translation motion of the end effector enhancing physical robustness of social robots when used for platform. This is achieved through three prismatic actuators therapeutic purposes, more specifically in interaction with and three universal joints. The authors faced difficulties in children suffering autism spectrum disorder. achieving pure translation motion at the end effector and they D. Adapting Low-Cost Platforms for Robotics Research successfully trace the source of the problems to be In this paper [10], Karimpanal et al. explain the design mathematical singularities and irregularities in the process of EvoBot, a low-cost, open source, general purpose construction of the universal joints purchased off the shelf. platform to enable testing and validation of robotics The authors further demonstrate how they learn from the algorithms. It has a differential base with two powered wheels initial attempt failures and device a new parallelogram based and two casters. It includes Bluetooth, a Wi-Fi enabled configuration for the universal joint mechanism in order to camera and several sensors. The paper describes specially the reduce backlash. design process and solutions of low-cost platforms for swarm H. Intelligence Level Performance Standards Research for robotics research, as well as the adaptation process of swarm Autonomous Vehicles robotics algorithms from simulation to real scenarios. The In this paper [14], written by Bostelman et al, the authors lessons learned when designing and adapting the robot are discuss standards development for performance of also discussed. Finally, the paper addresses how to adapt Autonomous Guided Vehicles (AGV) and optical some common representative tasks for the platform, along measurement systems that are used to measure such vehicle with some potential problems and possible solutions. performance. The paper discusses benchmarking standards E. Improvements and considerations related to human- for AGV and the issues faced with developing such a robot interaction in the design of a new version of the robotic standard. The paper focuses on standards in four areas. head Muecas Firstly, standards for vehicle navigation in order to measure uncertainties in navigation performance are detailed as In this paper [11], Felipe Cid and Pedro Núñez describe currently this information isn't provided by the manufacturers. some design improvements for a robotic head called Secondly, standards to determine uncertainties in vehicle “Muecas”. These improvements include both actuators and docking by measuring relative displacement from each of the sensors aimed at providing the system with better points are described. Thirdly, standards for obstacle detection communication capabilities for an enhanced human-robot and avoidance are presented to study the reaction of AGV in interaction. The authors support their design decisions on different situations such as when a human is detected and some psychological theories based on emotional and interaction with machines that are operated manually. And communicational phenomena. The paper focuses on finally, standards for 6DOF optical measurement of dynamic incremental design cycles for improving existent robotic systems are discussed as these systems are needed for platforms by incorporating new features and functions based performing ground truth measurements of AGV performance. on the lessons learned from the past. Experiments carried out for vehicle navigation, vehicle F. Lessons from the Design and Testing of a Novel Spring docking and optical measurement systems standards are also Powered Passive Robot Joint presented. This article [12], written by Short et al, narrates the I. Gualzru's path to the Advertisement World researchers’ journey towards the design, building and testing Presented by Fernández et al [15], in this paper the of a torsional spring joint. It focuses on the problems authors describes the genesis of Gualzru, a 1.60 m robot with encountered during this process, as well as the lessons learned an external cover built of resin and fiber glass, and a for the future. differential base with two powered wheels and two casters. It One problem engineers are often dealing with is the short is commissioned by a large Spanish technological company to time schedule they have to make certain assumptions and provide advertisements in open public spaces. The lessons estimations. This can often lead to troubles in the assembly learned during the three years of development from different and testing phase. As such, the spring joint prototype points of view are explained including hardware, software, designed by the authors went twice through a cycle of architectural decisions and team collaboration issues. assembly, testing, and redesign before the arriving at the final stage. During this process, the authors mention that they IV. CONCLUSION AND FUTURE WORK identified three problems and reported five learned lessons With this first edition of the FinE-R (Failure in Real from their design experience. Robots) workshop we pretend to open a door for researchers FinE-R 2015 Page 3 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 to address the analysis and discussion of failures and Workshop, pp. 15-19. IROS 2015 - Hamburg, Germany. October 2, methodologies when creating or designing robots. The 2015. [10] Thommen Karimpanal George, Mohammadreza Chamanbaz, Abhishek workshop allows for sharing research experiences with Gupta, Wen Lizheng, Timothy Jeruzalski and Erik Wilhelm. scientists facing similar situations and problems. In this paper “Adapting Low-Cost Platforms for Robotics Research,” Proceedings we also have provided a summary of the accepted FinE-R Workshop, pp.20-26. IROS 2015 - Hamburg, Germany. contributions, in which the authors were asked to describe October 2, 2015. their path to success roadmap and to provide clear [11] Felipe Cid Burgos and Pedro Núñez Trujillo. “Improvements and considerations related to human-robot interaction in the design of a explanations of what they learnt while deploying their robotic new version of the robotic head Muecas,” Proceedings FinE-R projects that could be of interest for other researchers working Workshop, pp. 27-35. IROS 2015 - Hamburg, Germany. October 2, in the same area. 2015. [12] Joel Stephen Short, Aun Neow Poo, Chow Yin Lai, Pey Yuen Tao and Taking into account the quality of the accepted papers, the Marcelo H Ang Jr. “Lessons from the Design and Testing of a Novel good response from the reviewers, program committee, and Spring Powered Passive Robot Joint,” Proceedings FinE-R Workshop, scientific community, as well as the importance that brings pp. 36-41. IROS 2015 - Hamburg, Germany. October 2, 2015. doing a deep analysis not only on the successful results but [13] Zheng Ma, Aun-Neow Poo, Marcelo Ang, Geok-Soon Hong and Feng Huo. “Design, Simulation and Implementation of a 3-PUU Parallel also on the path followed to reach them, as future work, we Mechanism for a Macro/mini Manipulator,” Proceedings FinE-R plan to continue organizing FinE-R in the context of IROS Workshop, pp. 42-47. IROS 2015 - Hamburg, Germany. October 2, conferences. Our desire is that by keeping open this forum, 2015. the expertise of worldwide researchers gained along several [14] Roger Bostelman, Tsai Hong and Elena Messina. “Intelligence Level years of working on robotic projects can be share with the Performance Standards Research for Autonomous Vehicles,” scientific community. By doing so, not only better research Proceedings FinE-R Workshop, pp. 48-54. IROS 2015 - Hamburg, Germany. October 2, 2015. projects can be conducted, but specially common or subtle failures can be avoided. In addition, we plan to open a special [15] Fernando Fernández, Moisés Martínez, Ismael García-Varea, Jesús Martínez-Gómez, Jose Pérez-Lorenzo, Raquel Viciana, Pablo Bustos, session or discussion panel where people participating on Luis Manso, Luis Calderita, Marco Gutiérrez, Pedro Núñez, Antonio shared tasks or competitions like the DARPA Robotics Bandera, Adrián Romero-Garcés, Juan Bandera and Rebeca Marfil. Challenge7, can explaining their experiences and problems “Gualzru's path to the Advertisement World,” Proceedings FinE-R encountered. Workshop, pp. 55-65. IROS 2015 - Hamburg, Germany. October 2, 2015.. ACKNOWLEDGMENT We want to thank to all the members of the program committee who helped during the review process and during the organization of the workshop. The full list of contributors and program committee members can be found at the workshop website8. REFERENCES [1] Thomas Alva Edison, “Electric lamp” U.S. Patent 0,223,898. Published Jan 27, 1880. [2] Michelson, Albert A., and Edward W. Morley. "On the Relative Motion of the Earth and of the Luminiferous Ether." Sidereal Messenger, vol. 6, pp. 306-310 6 (1887): 306-310. [3] Popper, Karl. "Science: Conjectures and refutations." (1953): pp-33. [4] Unreliable research: Trouble at the lab. The Economist, Oct 19th 2013. Available at http://www.economist.com/news/briefing/21588057- scientists-think-science-self-correcting-alarming-degree-it-not-trouble. [Seen on August 2015] [5] Fanelli, Daniele. "Negative results are disappearing from most disciplines and countries." Scientometrics 90.3 (2011): 891-904. [6] Curty, Renata G., and Jian Tang. "Someone's loss might be your gain: A case of negative results publications in science." Proceedings of the American Society for Information Science and Technology 49.1 (2012): 1-4. [7] Anders Billesø Beck, Anders Due Schwartz, Andreas Rune Fugl, Martin Naumann and Björn Kahl. “Skill-based Exception Handling and Error Recovery for Collaborative Industrial Robots”. Proceedings FinE-R Workshop, pp. 5-10. IROS 2015 - Hamburg, Germany. October 2, 2015. [8] Richard Wang, Manuela Veloso and Srinivasan Seshan. “Using Autonomous Robots to Diagnose Wireless Connectivity,” Proceedings FinE-R Workshop, pp.11-14. IROS 2015 - Hamburg, Germany. October 2, 2015. [9] Hong Tuan Teo and John-John Cabibihan. “Soft, Robust Robots for Children with Autism Spectrum Disorder,” Proceedings FinE-R 7 http://www.theroboticschallenge.org/ 8 http://finer-iros2015.appspot.com/ FinE-R 2015 Page 4 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Skill-based Exception Handling and Error Recovery for Collaborative Industrial Robots A. B. Beck, A. D. Schwartz, A. R. Fugl, M. Naumann, B. Kahl  Abstract— Moving robots from their carefully designed and recovery approach that allows non-robot expert users to encapsulated work cells into the open, less structured human operate a robotic system embedded in a human-centric workspace for collaboration with workers requires robust workspace. We briefly introduce our execution model, detail error detection and recovery strategies. Foreseeing all possible the Extended Markov Chain based Situation Awareness, uncertainties and unexpected events and to program in which forms the base for Exception Handling, and the Error recovery actions at setup time is unfeasible. Online learning of Recovery module employing a Bayesian network and beta- nominal execution behaviour and automatic detection of binomial inference algorithm. The prosed system has been anomalies using an Extended Markov Model, combined with implemented in a pick & place and in an assembly work cell, interactively trained Bayesian networks for mapping which are finally presented. anomalies to error causes and recovery actions, enables automatic recovery from previously experienced errors. A II. RELATED WORK three-layered user-friendly model of errors—causes— responses and a simple GUI allows non-expert user to define Research in exception handling is related to the area of error new recovery activities and error causes when not yet handled or fault recovery [17]. Error recovery has been defined as anomalies occur. “the process by which the system returns to a state where production can restart after an abnormal and disruptive I. MOTIVATION condition has occurred” [23]. For a robot coworker to Today’s robot systems for industrial applications rely on effectively handle an exception, whether through informing a structured environment to avoid errors. Parts, fixtures, the human worker or resolving the problem by itself, the tools and stations have defined positions and the workspace types of faults that typically occur in the manufacturing is encapsulated to avoid intruders that could possibly robotic assembly cases needs to be understood. Fault endanger this defined environment. Expected exceptions taxonomies have been presented in other related fields, from the nominal case that were either foreseen during the including mobile robots [7], computing [3], autonomous planning of the robot system, or occurred during the setup robots in RoboCup [21], workflow systems [16], service- phase of the system are coped with by integrating additional oriented architecture [6], and web service [8]. Reports show sensors, adapting tool-, fixture and part geometries and that many errors in manufacturing systems, including CNC adding additional branches to the robot program to cope with machines, are hardware related and that approximately 60% these deviations. Furthermore, as many robotic systems are of all stoppages are due to tool breakdown [23]. However, complicated, any exceptions and breakdowns occurring after system setup often require external technicians or engineers there has been a lack of study on the likelihood of common to diagnose and solve problems. errors and exceptions occurring during assembly tasks involving collaborative robots. One of the reasons can be Such strictly controlled and carefully designed work that robot coworkers have not yet proven to be robust cells are only economically feasible if the designed robot enough for industry application to be studied and system will run unobstructed for a long time. Small and mid- generalized based on real assembly cases [14]. sized enterprises (SMEs) are often characterized by a much more agile production style and consequently rely on human III. SKILL-BASED EXECUTION MODEL workspaces. Moving robots out of their strictly controlled At the base of the system is a Skill Execution Engine, and carefully designed spaces into human workspaces, which allows a more goal-oriented task description than which are by nature unstructured environments with a high strict motion based programming or planning. Without degree of uncertainty, requires significantly enhanced going into details of the skill-model [1], we assume skills to robustness towards unforeseen events and geometric or be independent, sensor-based motion or handling primitives other uncertainties. (The additional need for safety measures that adapt themselves to position uncertainties and other to protect the human co-worker from injuries is out of scope deviations from an ideal state using build-in sensing and of this work, see e.g. [22], [12] and many others.) A SME monitoring as well as (limited) internal error recovery. suitable robot system therefore needs semi automatic Robot tasks are constructed by chaining skills and control exception handling and error recovery capabilities that allow flow instructions, forming a state machine [2] based on non-expert users to manage exceptions (internally and SCXML1. While skills detect deviations from their expected externally triggered) occurring in daily operation. We performance and report these, the skill executor by itself propose a novel skill-based exception handling and error does not provide any error recovery functionality. Features * The research leading to these results has been funded by the European N. Naumann is with the Fraunhofer Institute for Production Systems Union’s seventh framework program (FP7/2007-2013) under grant and Automation. E-mail: Martin.Naumann@ipa.fraunhofer.de agreements #608604 (LIAA: Lean Intelligent Assembly Automation) and B. Kahl is with the Gesellschaft für Produktionssysteme GmbH #287787 (SMErobotics: The European Robotics Initiative for Stuttgart. E-mail: bjoern.kahl@gps-stuttgart.de Strengthening the Competitiveness of SMEs in Manufacturing by 1 integrating aspects of cognitive systems). Apache Commons SCXML executor, A. B. Beck and A. R. Fugl are with the Danish Technology Institute. E- http://commons.apache.org/proper/commons-scxml/. mail: anbb@dti.dk and arf@dti.dk FinE-R 2015 Page 5 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 of the skill executor that allow the implementation of error A. Situation Model recovery functionality at higher layers are: Situation Assessment uses a Situation Model as a  The skill executor knows and publishes the current template description to fuse together the different data points state of the system at any time. This allows an error for learning a Situation. The components of the Situation recovery module to relate errors on the one hand to Model (𝑑𝑖 in (2)) are real number data, which can come from specific skill models and on the other hand to any source and have any meaning. In our experience, specific application steps and therefore to draw combining space and time is critical to the success of conclusions like “this is an error that is very typical learning a skill. For instance, learning a skill using a 6D F/T for a pick operation” or “this is an error that sensor, the Situation Model 𝑠 could be defined as in (3). occurred already in the past at this specific 𝑠𝑎 = [𝑝𝑟𝑖𝑚𝑖𝑡𝑖𝑣𝑒, 𝐹x , 𝐹y , 𝐹𝑧 , 𝑇𝑥 , 𝑇𝑦 , 𝑇𝑧] (3) execution step of the application”. The component 𝑝𝑟𝑖𝑚𝑖𝑡𝑖𝑣𝑒 of 𝑠a is a data point that  The skill executor has an interface for an error uniquely identifies the current primitive being executed in recovery module to stop and later continue the the skill. In this case, the unique primitive ID provides the execution of the skill based application program understanding of time while the understanding of space is thereby allowing worker interaction to recover from provided by the F/T data. By using the primitive ID we can errors detected by an error recovery module. learn a skill time invariantly. This means that SA will only  The skill formalism used by the skill executor is learn the sequence of the events and is invariant towards the built on the concept of reusable hierarchical skills duration of the execution of specific primitives. We have that are easy to enhance or adapt. It is therefore found this feature particularly useful when the duration of easily possible to include additional mechanisms the primitives or skills is stochastic. Should it be necessary into an existing skill model to cope with errors that to catch anomalies in relation to when events occur (e.g. too could be detected by the system but just have not early or late), the primitive ID in (3) can be substituted with been considered yet. a time data point. Throughout our research, we have successfully applied SA to monitoring digital inputs, such as The Situation Assessment (SA) constantly monitors the the state of one or more grippers. Through the rest of the overall situation (robot task execution) using data published paper, we will use the following Situation Model for by the skills executor as well as by additional sensors implementation and testing: dedicated for situation assessment. Deviations flagged by the SA are further examined by the Exception Handling 𝑠𝑏 = [𝑝𝑟𝑖𝑚𝑖𝑡𝑖𝑣𝑒, 𝑔𝑟𝑖𝑝𝑝𝑒𝑟𝑜𝑝en, 𝑔𝑟𝑖𝑝𝑝𝑒𝑟𝑐𝑙𝑜𝑠𝑒𝑑] (4) (EH), which devises a possible cause and corrective where 𝑔𝑟𝑖𝑝𝑝𝑒𝑟𝑜𝑝𝑒𝑛 and 𝑔𝑟𝑖𝑝𝑝𝑒𝑟𝑐𝑙𝑜𝑠𝑒𝑑 are binary outputs of measure, potentially involving user interaction. The whole reed switches of the gripper: 𝑔𝑟𝑖𝑝𝑝𝑒𝑟𝑜𝑝𝑒𝑛 is 𝑡𝑟𝑢𝑒 when the system of skill executer, situation assessment and exception gripper is fully open and 𝑔𝑟𝑖𝑝𝑝𝑒𝑟𝑐𝑙𝑜𝑠𝑒𝑑 is 𝑡𝑟𝑢𝑒 when the handling is collectively referred to as “Exception Handling gripper is fully closed. Our assumption is that the gripper is Framework” or “EHF”. grasping an object when both readings are 𝑓𝑎𝑙𝑠𝑒, indicating IV. SITUATION ASSESSMENT the gripper is neither fully open nor closed. The role of Situation Assessment (SA) is to learn and B. Data Processing and Clustering monitor the (correct) skill execution and detect non-nominal The Situation Model serves as a template describing conditions. Deviations from the learned, nominal behaviour which sensors SA should fuse together into one single state. are interpreted as Anomalies, which are passed on to the In general, all data points in the Situation Model have to be Exception Handler (section V). Our implementation of SA real numbers. This allows the computation of one single is based on prior work by [4] and [5], where SA was applied metric for each 𝑋𝑖 in (1). We have so far used the Jaccard to mobile robotics. We implemented and expanded SA to similarity coefficient as a method for clustering similar learn skill based execution in a collaborative robotic system. states. Through experimentation, we have found the To learn how to perform a skill correctly, SA captures the algorithm to be useful despite its simplicity. essence of the skill by learning the timing and sequence of events that make up the skill. Our approach is to generate C. Dynamic Learning in Situation Assessment one parameterized model that includes parameters in the SA can autonomously learn a skill without the user space and time domain. SA learns the sequence of events having to manually specify the states of a skill. We have within a skill execution by learning a set of parameters with implemented a spatiotemporal model that allows for online a temporal component, recording the transition from one dynamic learning of states over time. For this purpose, we instantiation of the parameters to the next: are currently using the Extensible Markov Model (EMM) as it is useful for online learning of sequences of states [10]. An p(𝑋) = 𝑝(𝑋1, 𝑋2, … , 𝑋𝑛) (1) example of a dynamically learned model using the EMM where 𝑋, a Situation Model, denotes a set of parameters (a algorithm can be seen in Figure 1. In this example, a robot is state), 𝑛 denotes a discrete step in time, and 𝑝, a Situation, picking up a nut from a table and placing it on a pipe in a denotes the complete distribution of all the states within a single nonrecurring operation (therefore an open-ended skill. Each state 𝑋𝑖 of 𝑋 is parameterized: chain). The EMM is also useful in learning looped tasks. 𝑋 = [𝑑1, 𝑑2, … , 𝑑𝑚] (2) where 𝑑𝑖 are data components such as sensor values or robot’s internal state values. FinE-R 2015 Page 6 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 an error. The Fault node models the root cause of the Anomaly and the Response node models the solution to the Fault. This model resembles the diagnosis model used by physicians when examining a patient: Based on symptoms (here: the detected error) an illness is inferred (here the fault) and a therapy decided (here the response). The intermediate step of a fault is necessary, since one and the same observed error (symptom) can have multiple causes. For example an unexpected gripper state can be due to a failed grasping operation, a missing object at the pickup position or a defective gripper itself. B. Bayesian Network Figure 1. Example of learning a task. The upper left corner shows the temporal sequence of states the skill consists of. These states were learned online while the robot performed the task. A full video is available at https://www.youtube.com/watch?v=-CKbdQ3ocQo. D. Anomaly detection SA has two modes of operation: learning and detection during execution. In the learning mode, SA monitors the data points specified in the Situation Model and builds the Situation for the skill that is being learned. During execution of the same skill, SA loads the saved Situation and applies the same clustering process as during learning. However, should the clustering of the data result in a new state in (1), then SA will interpret that as an anomalous state has occurred and issue an Anomaly warning. Processing and handling the Anomaly is the task of the Exception Handler Figure. 2. Inference of cause and solution to Gripper Open anomaly. (EH) module. Both error nodes are dependent on both anomaly nodes, allowing the V. EXCEPTION HANDLER Bayesian network to further strengthen the belief about the cause of an anomaly. Simulated in GeNIe1. The task of EH is to receive an Anomaly from SA and provide a suggested solution that is most likely to solve the Figure 2 is an example of a Bayesian network with three problem. For each robotic system, EH maintains a Exception Scenarios for the two Anomaly nodes of the hierarchical four-layered Bayesian network with all sensors 𝑔𝑟𝑖𝑝𝑝𝑒𝑟𝑜𝑝𝑒𝑛 and 𝑔𝑟𝑖𝑝𝑝𝑒𝑟𝑐𝑙𝑜𝑠𝑒𝑑 in (4). Both Error exceptions and solutions relevant to that cell. The nodes are dependent on both Anomaly nodes, allowing the hierarchical structure allows EH to reason about the most Bayesian network to further strengthen the belief about the suitable solution to a problem. EH provides the suggested cause of an Anomaly (simulated in GeNIe2). The first two solution to the user along with all other possible solutions. scenarios with nodes 𝑠1 = {1,3,5,8} and 𝑠2 = {1,3,6,9}, offer The user is free to select the suggested solution, any other two Responses to the Gripper Open Anomaly while the third solution or to create a new solution. The selection is stored scenario 𝑠3 = {2,4,7,10}, offers a single Response to a in EH as a sample of user solution preference. Such samples Gripper Closed Anomaly. The numbers in curly braces are used in priming the network for inference with future indicate the node number in Figure 2. In this example we are anomalies. With the feedback of user samples, a closed modelling two faults {5,6} and Responses {8, 9} for the preference-learning loop is formed to provide suggestions Gripper Open Anomaly. If a gripper is unexpectedly open for solutions to future anomalies. In this section, we provide (Gripper Open = true, Gripper Closed = false), we could a detailed description of EH and begin with the role of the interpret that as either a pneumatics failure (e.g. loss of air Exception Scenario ES in EH. pressure) that can be solved by checking and replacing the A. Exception Scenario air supply {5,8}, or an actuator failure (e.g. broken gripper) The Exception Scenario (ES) is designed as a four- that can be solved by repairing the gripper {6,9}. In the layered model consisting of Anomaly, Error, Fault and reverse case of a closed gripper, we could interpret the Response, inspired by work in [18]. The hierarchy is a four- failure as there was no object to grip and the solution is layered binary Bayesian network that facilitates inferring the simply to replace the missing object. In Figure 2, the Faults most likely Response (solution) to an Anomaly (a {5,6} are modelled as belonging to the same Error, Gripper deviation), an example is shown in Figure 2. At the lowest Operations Error {3}. This allows the network to learn user level of the network, Anomaly nodes model anomalies selections for a specific Fault, Response pair over other pairs detected by SA. Each Anomaly node corresponds to a data belonging to the same Error node. The network is thereby component (di of (2)) in the Situation Model. Above able to encode knowledge specific to individual user Anomaly, the Error node models which kind of error the environments. Anomaly is and if the Anomaly should even be considered 2 Figure 2 shows a screen capture of GeNIe, a Bayesian modelling environment developed by the Decision Systems Laboratory of the University of Pittsburgh. Available at http://genie.sis.pitt.edu FinE-R 2015 Page 7 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 confirming and rejecting the selection of the node. The Beta distribution is a conjugate distribution to the Binomial C. Inference distribution, thereby offering analytical tractability of the The process of inferring a Response to an Anomaly in Bayesian inference process. The conjugate property ensures the Bayesian network, is the inference process of the EH. that when updating the prior Beta distribution (7) with new This process is an implementation of Bayes’ theorem: evidence following the Binomial distribution, the resulting posterior distribution is also a Beta distribution (8). 𝑝𝑜𝑠𝑡𝑒𝑟𝑖𝑜𝑟 ∝ 𝑝𝑟𝑖𝑜𝑟 ∙ 𝑙𝑖𝑘𝑒𝑙𝑖ℎ𝑜𝑜𝑑 (5) We have implemented Bayes’ theorem in three steps: (𝑝 | 𝛼∗, 𝛽∗) =Be(𝛼∗, 𝛽∗) (8) 𝛼∗ and 𝛽∗ is respectively the new number of selections  Calculate prior probabilities and rejections for the specific node. Thus, obtaining the  Introduce evidence to network posterior distribution in (8) becomes simply a matter of adding new confirmations to the existing, and then  Infer posterior probabilities calculating the mean (𝜇) and variance (𝑣𝑎𝑟) (9,10). In the following, we describe each of these steps. D. Inference A prerequisite for performing inference is the calculation of prior probabilities. As described in section IV, a feature of the EHF is to learn the user-preferred solution of a given In Figure 3, examples of Beta distributions for different anomaly. When the user selects a specific Exception values of 𝛼 and 𝛽 are shown. Distribution 1: 𝐵e(𝛼 = 1, 𝛽 = Scenario (i.e. an Error, a Cause and a Solution) to solve a 1) is a uniform distribution offering an uninformative prior problem, it is fed back to the database as a sample of the user with a mean, 𝜇 = 0.5 and a high variance (uncertainty) due selection, thus learning the preference of selecting this to the low sample size. In this case, the posterior will largely Exception Scenario for a specific Anomaly. The sample data be determined by the data. Distribution 4: 𝐵e(𝛼 = 30, 𝛽 = 5) is used to calculate the prior probability for each node of the has 𝜇 = 0.86 and a smaller variance, thus providing a network. We treat calculating the node’s prior probability as comparably less uncertain estimate of the user selection an inference process that adds another layer of Bayesian preference, 𝑝. The sequence of graphs 1-4 in Figure 3, can inference as described in (5). We introduce the sample data be seen as an example of a continuous learning cycle, from user selection of Exception Scenarios as the evidence starting with no knowledge of user selection (a uniform to infer each node’s posterior probability. Each node of the Bayesian network is a binary random variable modelling an event that either occurs or not. For instance, if the user selects the ES {1,3,5,8} in Fig. 2, then the user is confirming that the specific ES solved the problem (e.g. that a Gripper Open Anomaly did happen, it was caused by missing air pressure and the solution was to resupply the air). At the same time and equally important, the user is also confirming that alternative events {4,6} to ES {1,3,5,8} did not occur. Thus, with every selection of an ES, EH registers the confirmed nodes on all levels of the ES, as well as the rejected nodes. The process of selecting any node in the Bayesian network over time, can be viewed as a Bernoulli Figure 3. Four 𝐵𝑒(𝛼, 𝛽) distributions for different values of 𝛼, 𝛽. process following a binomial distribution as in (6). Note that 𝛼, 𝛽 > 0, thus 𝛼 = 𝛽 = 1 is equal to no samples. 1: 𝐵𝑒(1,1), 𝜇 = 0.50, 𝑣𝑎𝑟 = 0.083. 2: 𝐵𝑒(2,1), 𝜇 = 0.67, 𝑣𝑎𝑟 = 0.056. 3: 𝐵𝑒(15,5), 𝜇 𝑋~Binom(𝑛, 𝑝) (6) = 0.75, 𝑣𝑎𝑟 = 0.0089. 4: 𝐵𝑒(30,5), 𝜇 = 0.86, 𝑣𝑎𝑟 = 0.0034. where 𝑋 is the number of times a specific node has been selected. 𝑛 is the number of samples drawn in the sequence. distribution with no samples) towards more informative If this process is sampled sufficiently, a distribution distributions 2-4 as the sample size increases. When a new reflecting the user selection can be inferred from the sample node is created with no samples available (𝛼 = 𝛽 = 1), the set. However, in many cases it is not possible to provide a Beta distribution is uniform. However, to avoid the sample set of sufficient size and inference will be subject to uninformative uniform distribution we propose to query the uncertainty. To model this uncertainty, we model the user user to provide a subjective estimate of the selection (the selection for each node as a hyper-parameter 𝑝, thereby mean) of this node along with a confidence level (the modelling the user selection as a random variable itself and variance). Using the equations for the mean (9) and variance creating a hierarchical Bayesian model for calculating the (10), suitable values for 𝛼 and 𝛽 can then be calculated. prior probability [11]. This approach uses the samples of user selection as a likelihood function providing evidence to E. Introducing evidence the inference process. Given the binomial likelihood, we The Bayesian network described in section V.B and Fig. have chosen the Beta distribution as the prior distribution 2 receives evidence in the form of Anomaly information (7). gathered by SA. In the example shown in Figure 2, SA has detected that the gripper was unexpectedly fully open (thus (𝑝 | 𝛼, 𝛽) = Be(𝛼, 𝛽) (7) providing evidence that Gripper Open = true, Gripper In (7), the user selection is modelled as the Closed = false. The evidence is in practice introduced to the hyperparameter 𝑝, drawing samples from the Beta network by clamping the two nodes to their respective distribution. 𝛼 and 𝛽 is respectively the number of samples values. FinE-R 2015 Page 8 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 F. Posterior probabilities After introducing evidence, posterior probabilities for all nodes are calculated. We have used the SMILE reasoning engine [9] for inference. The Response having the highest posterior probability is selected as the suggested solution. For each Response, the tree is descended towards the root Anomaly nodes, thus mapping out each possible path towards the root. The resulting list will have the most probable ES listed first with all other less likely alternative ES following in descending probability. VI. USER INTERFACE FOR ERROR RECOVERY While section V discussed the inner working of the actual mapping process, we focus on a more user-centric Figure 5 Adding a new error cause or fault to the system. view in this section. Whenever an anomaly is detected, the error layer Anomaly. Hereafter, we introduced the GO Anomaly classifies it into an error cause. If no cause is found the user repeatedly, selecting the RA Response as the solution each is inquired and given the option to assign an existing cause, time. This process was repeated until EH started to suggest dismiss the anomaly as not indicating an error or to create a the RA Response, thus demonstrating EHF’s ability to learn new cause (including a resolution, if known). Figure 4 shows the user preference of selecting the RA Response over the the dialog box after successfully mapping an anomaly to an RP. error and further to a recovery action. The user can accept Test results are shown in Fig. 6. Initially, EH has five samples confirming the selection of the RP Response for the Figure 6. Posterior probabilities for solution nodes Replace Pneumatics (RP) and Repair Actuator (RA) to a Gripper Open Anomaly. The solid line represents the posterior probability of the Figure 4 The system identified an error including a recovery action. RP and the dotted line represents posterior probability of RA. For In case of misclassification the user can add a new exception or one sample of RP, it takes EH two samples of RA to learn the user dismiss the anomaly as not indication an error (button “Continue preference of selecting RA. Learning”). this solution or add a new solution. Figure 5 shows the GO Anomaly. Thus, when the GO Anomaly is introduced, corresponding dialog box for adding a new triplet of error, EH suggests RP as the most suitable Response to the GO error cause and recovery action. The dialog boxes shown in Anomaly with probability ~ 0.553. However, the user Fig. 4 and 5 are designed for use at system runtime and ignores the EH suggested RP Response and instead selects therefore as simplistic as possible. A more elaborated RA. Thus, when the GO Anomaly is introduced again, EH interface for managing the entire network of anomalies, now has six samples (five for RA and one for RP), errors, causes and recovery actions is also provided and computing the most likely Response to be RP with targeted at specifically trained users that setup a new robot probability ~ 0.545, and so on. At RA sample 5, EH application. computes the probability for each Response being identical (~ 0.526). Again, the GO Anomaly is introduced and this VII. EXPERIMENTAL EVALUATION time EH suggests the RA Response with posterior probability ~ 0.535. Thus, with five samples confirming RP, Within the scope of SMErobotics, this framework has it took six samples of RA for EH to suggest RA. been intensively evaluated using various experiments. A detailed example is the failure to grasp as described in the VIII. CONCLUSION AND FUTURE WORK following section. We have tested the system’s ability to learn the preference of selecting a solution by manually Through the test results in section VII, we showed that introducing the Gripper Open (GO) Anomaly, shown in Fig. EH is able to learn the user preference of selecting a solution, 6, during the execution of a skill. In this test, we have tested even when it had learned a different preference earlier. As the system’s ability to learn the user preference of selecting the user selects a specific solution to an Anomaly, the the Repair Actuator (RA) Response over the Replace solution becomes more probable for future selection. This is Pneumatics (RP) Response. For the purpose of the test, the normally helpful, but can be problematic if the user wishes system had initially no knowledge of user selections the system to select a different solution, since learning a new (samples), except for five samples confirming the choice of preference can take several iterations, as the test results the RP Response as the user preferred solution to the GO showed. This is especially true when the sample count for the prior solution is high. A possible future solution could be FinE-R 2015 Page 9 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 to introduce additional information in the Error Layer, e.g. [22] D. Stengel et al. "An Approach for Safe and Efficient Human- condition the error cause not only on the counting of user Robot Collaboration.", The 6th International Conference on Safety selections, but also on the state of various system variables of Industrial Automated Systems. 2010. [23] C. Syan, Y. Mostefai, "Status monitoring and error recovery in at time of the user selection. flexible manufacturing systems", Integrated Manufacturing The system is currently being integrated in further Systems, Vol. 6 Issue 4, pp.43 – 48, 1995 demonstrators in the context of the SMErobotics project and will see more in-depth testing and possibly enhancements in these demonstrators. Concept videos of these showing the SMErobotics vision of future industrial robotics are available at http://video.smerobotics.org; especially the D2 and D3 videos are relevant in the context of this work. REFERENCES [1] R. H. Andersen, “Definition of Hardware-Independent Robot Skills for Industrial Robotic Co-Workers”, ISR, 2014 [2] R. H. Andersen , L. Dalgaard , A. B. Beck, J. Hallam, “An Architecture for Efficient Reuse in Flexible Production Scenarios”, Accepted for IEEE Int. Conf. on Automation Science and Engineering (IEEE CASE 2015) [3] A. Avizienis, et al, “Basic concepts and taxonomy of dependable and secure computing,” IEEE Transactions on dependable and secure computing, vol. 1, no. 1, pp. 11–33, 2004. [4] A. B. Bech, “Situation Assessment for Mobile Robots”. PhD thesis, DTU Electrical Engineering, Danish Technological Institute, 2012. [5] A. B. Beck, C. Risager, N. A. Andersen, O. Ravn, "Spacio- Temporal Situation Assessment for Mobile Robots," in 14th International Conference on Information Fusion (FUSION), 2011. [6] S. Bruning, et al, "A Fault Taxonomy for Service-Oriented Architecture," High Assurance Systems Engineering Symposium, 10th IEEE , vol., no., pp.367,368, 14-16, Nov. 2007 [7] J. Carlson, R. R. Murphy, "How UGVs physically fail in the field," IEEE Transactions on Robotics, vol.21, no.3, pp.423,437, 2005 [8] K.S.M Chan, et al., “A Fault Taxonomy for Web Service Composition,” Service-Oriented Computing - ICSOC 2007 Workshops, Lecture Notes in Computer Science, pp 363-375, 2009 [9] J. M. Druzdzel, "SMILE: Structural Modeling, Inference, and Learning Engine and GeNIe: a development environment for graphical decision-theoretic models." AAAI/IAAI. 1999. [10] M. Dunham et al, ”Extensible Markov Model”. Fourth IEEE International Conference on Data Mining, 371-374, 2004 [11] N. Fenton, M. Neil, ”Risk Assessment and Decision Analysis with Bayesian Networks”, First edn., CRC Press, 2013 [12] T. Gecks, D. Henrich, "Human-robot cooperation: safe pick-and- place operations.", IEEE International Workshop on Robots and Human Interactive Communication. 2005. [13] S. Haddadin, et al. ”Towards the Robotic Co-Worker,” The 14th International Symposium ISRR, pp 261-282, 2011. [14] J. Huckaby, H. I. Christensen, “Toward a knowledge transfer framework for process abstraction in manufacturing robotics,” in ICML Workshop on Theoretically Grounded Transfer Learning, 2013. [15] C. Kemp, et al, "Challenges for robot manipulation in human environments [Grand Challenges of Robotics]," Robotics & Automation Magazine, IEEE , vol.14, no.1, March 2007 [16] M. Klein, C. Dellarocas; “Knowledge-based Approach to Handling Exceptions in Workflow Systems”, Computer Supported Cooperative Work, Volume 9, Issue 3-4, pp 399-412, 2000 [17] P. Loborg, “Error recovery in automation: An overview,” in Proc. AAAI., Stanford, CA, pp. 94–100, 1994 [18] J. Pearl, “Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference”, First edn, Morgan Kaufmann, 1998. [19] M. D. Schmill, et al, “The Role of Metacognition in Robust AI Systems,” AAAI-08 Workshop on Meta-reasoning, Chicago, 2008 [20] J. Shah, et al, “Improved human-robot team performance using chaski, a human-inspired plan execution system,” in Proceedings of the 6th international conference on Human-robot interaction, 2011. [21] G. Steinbauer, “A Survey about Faults of Robots Used in RoboCup,” RoboCup 2012: Robot Soccer World Cup XVI, Lecture Notes in Computer Science Volume, pp 344-355, 2013 FinE-R 2015 Page 10 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Using Autonomous Robots to Diagnose Wireless Connectivity Richard Wang, Manuela Veloso, and Srinivasan Seshan Carnegie Mellon University Due to the proliferation of wireless devices, many wireless struggle to resolve the wireless issues because: 1. they users treat wireless connectivity as a black box. When have limited time due to the large number of users to wireless performance does not meet expectations, it can administrators (25,000 to 6 in our case), 2. the problem must be a frustrating experience to try and resolve wireless be easy to replicate, and 3. network administrators control the issues. Wireless problems are more significant for mobile infrastructure APs but have limited visibility of the wireless robots due to strenuous requirements for sustained wireless medium. connectivity while moving [1]. Unfortunately, it can be Autonomous robots as a wireless tool can augment diag- difficult to understand the cause of wireless problems in real nosis of wireless problems by: environments. First, wireless signals transmitted across the 1) capturing fine-grain wireless maps reflecting actual wireless medium are susceptible to attenuation, interference, propagation of wireless signals and reflections from the surrounding environment and other 2) serving as a vehicle to subject wireless devices to wireless devices. Second, wireless connectivity depends on repeatable motions decentralized cooperation across heterogeneous devices. As This is made possible due to their ability continuously autonomous robots are introduced in our environments, localize with high accuracy and autonomously and precisely we believe they can be a perfect tool to capture detailed navigate without human assistance. Detailed wireless maps snapshots about our wireless environments to help diagnose help to reveal how the wireless medium is being used in wireless connectivity issues. In this paper, we show how order to eliminate unlikely causes of poor connectivity. They these insights helped us to diagnose our robot’s own motion- would also allow wireless users to diagnose simple dead zone based wireless connectivity issues. coverage issues and perhaps also empower them to create I. I NTRODUCTION more meaningful trouble tickets. Since wireless problems Understanding wireless connectivity in real environments with motion are often short-lived, the ability to reliably repeat is hard. Much of the complexity stems from wireless trans- motions is essential for understanding more complex motion- missions occurring over an open, shared medium with a based wireless connectivity issues. mixture of decentralized, heterogeneous devices [2]. Once In this paper, we will first show that autonomous robots devices begin to move, wireless problems become even more can be used to collect detailed wireless measurements. Next, difficult to diagnose since wireless conditions around the we show fine-grain insights allow us to better understand device can change rapidly. The emergence of telepresence how our wireless infrastructure uses the wireless medium. robots has shown that wireless devices in motion struggle to Finally, we show how we were able to diagnose our device’s sustain uninterrupted wireless connectivity [1]. In this paper, own motion-based wireless connectivity issues. we will show that autonomous robots can be a valuable tool II. I NSIGHTS ABOUT S URROUNDING W IRELESS for identifying the cause of poor wireless performance with C ONDITIONS direct observations of the wireless environment. We now show the detailed insights that autonomous robots We focus on enterprise wireless networks composed of can capture without access to any sensitive wireless in- access points (APs) distributed throughout the environment frastructure APs. With these insights, we will be able to to provide Internet access to devices at all locations. Today, understand how the wireless medium is being utilize and see motion-based wireless connectivity issues are difficult for if possible infrastructure configuration issues may be causing users to resolve because: our wireless connectivity issues. 1) wireless infrastructures are complex and vary over space and time A. AP Coverage 2) users have visibility and control over only their own AP coverage ensures every location has at least one AP device in range. Avoiding wireless dead zones is the responsi- 3) wireless communication problems can require signif- bility of network administrators who manage the wireless icant domain knowledge to deal with the range of infrastructure. They often try to place APs to provide a hardware, drivers, and protocol layers high minimum received signal strength indicator (RSSI) at As a result, a natural reaction is to submit trouble tickets every location. Our network administrators target a minimum and wait some time for network administrators to come RSSI of -60 dBm, which is much higher than -90 dBm that and resolve the problem. Even network administrators may generally signifies no connectivity. The process of verifying FinE-R 2015 Page 11 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 coverage simply requires sampling RSSI at all locations in the environment. Unfortunately, there are no practical solutions that require little human effort and achieve fine- grain sampling of the environment. As a result, there are situations where trouble tickets result in the discovery of wireless dead zones in practice. We can automate this search for wireless dead zones by deploying autonomous robots to measure coverage across the environment. We were able to cover four floors of our en- terprise environment. Figure 1 shows a histogram of median RSSI of the best available AP after dividing the environment into 1m x 1m grid regions. We see that AP coverage across two floors is very strong with few regions falling below the -60 dBm target. If there had been wireless dead zones, they would have been apparent in these histograms. As a result, wireless issues for these floors are unlikely to be due to wireless dead zones. Fig. 2: Median throughput across two floors. connectivity issues still persist. From our own empirical observations, these wireless issues appear intermittent and seemingly random while moving around. When we bring the robot back to revisit locations where it lost connectivity, the connectivity issues would not occur again so these problems must arise with motion. Autonomous robots will (a) Floor 1 (b) Floor 2 help to better understand these motion issues since they can Fig. 1: Coverage summary showing histograms of the best continuously collect of wireless performance measurements median RSSI for each floor. Network administrators typically while also reliably executing controlled motions. With the aim for a minimum of -60 dBm coverage. autonomous robots, we will methodically diagnose the root cause by enabling humans to search for similar patterns that lead to these poor connectivity situations. B. Throughput Samples A. Repeated Motions Coverage is an important pre-requisite for wireless con- Many factors including location and speed of motion can nectivity but not necessarily reflective of the actual rate cause variations in wireless performance so we subject the of data transmission. Unlike RSSI that are instantaneous wireless device to nearly identical situations. An autonomous measurements, throughput samples depend on state and co- robot itself is perfectly suited for subjecting the device to ordination with other wireless devices. Throughput tends to repeated traverals over the same path with the exact same vary more than RSSI since congestion and dropped packets speeds and device orientations. Deploying an autonomous is affect the rate of data transmission. As a result, throughput much preferred over fixed contraptions that are cumbersome maps are unliklye to be a predictable as the coverage maps. and require modifications to the environment [3]. Figure 2 shows throughput maps collect by the robot as The robot is instructed to follow a simple three-quarter it moved across the environment. These measurements show loop path around three hallways in the environment where how wireless performance varies over space. We can see connectivity issues occur frequently, as shown in Figure 3. that our robot’s own wireless connectivity problems are not We even instruct the robot to move in both clockwise isolated to small regions but spread across large regions of (Figure 3b and 3d) and counterclockwise (Figure 3a and 3c) our building. This points to more systemic wireless issues directions. We intentionally select a path where the robot that our robot is struggling with. If the robot was facing traverses each location at most once. With no overlapping region-specific wireless issues due to excessive congestion, measurements at any location, it will be much easier to these types of throughput maps would have been helpful. analyze the wireless performance variations using wireless maps. III. D IAGNOSING M OTION -BASED W IRELESS C ONNECTIVITY B. Analyzing Variations in Wireless Performance We have shown that our wireless infrastructure is well- While being driven along the given path, the wireless de- configured and AP coverage is not an issue. Nevertheless, vice simultaneously captures RSSI, throughput, and current our throughput maps showed that motion-based wireless AP it is associated with. We present four noteworthy runs FinE-R 2015 Page 12 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 (a) Run #1 RSSI (b) Run #2 RSSI (c) Run #3 RSSI (d) Run #4 RSSI (e) Run #1 Throughput (f) Run #2 Throughput (g) Run #3 Throughput (h) Run #4 Throughput Fig. 3: Simultaneous associated RSSI and throughput for 4 runs over the same locations. Runs 1 (3a) and 3 (3a) began in the bottom left corner with the robot moving counterclockwise while Runs 2 (3b) and 4 (3d) started in the top left and moved clockwise. Numbered labels reflect the first point of association with each AP while the shape and color reflect a unique AP whose corresponding coverage is shown in Figure 4. (a) Coverage AP A (b) Coverage AP B (c) Coverage AP C (d) Coverage AP D Fig. 4: Coverage for each AP corresponding to APs in Figure 3 identified with a unique shape and color. in Figure 3 that shows RSSI (top), throughput (middle), and around timing disassociations before connectivity degrade coverage for each AP (bottom). In the RSSI maps (top), significantly and then intelligently selecting the next AP to the unique shapes reflect the location where the device first switch to. With the help of our autonomous robot, we are associated with the corresponding AP as identified by the able to distinguish the effects of AP coverage, changing color and shape. The numbered labels identify the order in wireless conditions, and device motion to conclude that poor which they were visited. We show AP coverage for each AP handoffs are the cause of our robot’s wireless issues. of these uniquely identified APs (bottom). Corresponding throughput while moving (middle) is also shown where large IV. R ELATED W ORK stretches of white space reflect absence of connectivity. Past efforts to collect wireless measurements are unable These four runs provide some noteworthy insights. First, to ensure fine-grain accuracy, densely cover spatially diverse RSSI changes gradually over several meters as a function of areas, and provide timely updates. Unfortunately, it is diffi- the device’s distance from the AP. Notice that run #1 and #2 cult to predict the propagation of wireless signals in realistic, remained associated with the same AP for the duration of indoor environments so fine-grain wireless maps require the traversal. Irrespective of the direction of motion, RSSI measuring signals captured at each location. Measurement for these runs nearly perfectly matches corresponding AP studies have been performed by having humans carefully coverage. For these runs, throughput resulted in lengthy traverse a building and mark their locations on a map [4], [5]. stretches of no connectivity since the AP was out of range. This is a tedious process that suffers from accuracy issues In run #3 and #4, the device switches to another AP in due to human errors that make it undesirable to repeat often the middle of the path. This AP switch particularly benefits so it will be difficult to ensure maps are up-to-date. run #3 but not as much for run #4. The difference is that Dense deployments of static WiFi monitors can ensure run #3 switches APs just as it is about to enter the strongest timeliness but are limited by placement options for fixed AP coverage region for the selected AP. In contrast, run #4 location monitors and incur significant human effort and switches to an AP that is almost out of range. costs to deploy so typically they cannot achieve high spatial We can see in Figure 4 that there is at least one AP granularity. While some use dedicated sensor hardware [6], with high RSSI along the entire path so AP coverage is [7], [8], others reduce costs by adding WiFi dongles to strong. The motion-based challenges must stem from poor available USB slots [9]. Distributed synchronization and AP handoffs. The key challenges appear to be centered hardware calibration enables creation of a single, unified FinE-R 2015 Page 13 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 view from measurements collected across all WiFi monitors. R EFERENCES A global view can be used to infer aggregate performance [1] “Life telepresent: working vicariously through the beam metrics like number of active wireless clients, interference, robot,” ”http://www.theverge.com/2012/11/7/3611510/suitable- loss rates, and utlization [6], [7], [10] and even infer missing beam-robot-aims-for-bulletproof-telepresence”. [Online]. Available: ”http://www.theverge.com/2012/11/7/3611510/ packets [8]. These approaches are limited to the perspective suitable-beam-robot-aims-for-bulletproof-telepresence” of the wireless infrastructure and have difficulty accounting [2] D. Eckhardt and P. Steenkiste, “Measurement and analysis of the error for unreceived wireless client transmissions. In this paper, characteristics of an in-building wireless network,” in ACM SIGCOMM Computer communication review, vol. 26, no. 4. ACM, 1996, pp. we view the wireless network from the perspective of the 243–254. wireless client by accounting for the client’s movement and [3] R. Fish, M. Flickinger, and J. Lepreau, “Mobile emulab: A robotic considering the client’s intent of transmitting wireless data. wireless and sensor network testbed,” in IEEE INFOCOM, 2006. [4] P. Bahl and V. N. Padmanabhan, “Radar: An in-building rf-based Other efforts attempt to crowd-source collection of wire- user location and tracking system,” in INFOCOM 2000. Nineteenth less maps. These approaches end up sacrificing accuracy in Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE, vol. 2. Ieee, 2000, pp. 775–784. order to easily collect measurements. GPS can be used to [5] M. Youssef and A. Agrawala, “The horus wlan location determination provide location estimates [11], [12] but it operates primarily system,” in Proceedings of the 3rd international conference on Mobile in outdoor environment and suffers from poor location esti- systems, applications, and services. ACM, 2005, pp. 205–218. [6] Y.-C. Cheng, J. Bellardo, P. Benkö, A. C. Snoeren, G. M. Voelker, and mates of around 3 meters. FM signals [13] similarly suffer S. Savage, Jigsaw: solving the puzzle of enterprise 802.11 analysis. from the effects of indoor environments and cannot achieve ACM, 2006, vol. 36, no. 4. accurate location estimates. Recent efforts to take advantage [7] C. Reis, R. Mahajan, M. Rodrig, D. Wetherall, and J. Zahorjan, “Measurement-based models of delivery and interference in static of powerful sensors including odometry, magnetometer, and wireless networks,” in ACM SIGCOMM Computer Communication WiFi found in cell phones have been shown to have an Review, vol. 36, no. 4. ACM, 2006, pp. 51–62. accuracy of 1.69 m [14], [15]. Roomba robots have also been [8] R. Mahajan, M. Rodrig, D. Wetherall, and J. Zahorjan, “Analyzing the mac-level behavior of wireless networks in the wild,” ACM SIGCOMM used to collect wireless coverage maps by spinning in small Computer Communication Review, vol. 36, no. 4, pp. 75–86, 2006. grid areas [16], [17] but they cannot autonomously navigate [9] P. Bahl, J. Padhye, L. Ravindranath, M. Singh, A. Wolman, and B. Zill, to reduce human time and effort costs or execute complex “Dair: A framework for managing enterprise wireless networks using desktop infrastructure,” in HotNets IV, 2005. motions like our robot can. Our work takes advantage of [10] S. Rangwala, R. Gummadi, R. Govindan, and K. Psounis, much more powerful sensors that can localize within 10 “Interference-aware fair rate control in wireless sensor networks,” in cm using a wheeled platform that can reproduce complex ACM SIGCOMM Computer Communication Review, vol. 36, no. 4. ACM, 2006, pp. 63–74. movements. [11] Y.-C. Cheng, Y. Chawathe, A. LaMarca, and J. Krumm, “Accuracy Previous efforts have proposed techniques to use predic- characterization for metropolitan-scale wi-fi localization,” in Proceed- tions to reduce the duration of handoffs or inform applica- ings of the 3rd international conference on Mobile systems, applica- tions, and services. ACM, 2005, pp. 233–245. tions to allow for prefetching data and reduce the impact of [12] R. Mahajan, J. Zahorjan, and B. Zill, “Understanding wifi-based handoffs [18], [19]. Nearby access points have also been used connectivity from moving vehicles,” in Proceedings of the 7th ACM to opportunistically help mitigate WiFi handoffs for moving SIGCOMM conference on Internet measurement. ACM, 2007, pp. 321–326. vehicles when moving across multiple buildings [20]. Our [13] Y. Chen, D. Lymberopoulos, J. Liu, and B. Priyantha, “Fm-based in- work that helps to expose and reproduce fine-grain failures door localization,” in Proceedings of the 10th international conference in AP handoffs for moving devices is orthogonal to these on Mobile systems, applications, and services. ACM, 2012, pp. 169– 182. efforts as it provides a mechanism for robustly evaluating [14] H. Wang, S. Sen, A. Elgohary, M. Farid, M. Youssef, and R. R. handoff solutions. Choudhury, “No need to war-drive: Unsupervised indoor localization,” in Proceedings of the 10th international conference on Mobile systems, applications, and services. ACM, 2012, pp. 197–210. V. C ONCLUSION [15] A. Rai, K. K. Chintalapudi, V. N. Padmanabhan, and R. Sen, “Zee: Zero-effort crowdsourcing for indoor localization,” in Proceedings of Diagnosing wireless connectivity issues can be difficult the 18th annual international conference on Mobile computing and due to the many factors that potentially impact wireless networking. ACM, 2012, pp. 293–304. [16] S. Sen, B. Radunovic, R. R. Choudhury, and T. Minka, “Spot localiza- performance. We showed how autonomous robots can help tion using phy layer information,” in Proceedings of ACM MOBISYS, to methodically drill down to the root cause by capturing 2012. detailed wireless measurements that eliminate unlikely fac- [17] O. Rensfelt, F. Hermans, P. Gunningberg, L.-Å. Larzon, and E. Björnemo, “Repeatable experiments with mobile nodes in a re- tors. We were able to identify AP handoffs as the reason for locatable wsn testbed,” The Computer Journal, vol. 54, no. 12, pp. our own robot’s motion-based wireless connectivity issue by 1973–1986, 2011. analyzing variations in wireless performance while subjected [18] U. Javed, D. Han, R. Caceres, J. Pang, S. Seshan, and A. Varshavsky, “Predicting handoffs in 3g networks,” in Proceedings of the 3rd ACM to repeatable motions. This was a challenging wireless prob- SOSP Workshop on Networking, Systems, and Applications on Mobile lem that arose from poor decisions dependent on accurate Handhelds. ACM, 2011, p. 8. timing and it is unclear that we could have uncovered them [19] S. Seshan, H. Balakrishnan, and R. H. Katz, “Handoffs in cellular wireless networks: The daedalus implementation and experience,” without the accuracy and control of autonomous robots. Wireless Personal Communications, vol. 4, no. 2, pp. 141–162, 1997. Opportunities for future work include using these detailed [20] A. Balasubramanian, R. Mahajan, A. Venkataramani, B. N. Levine, wireless maps for better management of enterprise wireless and J. Zahorjan, “Interactive wifi connectivity for moving vehicles,” in ACM SIGCOMM Computer Communication Review, vol. 38, no. 4. networks, ensuring timely maps for wireless localization ACM, 2008, pp. 427–438. solutions, and automated diagnosis of wireless problems. FinE-R 2015 Page 14 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Toward Soft, Robust Robots for Children with Autism Spectrum Disorder Hong Tuan Teo and John-John Cabibihan Abstract—A meltdown is one of the most challenging A robot is not meant to replace therapists but is meant to behaviors of children with Autism Spectrum Disorder (ASD), be used as a mediator to provoke interaction between the where a child could not calm down or too overwhelmed with a child and another person. The objective behind using robot is certain situation. Because social robots are becoming useful as a to increase their interactions through longer eye contacts, therapy tool between the therapist and a child with ASD, as which are important to build the child’s confidence level. robot designers, we want to anticipate that a robot could be This can be done through touching, playing and engaging in thrown on the floor or to the therapist or caregiver. In addition, we want to investigate how to better protect the robot from imitation games with the robot. By doing so, they are able to being damaged. Typical robots are constructed in plastic open up and allowing themselves to engage in discussion material. In this paper, a sample of plastic material and a about the robot’s activities. sample of silicone material were compared in a drop test Research studies have shown that using robots as experiment at the heights of 0.0254 m, 0.5 m, and 1 m. These therapeutic tool for autism often lead to increase in certain heights simulate a possible situation where the robot can be areas such as engagement, attention, spontaneous imitation dropped. Our result shows the differences in the impact and novel social behaviour such as joint attention [9-11]. between the silicone and the plastic samples. This work provides Robots are nonthreatening and can be design in such a way a baseline study as a step toward soft, robust robots for children that they are engaging and allowing productive interaction. with ASD. The current robots seen to date have internal components I. INTRODUCTION consisting of microcontrollers, mechanisms, sensors, and actuators. However, most of the robots are lacking the Autism Spectrum Disorder (ASD) is characterized by a robustness in the design. Robustness refers to the ability to triad of impairments in social communication, social operate without failure when subjected to a variety of harsh interaction and imaginative skills [1]. The Center for Disease handling conditions. In order for a robot to be robust, the Control in the USA estimates that 1 out 68 children are robot must be able to absorb impact in situations such as diagnosed with ASD. Some children with ASD could go into dropping onto the concrete ground from high ground, thrown a meltdown because they could find themselves against the wall or knocked repeatedly by force. overwhelmed in a certain situation. Such situations could The soft and robust features of a robot are especially include loud noises, bright lights, strong smells, and many important when children with ASD are in a meltdown other situations. situation. This will occur when their needs and wants are not Research efforts have been put into the field of social met or when they are not able to adapt to the changes in the robotics in an attempt to use robots to assist humans in a environment. If they lose control, the child may pick up a diverse number of ways. Socially interactive robots are used robot that is in sight and exert force on it. There is a to communicate, express and perceive emotions, maintain possibility that the exterior structure housing the components social relationships, interpret natural cues, and develop social will crack under impact with another structure. Furthermore, competencies [2,3]. To ensure the suitability of the robot’s the robot may cease functioning because of damages in the design, research studies have been conducted to obtain internal components. requirements from the end-user group who are children with For a robot to be robust, the materials and the embedded autism. Since these children have impaired communication, technologies are important in ensuring that a robot can therapists, parents and teachers were asked to give their withstand harsh handling conditions. Materials that are able feedback on suitable design of robots [4]. Other efforts have to cushion the impact upon landing are generally preferred. also been made to compile a detailed set of design Such shock absorbing materials are commonly used by requirements that are not subjective, but can be generalized designers to protect products such as phones, hard disks and to most of the children’s preferences [5,6]. Robots with equipment. For example, a hard disk is incorporated with an overly mechanized appearances may also not derive the best accelerometer that will send a signal to immediately unload results since too many exposed mechanical parts can cause its head when it is under free fall. This prevents the hard the child to shift focus from the interaction itself [7,8]. disk’s head from coming in contact with the platter, which H.T. Teo was with the Electrical and Computer Engineering Department, can cause considerable damage to the device. National University of Singapore, Republic of Singapore, 117576. Most of the manufacturers prefer using plastic as the J.J. Cabibihan is with the Mechanical and Industrial Engineering exterior structure for robots because it can be readily molded Department, Qatar University, Doha, 2713, Qatar (e-mail: to shape. Rubber materials have generally excellent tensile john.cabibihan@qu.edu.qa). strength, elongation, tear resistance, and resilience properties and are commonly used to function as a shock absorber, as FinE-R 2015 Page 15 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 vibration isolator or as dampers. Rubber has low modulus of With the structure selected, a polyester resin sample and a elasticity, it is capable of sustaining a deformation and will silicone rubber sample were prepared to identical size and return to its original dimension. shape for better comparison. How well the robot reacts to the shock is dependent on the C. Procedures choice of material. In the next sections, we describe a series of drop test experiments at 0.0254 m, 0.5 m and 1 m heights. The silicone rubber sample was subsequently brought to These heights simulate possible conditions that a robot might the 0.0254 m height and held in stationary position for a few be subjected to. seconds to allow the registration of 1 G value before dropping onto the concrete floor. Caution has to be taken to II. DROP TEST EXPERIMENT prevent exerting extra pressure to the experimental object to avoid erroneous readings. Any suspected pressure applied to A. Calibration of Accelerometer and Conversion of Units the experimental object during the trials will not be used for An accelerometer was embedded in the internal structure analysis. Both samples were subjected to the same starting of the test object. Acceleration is the rate of change of drop position with the cross sectional area of the sample velocity over time. Dynamic responses can be inferred from parallel along to the axis of the concrete floor. The sample the experiment to which the accelerometer is mounted. In must land with the cross sectional parallel along the concrete order to convert the voltage output from accelerometer to floor during the impact and after the impact for accurate acceleration in G, intermediate steps were needed. Firstly, comparison. This experiment was conducted 8 times for the the analog voltage reading from the output of the same height. The data from the drop test was then plotted accelerometer was obtained under static acceleration when it out. This same procedure was repeated for other heights. was in the direction of Earth’s gravity field (9.8 m/s2). After the experiment for the silicone rubber sample was Secondly, this analog voltage value will be reduced when it completed, the same set of experiment procedure was applied is not in the direction of Earth’s gravity field to obtain the to the polyester resin sample. An illustration of the difference with respect to 0 G point. Lastly, the value is experimental set-up is shown in Fig. 2. divided by the sensitivity of the device to obtain the acceleration value in G. Calibration of the accelerometer and a conversion of unit were needed to convert the output analog reading to the correct corresponding G values. This conversion of values allowed better analysis of the results. As the output voltages reading from the tri-axis accelerometer are different from the ideal case, the axis had to be calibrated individually. Attention was be placed on the square root of the sum of the 3-axis as it represents the total acceleration acting on the device during the drop test experiment. B. Experimental Samples Fig. 2. Experimental set-up A proper cylindrical housing for the devices was first selected. The devices that were secured in the housing included a 9-volt battery to provide power supply to the III. RESULTS Arduino board, accelerometer, and data logging device. Caution has been taken to ensure that there was enough space In this drop test experiment result, we focused on the for the impact to take place. The side of the housing was acceleration peak and time interval between impact designed not to hit the accelerometer during the impact to experienced by the material to the time that the material was prevent erroneous reading. at rest. Each figure shows the response graph at different heights. Fig.1. Experimental samples, silicone (left), polyester (right). FinE-R 2015 Page 16 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Fig. 3. Silicone Response at 0.0254m Fig. 4. Polyester Response at 0.0254m Fig. 5. Silicone Response at 0.5m Fig. 6. Polyester Response at 0.5m Fig. 7. Silicone Response at 1m Fig. 8. Polyester Response at 1m FinE-R 2015 Page 17 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 The settling time of a sample from the time of impact to the time the sample is at rest is shown in Table 1. Height (m) Material Settling time response (sec) 0.0254 Silicone 0.4 0.0254 Polyester Resin 0.3 0.5 Silicone 0.9 0.5 Polyester Resin 0.8 Fig. 9. Results of drop test from 1 m height for silicone rubber (left) and polyester resin (right) samples. The polyester resin has noticeable cracks. 1.0 Silicone 1.6 1.0 Polyester Resin 0.8 IV. CONCLUSION Analysis of the experimental result shows that silicone Table 1. Settling time response at different heights rubber material displayed lower G value as compared to From the experiment, it can be observed that the polyester resin, which was noticeable during the first impact. acceleration is much higher in polyester resin wheel than The subsequent number of damping is an indication of how silicone rubber wheel especially during the first impact for well the material reacts to the impact. From the results, it can the same drop height. be concluded that rubber material took more time to react to Acceleration is defined as the rate of change velocity over the change of velocity during the impact as most of the time. Since velocity is independent of mass during free fall impact would have been absorb and dissipated in the period as it is under the influence of gravity (g = 9.8 m/s2), material. the shorter the rate of change of time, the higher the Social robots are now being used as a tool for autism acceleration. From Fig. 2, the downward velocity V1 therapy and diagnosis [12]. Experiments have shown that represents the velocity before the impact while V2 represents children with autism prefer playing with interactive, robotic the upward velocity after the impact. From Eqn. 1-3, the toys rather than passive toys [13,14]. They also direct more variables used were ax (constant gravitational acceleration), eye gaze and focus more attention towards robots [15]. xf (final distance), xi (initial distance), Vxf (final velocity) and Therapy for children with autism not only applies to their Vxi (initial speed). impairments but also to their growth needs, hence encompassing their educational needs as well. Robots are (1) less intimidating than humans; they not only act as playmates V xf2 = V xi2 + 2a x ( x f − xi ) for the child, but they can be used as small, colourful toys, ensuring that the child feels at ease during the interaction V1 = − 2g(D1 ) (2) [5,14,16,17,18]. They can be programmed to adapt their behaviour in accordance to the specific needs of a child with V2 = 2g(D2 ) (3) whom it is interacting, hence customizing the therapy for a child [5,19] The robustness of the robot is especially important when V2 − V1 (4) the child with ASD is in a meltdown situation. The Acceleration = t 2 − t1 consequences and damages due to the child’s action during meltdown situation could not be predicted. The experimental High acceleration especially for the first impact is harmful result conducted from three different heights shows that as it shows that the material is stiffer and does not respond silicone rubber material displayed lower G value noticeable well to the impact. It is noticeable that it took much longer on the first impact as compared to the response graph of the time for the silicone rubber sample to settle down to polyester resin. The rubber material took more time to react stationary as compared to polyester resin sample. Longer to the change of velocity during the impact as most of the time period shows the presence of elasticity in the material, shock would have been absorbed and dissipated in the which was required to absorb the shock within the material. material before changing the course of direction. This is Evidence of cracks was subsequently observed on the different from the polyester material whereby it received polyester resin wheels while the trials are being conducted at most of the impact, which eventually lead to cracks. The 1 m height. A comparison between the two experimental outcome shows that rubber material is more robust and objects is shown in Fig. 9. Cracks are shown on the polyester should be used to protect the hardware and software of the resin sample. robot as it is capable of absorbing the impact better. Future work involves recreating and analyzing a scenario where an object is thrown to the wall. FinE-R 2015 Page 18 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 REFERENCES [10] Ricks, Daniel J, Colton, Mark B, “Trends and considerations in robot- [1] A. Cashin and P. Barker, “The triad of impairment in autism assisted autism therapy,” Robotics and Automation IEEE International revisited,” J. Child Adolesc. Psychiatr. Nurs., vol. 22, no. 4, pp. 189– Conference, pp. 4354–59, 2010. 193, 2009. [11] How social robots will help us to diagnose, treat and understand [2] Fong T, Nourbakhsh I, Dautenhahn K (2003) A Survey of Socially autism,” 12th International Symposium of robotics research, pp. 552- Interactive Robots. Robotics and Autonomous Systems 42 (3-4). 63, 2007 [3] Li H, Cabibihan JJ, Tan YK (2011) Towards an effective design of [12] Bird G, Leighton J, Press C, Heyes C (2007) Intact automatic imitation social robots. International Journal of Social Robotics:1-3. of human and robot actions in autism spectrum disorders. Proc Biol Sci [4] Ferrari E, Robins B, Dautenhahn K Therapeutic and educational 274 (1628):3027-3031. objectives in robot assisted play for children with autism. In: Robot [13] Dautenhahn K, Werry I (2004) Towards interactive robots in autism and Human Interactive Communication, 2009. RO-MAN 2009. The therapy: Background, motivation and challenges. Pragmatics & 18th IEEE International Symposium on, Sept. 27 2009-Oct. 2 2009 Cognition 12 (1):1-35. 2009. pp 108-114. [14] Dautenhahn K (2003) Roles and functions of robots in human society: [5] Michaud F, Duquette A, Nadeau I Characteristics of mobile robotic implications from research in autism therapy. Robotica 21 (4):443-452. toys for children with pervasive developmental disorders. In: Systems, [15] Dautenhahn K (2007) Socially intelligent robots: dimensions of Man and Cybernetics, 2003. IEEE International Conference on, 5-8 human-robot interaction. Philos Trans R Soc Lond B Biol Sci 362 Oct. 2003 2003. pp 2938-2943 vol.2933. (1480):679-704. [6] Giullian N, Ricks D, Atherton A, Colton M, Goodrich M, Brinton B [16] Woods S (2006) Exploring the design space of robots: Children's Detailed requirements for robots in autism therapy. In: Systems Man perspectives. Interacting with Computers 18 (6):1390-1418. and Cybernetics (SMC), 2010 IEEE International Conference on, 10- [17] Michaud F, Larouche H, Larose F, Salter T, Duquette A, Mercier H, 13 Oct. 2010 2010. pp 2595-2602. Lauria M Mobile robots engaging children in learning. In: Canadian [7] Kozima H, Nakagawa C, Yasuda Y (2007) Children-robot interaction: Medical and Biological Engineering Conf, 2007. a pilot study in autism therapy. Progress in Brain Research 164:385. [18] Marti P, Pollini A, Rullo A, Shibata T Engaging with artificial pets. In: [8] Kozima H, Zlatev J An epigenetic approach to human-robot Proceedings of the 2005 annual conference on European association of communication. In: Robot and Human Interactive Communication, cognitive ergonomics, 2005. University of Athens, pp 99-106. 2000. RO-MAN 2000. Proceedings. 9th IEEE International Workshop [19] Robins B, Otero N, Ferrari E, Dautenhahn K Eliciting Requirements on, 2000. IEEE, pp 346-351. for a Robotic Toy for Children with Autism - Results from User [9] Diehl JJ, Schmitt LM, Villano M, Crowell CR, “The clinical use of Panels. In: Robot and Human interactive Communication, 2007. RO- robots for individuals with autism spectrum disorders: a critical MAN 2007. The 16th IEEE International Symposium on, 26-29 Aug. review,” Research in Autism Spectrum Disorders, vol 6, pp. 249–262, 2007. pp 101-106. 2012. FinE-R 2015 Page 19 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Adapting Low-Cost Platforms for Robotics Research Karimpanal T.G., Chamambaz M., Li W.Z., Jeruzalski T., Gupta A., Wilhelm E. Abstract— Validation of robotics theory on real-world hard- available here: ware platforms is important to prove the practical feasibility https://github.com/SUTDMEC/EvoBot Downloads.git of algorithms. This paper discusses some of the lessons learned • Adaptable: The final platform is intended to be as while adapting the EvoBot, a low-cost robotics platform that we designed and prototyped, for research in diverse areas general purpose as possible, with minimum changes in robotics. The EvoBot platform was designed to be a low- needed to be made to the base firmware by users in cost, open source, general purpose robotics platform intended order to scale to a wide variety of common research to enable testing and validation of algorithms from a wide applications. Some representative applications are de- variety of sub-fields of robotics. Throughout the paper, we scribed in section IV. . . . outline and discuss some common failures, practical limitations and inconsistencies between theory and practice that one may In the process of designing the EvoBots platform, a encounter while adapting such low-cost platforms for robotics great deal of failure-based learning was involved, and this research. We demonstrate these aspects through four represen- paper compiles and synthesizes this process in order for it tative common robotics tasks- localization, real-time control, to be useful for future robotics researchers who intend to swarm consensus and path planning applications, performed using the EvoBots. We also propose some potential solutions to develop their own platforms, or adapt commercially available the encountered problems and try to generalize them. platforms. Specific emphasis will be on challenges common Index Terms — low cost, open-source, swarm robotics, lessons to robotic platforms in general, and approaches used to avoid learned failure when attempting to solve them. The authors would like to point out that each robotics project presents a unique I. I NTRODUCTION sets of problems, so we have attempted to only describe problems we think may be generalized. The Motion, Energy and Control (MEC) Lab at the Singapore University of Technology and Design (SUTD) is II. P RECEDENTS AND E VO B OT D ESIGN engaged in an ambitious research project to create a swarm This section will discuss commonalities and differences of autonomous intelligence-gathering robots for indoor en- between the EvoBot and other comparable low-cost robotics vironments. After reviewing commercially available low- platforms, as well as provide insights into the lessons learned cost robotics platforms, some of which are shown in Table during the design process. In order to reduce the time be- I, the decision was taken to build a custom ground-based tween design cycles, the EvoBots were prototyped with a 3D robot platform for performing swarming research, dubbed printed body and developed across three major generations the ’EvoBot’. The primary trade-off is between platform and several minor revisions. An exploded view of the EvoBot flexibility and cost, where existing robots are intended to is shown in Figure 1. either be applied in specific research areas, and are hence Like the Khepera, Finch, Amigobot (table I) and most of equipped with limited and specific sensor and communica- the other platforms, locomotion on the EvoBot is achieved tion capabilities, or are more flexible with respect to sensor and firmware packages, but are also more expensive. In terms of software control development environments, well- established frameworks such as ROS [1] or the Robotics Toolbox [9] have extremely useful modular functions for performing baseline robotics tasks, but require substantial modification between applications and require specific oper- ating systems and release versions. With this in mind, the MEC Lab set out to develop the EvoBot platform with the following goals: • Low cost: affordable for research groups requiring a large number of swarming robots (e.g. more than 50) with sufficient sensing and control features. • Open source: The hardware, body/chassis design, appli- cation software and firmware for the EvoBots are fully open source in order to enable any group to replicate Fig. 1. The 3-D printed case has two slots at the bottom for the optical flow sensors, a housing for the left and right tread encoders, and 5 IR depth the platform with minimal effort. All the hardware and sensors. The encoders on the forward wheels and the optional ultrasonic software files used for the design of the EvoBots is sensors are not shown FinE-R 2015 Page 20 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 TABLE I high even after using methods such as the Kalman filter [15]. P RECEDENTS FOR LOW- COST RESEARCH ROBOTICS PLATFORMS Although obtaining the position estimates from the IMU is Platform Est. Cost (USD) Commercial Scholar Hits Schematics Code not recommended in general, good estimates can be obtained Khepera [19] 2200 Yes >1000 No Yes with units that are capable of a much higher resolution and Kilobot [22] 100+ Yes 82 No Yes data rate. Such units however, come at an expense and thus e-Puck [18] 340+ Yes >1000 Yes Yes may not be the ideal choice for a low-cost platform. Jasmine [4] 150+ No >1000 Yes Yes Formica [11] 50 No 10 Yes Yes A. Sensing Features Wolfbot [6] 550+ No 4 Yes Yes While the sensors mentioned earlier in this section focused Colias [3] 50+ No >1000 No No on features shared with other platforms, there are some Finch [17] 100 Yes 73 No Yes sensing, control and communication features specific to the Amigobot [2] 2500 Yes 170 No Yes EvoBots. EvoBot 300 Yes 0 Yes Yes In almost all robotics applications, errors in the robot’s position estimates may gradually accumulate (e.g. for ground robots, skidding or slippage of the wheels on the ground surface confound the wheel encoders). In the first two gener- using a differential drive system, with motors on either side ations of the EvoBots, this error in the wheel encoders caused of its chassis. Although this system introduces kinematic substantial challenges for localization. In order to tackle this constraints by restricting sideways motion, the associated problem, optical flow sensors were placed on the underside of simplicity in manufacturing and assembly, and in the mathe- the EvoBot. They detect a change in the position of the robot matical model for use in control applications are significant by sensing the relative motion of the robot body with respect advantages. The wheels are coupled to the motors through a to the ground surface. This ensures that the localization can gearbox with a gear reduction ratio of 1:100. After reduction, be performed more reliably even in the case of slippage. As the final speed of the robot can be varied from -180 to shown in Figure 1, there are two such sensors placed side 180 mm/s, so that both forward and backward motions are by side at a specific distance apart. This arrangement allows possible. inference of heading information of the robot along with The speed of each motor is controlled by a pulse width information of the distance traveled. Although the optical modulated voltage signal. In order to ensure predictable flow sensors perform well in a given environment, they need motion, an internal PI controller is implemented by taking to be calibrated extensively through empirical means. In feedback from encoders that track the wheel movement. addition, it was found that the calibration procedure had to The controller parameters may need to be hand-tuned to be repeated each time a new surface is encountered, as the compensate for minor mechanical differences between the associated parameters are likely to change from surface to two sides of the robot, arising from imperfect fabrication surface. and assembly. As described above, the wheel encoders, optical flow as For obstacle detection and mapping applications, 5 infra- well as the IMU sensors can all be used to estimate the red (IR) sensors were placed on the sides of a regular robot’s position and heading. Each of these becomes relevant pentagon to ensure maximum coverage, with one IR sensor in certain specific situations. For example, when the robot facing the forward direction. Similar arrangement of range has been lifted off the ground, the IMU provides the most sensors is found in platforms such as Colias and e-puck reliable estimate of orientation; when the robot is on the (table I). Despite this arrangement, there exist blind spots ground and there is slippage, the optical flow estimates are between adjacent sensors, which could lead to obstacles not the most reliable; and when there is no slippage, the position being detected in certain orientations of the robot relative estimates from the encoders are reliable. Some details on to an obstacle. The use of ultrasonic sensors instead of the use of multiple sensors for localization are mentioned in IRs, could reduce the probability of occurrence of blind section IV-A. spots due to their larger range and coverage, but they are In their current configuration, the EvoBots also include also more expensive. It is thus advisable to choose sensors the AI-ball, a miniature wifi-enabled video camera with appropriately and to thoroughly test their performance and comprehensive driver support and a low cost point to capture limitations before a final decision on the physical design and video and image data [25]. The camera unit is independent sensor placements is made. of the rest of the hardware and thus can be removed without Another feature present in the EvoBots common to other any inconvenience if it is not needed. platforms is the use an inertial motion unit (IMU). The 6DOF IMU gives information regarding the acceleration of B. Control and Communication Features the robot in the x, y and z directions, along with roll, There are a wide variety of microcontrollers which are pitch and yaw (heading) information. Although in theory, presently available on the market. The chipset used on the the position of the robot can be inferred using the IMU data, EvoBot is the Cortex M0 processor, which has a number in practice, these sensors are very noisy, and the errors in of favorable characteristics such as low power, high per- the position estimates from these sensors are unacceptably formance and at the same time, is cost efficient, with a FinE-R 2015 Page 21 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 large number of available I/O (input/output) pins. One of the A. Sensing primary advantages of this processor is that it is compatible While it may be considered acceptable to use ideal sensor with the mbed platform (https://developer.mbed.org/), which models in simulation, in real-world robotics applications, is a convenient, online software development platform for and especially in the case of low-cost platforms, sensing ARM Cortex-M microcontrollers. The mbed development information about the environment can be done only with platform has built-in libraries for the drivers, for the motor a certain degree of range and accuracy. It is thus important controller, Bluetooth module and the other sensors. to be judicious in choosing the location of the sensors. For Like the e-puck, the EvoBots have Bluetooth communica- the case of range sensors for obstacle detection, blind spots tion capability. After experimenting with various peer-to-peer must be avoided as much as possible to prevent unexpected and mesh architectures, it was determined that in order to behaviours. Also, the upper and lower limits of the sensor maintain flexibility with respect to a large variety of potential range must be considered. As in the case of the EvoBots target applications, having the EvoBots communicate to a or other low-cost platforms, the sensor range may be quite central server via standard Bluetooth in a star network limited and prone to noise, and blind spots exist despite topology using the low-cost HC-06 Bluetooth module was careful sensor placement. In case of occurrence of blind the best solution. Sensor data from the various sensors spots, it may be worth encoding default behaviours into gathered during motion is transmitted to the central server the robot such as conditional wall following or automatic every 70ms. The camera module operates in parallel and uses steering away from detected obstacles. Wi-Fi 802.11b to transfer the video feed to the central server. The Bluetooth star network is also used to issue control B. Calibration commands to the robots. The behaviour of sensors are typically considered to be In addition, having a star network topology with a central same for different environments during simulation. In prac- server indirectly allows additional flexibility in terms of tice, most sensors need some form of calibration. In addition, programming languages. Only a Bluetooth link is required, the calibration parameters for certain sensors may depend on and all the computation can be done on the central server. the environment in which they operate. For example, the For example, the EvoBot can be controlled using several calibration parameters for the optical flow sensors in the programming languages, including (but not limited to) C, EvoBots vary based on the surface on which they operate. In python and M AT LAB. The Bluetooth link also opens up addition, the parameters are very sensitive to small variations the possibility of developing smartphone mobile applications in ground clearance which can be caused by variations in the for it. Table II summarizes all the sensors used in the EvoBot. robot body due to imperfect 3D printing or during assembly. It is often the case that parameters will have to be set TABLE II and reset from time to time. This can considerably add T HE FINAL SENSING CAPABILITIES OF THE 3 RD GENERATION E VO B OT to development time and effort. Automating the calibration PLATFORM process should be done whenever possible. If automation is not possible, one way to mitigate this issue is to have the software package include user commands that can change Signal Sensor Frequency Full Range Accuracy calibration parameters on the fly. This was the solution acceleration MPU6050 1000 Hz +/- 0.1% adopted for the EvoBots in order to calibrate the encoders Temperature MPU6050 max 40 Hz +/-1 ◦ C from -40 to 85 ◦ C and optical flow sensors. x/y pixels ADNS5090 1000 Hz +/- 5mm/m C. Timing Encoders GP2S60 400 Hz +/- 5mm/m Battery current ACS712 33 Hz +/- 1.5 % @ 25C In simulation, the robot’s update loop and solvers can Proximity GP2Y0A02YK0F 25 Hz +/- 1mm have either fixed or variable time steps, but it is most likely Camera AI-ball 30fps VGA 640x480 to be variable in practice. Thus, keeping track of time is critical, especially for real-time applications. For example, in the EvoBots, the performance of the localization algorithm (section IV-A) significantly depreciates if the time step is considered fixed. III. G ENERAL P ROBLEMS D. Communication The EvoBot platform was designed to be useful for a When a large amount of sensor data and instructions wide range of research problems, and to enable researchers needs to be exchanged between platforms or between a working in theoretical robotics to easily shift their algorithms platform and a central machine, maintaining the integrity from simulation to hardware. Moving from virtual to real of the communicated data is critical. When the exchanged platforms introduces a set of additional issues which must information is delayed or lost either partially or completely be considered. Some of these, which were encountered while during information exchange, it could lead to a series of developing applications for the EvoBots, are discussed in this unwanted situations such as improper formatting of data, section. lack of synchronization, incorrect data etc., These aspects are FinE-R 2015 Page 22 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 seldom accounted for in simulation. In the EvoBots, the issue 1 0.8 of lack of data integrity was tackled by performing cyclic 0.6 Robot 2 redundancy checks (CRC) [21] as part of the communication 0.4 routine. It is ideal to perform bit-wise checks as well, but this 0.2 Robot 1 Y (m) 0 has not been done in the case of the EvoBots. −0.2 The issues listed in this section apply to almost all robotics −0.4 Ground Truth applications in general. The following section discusses some −0.6 Estimated (Kalman Filter) −0.8 Just Model of the applications and the associated issues encountered −1 Measurment Only −1 −0.5 0 0.5 1 1.5 2 during their implementation using the EvoBots. X (m) IV. A PPLICATIONS Fig. 2. The Kalman Filtering process improves the state estimate beyond what the model and the measurements are capable of on their own. This section discusses application-specific issues in diverse sub-fields of robotics such as real-time control, swarm and artificial intelligence applications. These applications were selected because they are commonly implemented tasks from in the estimated state.One solution to this problem is to label a wide variety of sub-fields in robotics. each sensor data package with time, compute the sampling time on the central computer and use the computed real-time A. Localization sampling time in the extended Kalman filter formulation. Ground robots typically need to have an estimate of With these factors in mind, labeling the sensor data package position and heading. The EvoBot platform, as well as many with time seems to be an efficient solution to deal with other robots, are designed to be used in an indoor environ- variable sampling time. ment, and therefore it is not possible to use GPS without A common problem associated with estimating the state modifying the environment in question; also, positioning using the velocity sensors is “slippage”. In cases where information provided by GPS has the accuracy in the order a robot slips on the floor, the wheel encoders reflect an of a 101 meters which is not suitable for small ground robots. erroneous result. In the worst case, when the robot is stuck, For this reason, the set of on-board sensors are used to the encoders keep providing ticks as if the robot is moving, estimate the robot’s states of position and heading. The on- which leads to a significant error in the estimated position. board sensors however, are typically associated with some To solve this, we utilized the information from the optical noise and retrieving positioning information from them leads flow sensors and designed an adaptive Kalman filter. The to erroneous results. For instance, integrating acceleration velocities reflected by the encoders and optical flow sensors data to get the velocity and position does not provide accurate are compared and in case there is a significant difference state estimation and leads to huge offsets from the true between them, the occurrence of slippage is inferred. Then, value due to the associated noise. To overcome this issue, the noise covariance matrix in the extended Kalman filter is state estimation is performed using a method based on the changed so that the filter relies more on the velocity provided Extended Kalman Filter [15]. by the optical flow sensor. Thus, we infer that although We used the information provided by the wheel encoders, having multiple sensors sensing the same quantity may seem optical flow sensors and the heading provided by IMU’s redundant, each of them, or a combination of them may be gyroscope to estimate the position and heading. The sensor useful in different contexts. information is combined with the mathematical model of the robot to estimate the position and heading of the robot. Figure 2 presents the result of an experiment where two robots follow a trajectory (red line) and the data from the B. Application in Real-time Control sensors are used in the extended Kalman filter to estimate its position and heading. In the next two paragraphs we explain In a perfect scenario where there is no disturbance and two problems we faced while implementing the designed model mismatch, it is possible to use some feed-forward Kalman filter and we describe the corresponding solutions control to drive the robot on a desired trajectory. However, in used in our platform. the real world, a number of issues such as model mismatch, Timing is one of the most important factors for localization disturbance and error in the internal state deviates the robot and control tasks. The delay in communication between from its desired path. Therefore, feedback control is vital to the robot and central computer where the Kalman filter is achieve an accurate tracking behavior. In general, the navi- running, causes some variation in the time intervals within gation problem can be devised into three categories: tracking which each sensor data package is received. For example, the a reference trajectory, following a reference path and point time intervals can vary between 50 to 100 ms Therefore, as stabilization. The difference between trajectory tracking and mentioned in section III, the sampling time in the extended path following is that in the former, the trajectory is defined Kalman filter cannot be fixed a-priori. The state estimation over time while in the latter, there is no timing law assigned procedure highly depends on the corresponding sampling to it. In this section, we focus on designing a trajectory time and using a fixed sampling time leads to a large error tracking controller. FinE-R 2015 Page 23 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 The kinematic model of our platform is given by: 2 Reference Trajectory Current Position   1.5 v1 + v2 x˙c = cos θc 2 1 y (meter)   v1 + v2 0.5 y˙c = sin θc 2 0 1 θ˙c = (v1 − v2 ) (1) −0.5 l −1 −1.5 −1 −0.5 0 0.5 1 1.5 where v1 and v2 are velocities of right and left wheels, x (meter) θc is the heading (counter clockwise) and l is the distance between two wheels. As stated earlier, one of the reasons Fig. 3. The trajectory of the robot for adopting a differential drive configuration for the EvoBot is the simplicity of the kinematic model (1). Hereafter, we 2 Error in x Error in y use two postures, namely, the “reference posture” pr = 1.5 Errrot in θ (xr , yr , θr )0 and the “current posture” pc = (xc , yc , θc )0 . The 1 error posture is defined as the difference between reference posture and current posture in a rotated coordinate where 0.5 (xc , yc ) is the origin and the new X axis is the direction of 0 θc . −0.5     xe cos θc sin θc 0 −1 0 50 100 150 200 250 pe =  ye  =  − sin θc cos θc 0  (pr − pc ). Time (sec) θe 0 0 1 Fig. 4. The errors in x, y and θ (2) The goal in tracking control is to reduce error to zero as fast as possible considering physical constraints such as One of the important factors in implementing the real time maximum velocity and acceleration of the physical system. controller is time synchronization. The controller needs to The input to the system is the reference posture pr and keep track of time in order to generate the correct reference reference velocities (vr , wr )0 while the output is the current point. This issue is critical especially when a number of posture pr . A controller is designed using Lyapunov theory robots need to be controlled. In this scenario, an independent [16] to converge the error posture to zero. It is not difficult counter can be assigned to each robot to keep track of its to verify that by choosing time and generate the correct reference point at each time ( step. v1 = vr cos θe + kx xe + 2l (wr + vr (ky ye + kθ sin θe )) C. Application in Swarm Robotics v2 = vr cos θe + kx xe − 2l (wr + vr (ky ye + kθ sin θe )) (3) The overarching application of the EvoBots platform is the resulting closed-loop system is asymptotically stable for to enable low-cost swarm robotics research. The study of any combination of parameters kx > 0, ky > 0 and kθ > 0.1 emergent collective behaviour arising from interactions be- The tuning parameters highly affect the performance of the tween a large number of agents and their environment is an closed loop system in terms of convergence time and the area which is gaining increasing importance recently. The level of control input applied to the system. Hence, we chose robots used for these purposes are usually simple and large parameters kx , ky and kθ based on the physical constraints in number, with communication capabilities built into them. of our platform e.g. maximum velocity and acceleration. Although it is ideal to have peer-to-peer communication Extensive simulation is performed to verify the controller (3). between the agents, the same effect can be simulated through Figure 3 shows some experimental results where the robot a star-shaped network, where all the agents share information follows a reference trajectory (blue curve). The reference with a central machine. trajectory is a circle of radius 1m. The initial posture is A typical swarm robotics application is to have the swarm selected to be (0, 0, 0)0 . Errors xr − xc , yr − yc , and θr − θc arrive at a consensus about some common quantity, and are shown in Figure 4. We remark that reference linear to use this as a basis to collectively make decisions about and angular velocities vr and wr are difficult to obtain for the next actions of each member [5]. Figure 5 shows the arbitrary trajectories. We solved this problem using some configuration of a set of 6 robots at different times. The numerical methods where the point-wise linear and angular heading/orientation of each robot depends on those of the velocities are computed numerically. other robots and is governed by the update equation: 1 Choosing V = 1 (x2 + y 2 ) + (1 − cos θ ) and taking its derivative 2 e e e θi ← θi + K(θm − θi ) (4) confirms that V̇ ≤ 0. Hence, V is indeed a Lyapunov function for the system (1). where FinE-R 2015 Page 24 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Fig. 6. Planned path of the robot shown in blue designed to enable this objective falls under the fields of ma- chine learning and artificial intelligence (AI). This includes approaches such as neural networks [14], support vector machines [10], evolutionary robotics [20], reinforcement learning [23], probabilistic methods [24], etc., which are used for a range of tasks such as object recognition, clustering, path planning, optimal control, localization and mapping etc. Although most algorithms can be tested out in a simulated environment, the uncertainties and nuances associated with a real environment limit the utility of simulated solutions, es- Fig. 5. Overhead view of the robots at different times during the heading pecially in fields such as evolutionary robotics [7], [8], [12]. consensus. The robots are initially unaligned, but arrive at a consensus on heading at t=10s A more thorough validation necessitates implementation of these algorithms on real-world physical systems. To illustrate some of the primary issues and requirements of a platform for use in AI, a path planning task is demon- X strated using the standard A-star algorithm [13]. The scenario θm = θi /N (5) involves planning a path from the starting position of a Here, θi is the heading of an individual agent, K is a constant robot to a target position using a previously constructed and θm is the mean heading of all N agents. These updates infra-red sensor based map of the environment as shown in are performed on each agent of the swarm till the heading Figure 6. Most of the arena shown in Figure 6 has been converges to the same value. mapped; the mapped areas are shown as black patches along the boundaries of the arena. The planned path (shown in As seen from Figure 5, the robots eventually orient them- blue) is generated by the A-star algorithm, using cost and selves in a direction that was determined using the consensus heuristic functions. The heuristic function is proportional to algorithm. the euclidean distance from the target position. Some of the Swarm algorithms rely on the current information important issues while implementing this algorithm for path regarding other members. So, delays in communication planning and for AI applications in general, are listed below: can cause swarming algorithms to diverge instead of Computing Resources: The A-star algorithm could have converging. As mentioned in section III, performing also been implemented locally on the robots. However, this routines to maintain the integrity of the exchanged data would have implied storing a copy of the IR map and is critical. The communication rate should ideally be performing the A-star algorithm locally using this map. This independent of the number of agents involved. Usually, in may not have been feasible due to the limited on-board communication systems, it is common practice to make use memory and computational complexity of the algorithm. of communication data buffers. For swarm applications, Also, for path planning, the map of the environment gets one should ensure the latest data is picked out either by larger as the space explored gets larger. Accordingly, the matching similarity in time-stamps or by refreshing the A-star path planning could potentially become more com- buffer before each agent is queried. putationally intensive. So setting a resolution parameter (or deciding to have one) is recommended before starting the mapping. D. Application in Artificial Intelligence In general, sufficient computing resources will be needed to One of the long-term goals of robotics is to develop carry out AI algorithms. Having a thin client system that systems that are capable of adapting to unknown and unstruc- offloads the computational effort to a central server helps tured environments autonomously. The general set of tools deal with the high computational costs imposed due to high FinE-R 2015 Page 25 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 dimensionality in certain problems. [4] F. Arvin, Kh. Samsudin, A. R. Ramli, and M. Bekravi. Imitation Dealing with the real world: Readings from the robot’s of honeybee aggregation with collective behavior of swarm robots. International Journal of Computational Intelligence Systems, 4:739– sensors involve an inherent component of noise, even after 748, 2011. they have been calibrated. In addition, it may be common [5] P. Berman and J.A. Garay. Asymptotically optimal distributed con- for one to consider an agent as a point moving through sensus (extended abstract). In Proceedings of the 16th International Colloquium on Automata, Languages and Programming, ICALP ’89, a space during the simulation stage. In reality, the agent pages 80–94, London, 1989. Springer-Verlag. cannot be considered to be a point in space.These aspects [6] J. Betthauser, D. Benavides, J. Schornick, N. O Hara, J. Patel, J. Cole, of reality could potentially hinder one from achieving the and E. Lobaton. Wolfbot: A distributed mobile sensing platform for research and education. In Proceedings Conference of the American desired results. Taking the physical dimensions of its body Society for Engineering Education (ASEE Zone 1), pages 1–8. IEEE, into consideration is one way to counter this. 2014. In the above mentioned case of path planning, noise points [7] H.J. Chiel and R.D. Beer. The brain has a body: adaptive behavior emerges from interactions of nervous system, body and environment. were first removed using median filters from the raw map. Trends in neurosciences, 20:553–557, 1997. Also, in order to introduce a safety margin, (and to com- [8] D. Cliff, P. Husbands, and I. Harvey. Explorations in evolutionary pensate for the loss of information introduced due to blind robotics. Adaptive Behavior, 2:73–110, 1993. [9] Peter I. Corke. Robotics, Vision & Control: Fundamental Algorithms spots as well as due to the resolution parameter) the detected in Matlab. Springer, 2011. obstacles were inflated by exaggerating their size. The safety [10] N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector margin could also be expanded to account for the dimensions Machines: And Other Kernel-based Learning Methods. Cambridge University Press, New York, 2000. of the robot body, thus ensuring a smoother transition from [11] S. English, J. Gough, A. Johnson, R. Spanton, J. Sun, R. Crowder, simulation to reality, without having to explicitly encode and K.P. Zauner. Strategies for maintaining large robot communities. the details of the body geometry during simulation, thereby Artificial Life XI, pages 763–763, 2008. [12] D. Floreano and L. Keller. Evolution of adaptive behaviour in robots saving considerable development time. It should be noted by means of darwinian selection. PLoS Biol, 8:1–8, 2010. that excessive expansion of obstacles could lead to failure of [13] P.E. Hart, N.J. Nilsson, and B. Raphael. A formal basis for the heuristic A-star to find a valid path to the goal. determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4:100–107, 1968. Noise, and mismatches between virtual and real environ- [14] S. Haykin. Neural Networks: A Comprehensive Foundation. Prentice ments is inevitable. Noise removal using appropriate filters, Hall PTR, Upper Saddle River, NJ, USA, 2nd edition, 1998. and compensating for mismatches with the real world by [15] S.J. Julier, J.K. Uhlmann, and H.F. Durrant-Whyte. A new approach for filtering nonlinear systems. In Proceedings of the American Control erring on the side of caution may be considered to be useful Conference, volume 3, pages 1628–1632, 1995. measures to ensure a smooth transition from virtual to real [16] Y. Kanayama, Y. Kimura, F. Miyazaki, and T. Noguchi. A stable track- environments. ing control method for an autonomous mobile robot. In Proceedings of 1990 IEEE International Conference on Robotics and Automation, V. C ONCLUSION pages 384–389, 1990. [17] T. Lauwers and I. Nourbakhsh. Designing the finch: Creating a This paper discussed solution strategies to address a set of robot aligned to computer science concepts. In AAAI Symposium common problems which may be encountered while adapting on Educational Advances in Artificial Intelligence, pages 1902–1907, 2010. a low-cost robotics platform such as the EvoBot for research. [18] F. Mondada, M. Bonani, X. Raemy, J. Pugh, Ch. Cianci, A. Klaptocz, Some insights into the factors to be considered while choos- S. Magnenat, J.Ch. Zufferey, D. Floreano, and A. Martinoli. The e- ing the sensing, control and communication capabilities of puck, a robot designed for education in engineering. In Proceedings of the 9th conference on autonomous robot systems and competitions, low-cost platforms were discussed. The eventual success of volume 1, pages 59–65, 2009. our approach was described in the context of four diverse yet [19] F. Mondada, E. Franzi, and A. Guignard. The development of khepera. common robotics tasks - state estimation, real-time control, a In Experiments with the Mini-Robot Khepera, Proceedings of the First International Khepera Workshop, pages 7–14, 1999. basic swarm consensus application and a path planning task [20] S. Nolfi and D. Floreano. Evolutionary Robotics: The Biol- that were carried out using the EvoBots platform. ogy,Intelligence,and Technology. MIT Press, Cambridge, MA, USA, Some of the potential issues faced by end users who are 2000. [21] W.W. Peterson and D.T. Brown. Cyclic codes for error detection. not familiar with real-world implementation of algorithms Proceedings of the IRE, 49(1):228–235, Jan 1961. are also discussed and potential work-arounds for a smooth [22] M. Rubenstein, Ch. Ahler, and R. Nagpal. Kilobot: A low cost transition from simulation to reality were suggested. The scalable robot system for collective behaviors. In Proceedings IEEE International Conference onRobotics and Automation (ICRA), pages points discussed in this paper aim to serve as a guide to 3293–3298, 2012. groups who intend to adapt similar low-cost platforms in [23] R.S. Sutton and A.G. Barto. Introduction to Reinforcement Learning. the future, as well as to researchers working on theoretical MIT Press, Cambridge, MA, USA, 1st edition, 1998. [24] S. Thrun, W. Burgard, and D. Fox. Probabilistic Robotics (Intelligent aspects of robotics, who intend to validate their algorithms Robotics and Autonomous Agents). The MIT Press, 2005. through real-world implementation. [25] Trek SA. Ai-ball specs, 2014. R EFERENCES [1] ROS: Robot operating system. http://www.ros.org/. [2] Adept Mobile Robots. AmigoBot robot for education and research, 2014. [3] F. Arvin, J. C. Murray, L. Shi, C. Zhang, and Sh. Yue. Development of an autonomous micro robot for swarm robotics. In Proceedings IEEE International Conference on Mechatronics and Automation (ICMA), pages 635–640, 2014. FinE-R 2015 Page 26 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Improvements and considerations related to human-robot interaction in the design of a new version of the robotic head Muecas Felipe Cid1 , Luis J. Manso2 and Pedro Núñez2 Abstract— In this article the different decision-making that mesh model that analyzes the facial deformations [2], the were followed in the design of the expressive robotic head changes in the prosody of the voice [3] or a model that Muecas are explained. Muecas is a system with a human- tracks changes in the skeleton of the user [4]. caricatured shape, equipped with a pair of robotic eyes, eyebrows, neck and mouth. The main goal in the design was The main contribution of this work is related to an to provide the robot with basic skills for an affective human- extension of the publication that describes in detail the robot interaction, where the emphaty and attention plays an robotic head Muecas in [3]. This extension has as objective important factor. When developing this robotic head, it was to describe the changes that are being carried out to develop necessary to study a number of parameters and characteristics a second version of the robotic head that consider aspects related to human anatomy and psychology, as well as other similar robotic heads or the opinions of experts. All the results of different areas of the sciences, such as: psychology, of these study are pointed out in this paper. Throughout this mechanics, robotics, the morphology and the anthropomor- work, the step followed in the design in conjuction with the phism. Despite the importance of functionality or external main conclusions drawn from it, are explained. We want that appearance of the robot, there are other theories that support our work helps researchers in this field in their decision making. the design of robots based on the different processes of I. INTRODUCTION learning [5] or action that will be a platform. These designs have as objective to improve the exchange of information The development of robotic platforms with anthropomor- [6], empathy with the user [7], and increase the number of phic forms is an area of interest within the social robotics possible actions with certain elements of the environment. during recent decades. This is due to the rise of the algo- This paper is organized as follows: After discussing previ- rithms based on the recognition of emotions, objects and ous works in the literature related to design of robotic heads., manipulation algorithms based on robotic hands. Within this in Section II. In the Section III, presents an overview of the topic, the use of natural language in robotics is presented as a improvements and considerations to take into account in the basis for improving the interaction with the people. Being the robotic head Muecas. Next, Section IV described the new primary motivation for the development of systems capable mobile elements, degrees of freedom and facial expressions. of recognizing and imitating not only the emotions, but also In Section V, the multimodal systems are described, and the behavior. Therefore, it is necessary an evolution of the finally, Section VI summarizes the conclusions and future current systems and robots, to remain in the field of human works of the approach. robot interaction. The current robotic heads are presented as platforms II. RELATED WORKS developed directly for the acquisition of information from the environment or objects, by means of systems that use The design of robots with human characteristics, so that movement or actions similar to the human. However, these they can communicate by means of natural language in- heads tend to possess physiological characteristics similar to teraction is an area in constant progress. Currently the the human, which allows them to improve the interaction development of robotic heads is based on the interaction by means of multiple modalities of communication based of different modality, whether facial expressions, voice or on natural language. In the social robots, these physiological visual messages or body language. However, the most well- characteristics related to the anthropomorphism [1], allow known cases at the level of research are: iCub [8], kismet [9], the development of systems of recognition and imitation ROMAN[10][11], KHH [12], WE-4RII [13], SAYA [14][15], based on different modalities of communication, such as: Barthoc [16], among others. facial expressions, speech and body language. Therefore, In the development of robotic heads there are several these modalities are a constant source of information on the works that describe the changes and improvements to older emotions and intentions of the users in the communication, versions of a prototype or product. Examples of this are the either through methods that acquire characteristics from a robotic heads: Flobi [17], Barthoc, ROMAN, SAYA, among others. Where, ROMAN and Barthoc describe step by step *This work was not supported by any organization 1 Felipe Cid is with Institute of Electrical and Electronics, evolution to achieve a similar appearance to the human, Universidad Austral de Chile, Independencia 460 Valdivia, Chile. while, Flobi describes in detail the development of this plat- felipe.cid@uach.cl form to get an appearance caricatured and anthropomorphic. 2 Luis J. Manso and Pedro. Núñez are members of the Robotics and Artificial Vision Lab. Robolab Group, University of Extremadura, Spain. In the design of the different platforms used for the lmanso@unex.es; pnuntru@unex.es interaction it is possible to analyze four different types of FinE-R 2015 Page 27 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Fig. 1. Overview of the system of recognition and imitation of natural language by the robotic head Muecas. (a) (b) Fig. 2. Different perspectives of the robotic head Muecas appearances, such as: Anthropomorphic, Zoomorphic, cari- catured and Functional. Among these types of appearance, there are some that are designed to avoid the Uncanny Then it will describe describe the major consideration in Valley [18], creating another type of relations. For example, the design, which hope to improve much interaction with the Zoomorphic focuses on the natural relationship between users. Also, the technical considerations that will provide to an animal and a human. Meanwhile, the anthropomorphic the robotic head Muecas, qualities to represent a movement appearances are looking for an advanced understanding by similar to a human. means of natural language. In the case of the robotic head Muecas was designed with a caricatured and anthropomor- • One of the first considerations would be the incor- phic concept to support the interaction with young people, poration of new elements and degrees of freedom to but keeping a bit of functional appearance. Finally, Table I the robotic head Muecas, in order to represent a body shows a comparison of the different existing robotic heads movement and behavior more similar to that of human and the head Muecas. beings by part of the robot. • The implementation of more systems natural language III. SYSTEM OVERVIEW recognition, through new modalities that deliver a better In this section, the design of a new version of the robotic understanding of human communication. Since this head Muecas, is presented. The original version of this robot only analyzes facial expressions and speech (voice robotic head was published in [3], therefore, this work and content), as explained in [3]. will be limited to explain some new improvements and • The incorporation of learning processes to improve considerations in order to improve the existing platform. interactions uncontrolled in the implementation of tests, Muecas is a robotic head designed to study new meth- creation of database and algorithms based on Affor- ods of human-robot interaction based on natural language, dances [19]. through different modalities of communication, such as facial • One factor to consider in this new design, it is the ability expressions, speech, body language, among others. For this to integrate a localization system, by means of audio, reason, this platform uses multiple systems of recognition vision, and depth (RGB-D). With this, it is expected that and imitation of natural language (See Figure 1), that take this robotic platform be able to locate the user to follow advantage of the 12 degrees of freedom to imitate the him with the cameras at all times, of form invariant to movement of the head in a similar manner to human, through the obstacles or problems that may occur. elements such as: neck (four), mouth (one), eyes (three) • The development of new facial expressions, due to the and eyebrows (four). Besides, a series of sensors are used implementation of new mobile elements in the robotic to acquire a large amount of visual, auditory, and depth head. information, being this information necessary for the systems • One aspect to analyze is the integration of this robotic of recognition and imitation. Figure 2 shows the robotic head head in a humanoid robot or mobile platform, with Muecas from different perspectives. emphasis on how it affects the interaction with the user. The main aspects to consider in this new version would In addition, take into consideration the fact to take full be associated with incorporation of new degrees of freedom advantage of the capabilities of the robotic head with through new mobile elements and the modification of ex- respect to the main robot manipulator isting ones. The most important changes are related to the • In order to avoid the Uncanny Valley [20], a human- elements, such as the eyelids, one degree more of freedom in like appearance is maintained, but caricatured by covers the neck, new facial expressions and new covers to change with different designs. Meanwhile, maintains a similar the external appearance of the robot. The implementation of movement to humans, but not identical. these changes, aims to improve the human-robot interaction through new movements and a change in the external appear- In later sections these aspects will be analyzed one by ance of the platform, but maintaining the anthropomorphic one, explaining the improvements to be made in the current and caricatured concept which defines this robotic head. version of Muecas. FinE-R 2015 Page 28 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Robotic Head Neck Eyebrow Eye Mouth FACS Stereo Stereo RGBD Inertial Apperance DoF DoF DoF DOF Vision Audio Sensor Sensor WE-4RII 4 8 3 5 yes yes yes no yes Ant. ROMAN 4 2 2 1 yes yes no no yes Ant. SAYA 3 no 2 1 yes no yes no yes Ant. KHH 4 no 3 no no yes yes no yes Tech. BARTHOC 4 2 3 3 yes yes no no no Ant. iCub 3 LED 3 LED no yes yes no yes Ant. ICAT 2 2 0 5 no no yes no no Zoo Muecas 4 4 3 1 yes yes yes yes yes Ant. TABLE I C OMPARISON OF DIFFERENT ROBOTIC HEADS IN THE LITERATURE (A NT.- A NTHROPOMORPHIC , T ECH .- T ECHNOMORPHIC , Z OO . - Z OOMORPHIC ) (I NFORMATION COLLECTED FROM : [3]). IV. PROBLEMS IN THE CURRENT DESIGN The old design of Muecas was presented as a platform with anthropomorphic features that was aimed at improving the empathy and naturalness of interactions with different types of users trained and untrained. However, the original design of the head contained many more elements than those described in the above publication [3]. An example of these missing elements are the same eyelids, given that Muecas has a oriented architecture for imitation of facial expressions, the Fig. 3. Future changes in the design absence of this mobile component is presented as an impor- tant gap. In this way, the eyelids are an important source of information related to the emotions of users in a interaction A. Robotic eyelids (mainly associated with the intensity of emotions) that allows The eyelids are a major challenge in presenting a total us to integrate new facial expressions or emotional states, as change in the current structure of the eyes, since they need described in the works of P. Ekman [21]. For this reason, a minimum of two degrees of freedom to generate new in order to improve the functionality in different types of facial expressions (See Figure 13). These degrees of freedom interactions, have been gradually integrated different aspects will be implemented through two servomotors HITEC HS- that were not incorporated or raised in the design, such as: 45HB (one in each eye), being the same motors used in the mobile elements, multimodal recognition systems, new facial degrees of freedom of the eyebrows. This new movement expressions, appearance, among others. related to the eyelids needs of a complex system that would The changes in this new version considered some features allow an individual movement and synchronized movements that other similar robots possessed and some little analyzed in each eye, as shown in Figure 4. For this reason, it is trends in the current robotic. On the one hand, were designed developing a system responsible for moving the eyelids, new degrees of freedom associated with the neck and the through a mechanism based on folding layers covering the eyelids, which allow to deliver more information through eyeball from the top to the bottom (See Figure 5(a)). body language. The latter being an important factor, since Within the literature, there are many works that demon- the eyelids delivered much of the information related to the strate in practical ways that the eyelids are an essential part in intensity of facial expressions. On the other hand, within the the generation of facial expressions and the transmission of current robotic heads there are very few that are based on visual information [22][16]. However, in order to produce a works from psychological studies, to support the possible greater degree of expressiveness and facial expressions more facial expressions or the basic emotions that should generate complex and realistic on the part of Muecas, is required of a robot with anthropomorphic features. In this same way, four degrees of freedom, as shown in Figure 5(b) and 6. we analyzed the possibility of changing the appearance These degrees of freedom allow generating a movement that of Muecas, with the objective to promote its appearance the user perceives as similar to humans, which is necessary caricatured by means of covers. These covers allow the for the generation of facial expressions based on FACS [21] robot, the development of interactions closest and prevent that classifies these movements (e.g. AU45 or AU46). manipulation of the mechanical elements by children. One of the main advantages in the implementation of the This paper is divided into subsections that describe differ- eyelids, is the ability to dramatically increase the emotional ent aspects to consider in the design of new types of social expressiveness of this platform. Due to that greater facial anthropomorphic robots (See Figure 3). expressiveness is associated mainly with high levels of exci- FinE-R 2015 Page 29 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 (a) (b) Fig. 6. a) Example of the importance of the eyelids in the expressiveness of emotions of high arousal; and b) Example of a basic gesture of body language. Fig. 4. Image of the degrees of freedom present in the eyelids. (a) (b) Fig. 7. Design of degree of freedom related to the pitch.: a) Pitch A; and c) Pitch B. the degrees of freedom in the existing robotic head, including (a) (b) the new degree of freedom associated with a second pitch. Fig. 5. Design of the system of movement of the eyelids. Finally, some examples of other platforms that implemented this movement, are: ROMAN [10][11], Barthoc [16], among others. tation in the emotions transmitted by the robot. Figure 6(a) shows an example of how the eyelids affect the perception C. mouth of some facial expressions. Meanwhile, Figure 6(b) shows The mouth plays a significant role in how we perceive how the eyelids give the possibility to make some gestures speech of an announcer in a conversation. This is due to the typical of natural language as a simple Wink. fact that much of the information, even auditory, requires a feedback through visual information. This is known as the B. neck McGurck effect [23]. However, the current design of robotic The neck of the robotic head Muecas is one of the more mouth does not allow a full perception of this effect. Given complex parts in the design and implemented these plat- that this robotic mouth has only one degree of freedom, forms. Since this element represents the key to the imitation with the only function of opening and closing the mouth of the movements of the human body language, introducing (See Figure 9(a)). For this reason, it has established the a system that recreates the movement of the human neck design of a robotic mouth that has the minimum capabilities in such a way as to be recognizable movements by a user. of deformation, either in the opening, the contraction of However, these movement are generated in a different way the contours, and the extension of the lips forward (See to humans, which makes it possible to avoid the Uncanny Figure 9(b)). However, these movements are in development Valley [18]. Because the design of this robot does not expect because they are a challenge to the current physical design. imitate identically human movements, but generate similar Finally, it is important to mention that within most of the representations which are recognizable by users. robotic heads, the incorporation of these mobile elements However, after a series of analyzes, it was decided to is not considered necessary, despite its importance in a implement a movement missing on the base, known as a communication based on auditory information. second pitch. This movement is basic in users, and allows a greater angle of inclination for the head that the first pitch. D. External appearance Figure 7(b) shows the implementation of this second Pich In the design of the robotic head Muecas, external ap- in a 3D model of the physical components of the neck, pearance plays a crucial role to perform tasks of emotional whereas in Figure 7(a) is shown the first pitch. In each of interaction with users. For this reason, is not only is intended the mobile elements of the neck were used DC-micromotor to catalog the appearance of the robot as anthropomorphic 1724-024 SR (Encoder IE2-16 with 76:1 gear ratio), even in and caricatured, but develop a series of covers that allow the new degree of freedom. Figure 8 shows the structure of experimenting with the human robot interaction. FinE-R 2015 Page 30 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 (a) (b) Fig. 8. Image of the degrees of freedom present in the neck, more the (c) degrees of freedom associated to a second pitch in the base of support. Fig. 10. Different prototypes of the cover for the robotic head Muecas.: a) First prototype of cover; b) Second prototype of cover. and c) Final prototype of cover. (a) (b) Fig. 9. a) Current robotic mouth of Muecas with a one degree of freedom; and B) Movements that should be emulated by Muecas, such as: contraction of the lip corners (blue), extension of the lips forward (purple), and the opening of the mouth (red). Figure 10 shows the covers designed for the robotic head Muecas through 3D models. The objective of these covers is (a) (b) to improve the level of empathy and care of people, mainly Fig. 11. a) Current image of the robot Loki.; and b) 3D model of the new cover of the robot Loki. (Image (a) obtained from the publication: [19]) children, and bring them to a higher level of interaction without resorting to too many humanoid forms similar to the human that will lead us to the Uncanny Valley [18].For this reason, the choice of shapes similar to toys or cartoons seek to promote better interaction, as has been demonstrated is identifiable as similar to humans but with a different in multiple works as: Kismet [9], Flobi [17], iCat [24], Probo kinematic. However, the constant interaction of this head [25], among others. with non-trained users and the general public, it allowed Currently, the autonomous robot Loki [26] has integrated us to analyze the opinion of the user with regard to what the robotic head Muecas, in order to improve the user’s elements are necessary to improve the current design. Thus, response before a robot of large proportions. Figure 11(a) this feedback directed us toward a platform containing more shows the robot Loki, and Figure 11(b) shows the cover mobile elements such as: eyelids, a second pitch, and more ready to Loki. Given that this cover expects to achieve the movements in the mouth. For this reason, new degrees of same result that the prototypes described above, through an freedom need a series of actuators responsible for carrying increase in the empathy and care in a user interaction with out this function. Table II describes the current elements with the robot. the degrees of freedom, together to the new actuators and the degrees of freedom that are incorporated into this platform. E. New sensors and actuators In the case of the sensors, this new version of Muecas only The original design of the robotic head Muecas possessed incorporates two microphones in the positions of the ears of 12 degrees of freedom associated with a number of actuators the robotic head, in order to acquire auditory information that allow the movement of the neck (Pitch, Roll and Yaw), for the development of tracking systems for users through eyes (pitch and roll), eyebrows (Tilt and Pan), and mouth the audio. Table III summarizes the current and new devices (aperture). Where is prioritized a natural movement that related to information visual, auditory and depth. FinE-R 2015 Page 31 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Element DoF Motors Eyes 3 3 motors divided into 2x Faulhaber LM- 1247 linear DC-servomotor (Pan) and 1 x Faulhaber LM-2070 lineal DC-servomotor (Tilt). Eyebrows 4 4 HITEC HS-45HB servomotors Neck 4+1 4+1 DC-micromotor 1724-024 SR (Encoder IE2-16 with 76:1 gear ratio) Mouth 1 1 HITEC HS-45HB servomotors. Eyelids 2 2 HITEC HS-45HB servomotors. TABLE II D ESCRIPTION OF THE DIFFERENT DEGREES OF FREEDOM AND MOTORS OF THE ELEMENTS OF THE ROBOTIC HEAD M UECAS . (I NFORMATION Fig. 12. Facial expressions recognized and imitated by the robotic head Muecas. (Image obtained from the publication [3]) OBTAINED FROM : [3]). Type Sensors Position Stereo Vision Two cameras Point Grey Eyes. Dragonfly2 IEEE-1394 Stereo Audio Two microphones Sides of the head Input connected to the pre- (ears). amplifier M-AUDIO MOBILEPRE USB Stereo Audio Two speakers connected Base of the head. (a) (b) (c) Output to the pre-amplifier M- Fig. 13. New facial expressions: a) Bored; b) Lonely; and c) Confused. AUDIO MOBILEPRE USB Inertial Sensor PhidgetSpatial 1042–0 Back of the head. 3/3/3 to present an idea or concept to the user, whether the RGBD Sensor Microsoft Kinect Sen- On the head. information that delivery is not clear and it causes problems sor or Xtion PRO LIVE to the robot. With regard to emotional states in Table IV, it is Sensor (optional) important to mention that the names of some states may vary TABLE III between emotional theories, for example: F ear is Af raid D ESCRIPTION OF THE DIFFERENT SENSORS OF THE ROBOTIC HEAD in the theory of Russell. M UECAS . (I NFORMATION OBTAINED FROM : [3]). Emotional State Ekman [21] Russell [27] Plutchik [28] Sadness x x x Happiness x x x Fear x x x F. New Emotional States Anger x x x Neutral – – – The development of the human-robot interaction is present Bored x x in the day to day, for this reason it is necessary to extend Depressed x x Confused – – – the capabilities of the current robots, in order to improve the skills of verbal, visual and emotional communication. With TABLE IV this objective, we carried out a study of the state of the art E MOTIONAL STATES OF THE SYSTEM VS . T HEORIES OF BASIC of the different theories about what are the basic emotions EMOTIONS IN THE FIELD OF PSYCHOLOGY. in human beings. The results of this study allowed for the selection of three theories of emotions studied and used in the robotics, such: the theory of Ekman [21], Russell [27] These emotional states can be regarded as basic in the and Plutchik [28]. These three theories represent the basis theories of the emotions, because they can be found in several for the choice of the emotional states with what will work works with different points of view, such as: Arnold [29], the robotic head, which are: happiness, fear, sadness, anger Frijda [30], Gray [31], Izard [32], James [33], McDougall and neutral. Where each emotional state is associated with [34], Oatley [35], Tomkins [36], among others. Where the a facial expression that will be recognized and imitated by emotions are classified by the inclusion criteria given by the the robotic head. Figure 12 shows these five emotional states author, as for example: Ekman considered a small group with their respective facial expressions, which are performed of universal emotions, Plutchik classifies the emotions with by the user and imitated by Muecas. regard to adaptive biological processes, or Tomskins that In the Table IV shows the correspondence of the emotional considers the density of neuronal activity. (the reader can states within the theories mentioned above. However, it is refer to some interesting works in [37]). Finally, Figure important to comment that the state N eutral is a state 13 shows the new emotional states through the simulator not associated with any emotion, because it indicates the InnerModelSimulator of the framework RoboComp [38]. absence of a predominant emotion. In the same way, the One aspect to take into consideration, in the case of the state Conf used is presented as an emotional state necessary robots, is that the artificial facial expressions should be more FinE-R 2015 Page 32 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 exaggerated, to avoid any type of emotional confusion or ambiguity in communication. This is due to the fact that communication systems based on natural language have a limited amount of facial expressions, associated with the low number of actuators that generate movements in the elements of the face. In contrast, a human has a large amount of facial muscles that generate distortions studied and classified by systems, such as the FACS [21]. V. MULTIMODAL SYSTEM The recognition systems of the robotic head Muecas [3], Fig. 14. The user interacts with the robot Loki and the robotic head Muecas can be divided into two basic systems: Facial expressions and speech. However, in order to improve the system of recognition of emotions, and analyze the most of the in- the experiment. Given that the conditions of the experiment formation related to natural language, a module capable of are focused on improving the interaction, and that Muecas analyzing the information of the body language through the only has a limited amount of facial expressions associated depth sensor of Muecas has been integrated. This system with different emotional states, the process of imitation will analyzes a model of the human skeleton by extracting 7 provide of exaggerated facial expressions and specific for features related to the movement, which allows to analyze each emotion (See Table V) that will be recognized by the the body language and estimate the emotional state of the user. user (more details in [4]). The output of this system provides This process of interaction, it is carried out by means of information relating to the AUs for each element of a facial verbal messages, such as: expression in the user. Thus, these AUs that are updated by • Muecas: The test has begun, please express a facial each expression allow the robot not only recognize, if not expression. also imitate these expressions in real time, in accordance • User: (The user performs a certain facial expression, with the relationship between the facial deformities and the trying to show a specific emotion) movements of the mobile elements of Muecas (See Table V). • Muecas: (The robot, to recognize the facial expression These elements of the robotic platform include: eyebrows, of the user, imitates the expression through the move- eyelids, eyes and mouth. ment of its mechanical components) • User: (Evaluates the success/error in the recognition of Emotion AUs Mobile components facial expression) of Muecas Neutral - - • Repeat this sequence until complete the experiment. Happy AU6-AU12-AU25 Eyebrows-Eyelids- Finally, this process is repeated for all emotional states Eyes-Mouth Sad AU1-AU4-AU15-AU17 Eyebrows-Eyelids- required in 20 repetitions. Eyes An example of systems of learning related to robotic Fear AU1-AU4-AU20-AU25 Eyebrows-Eyelids- head Muecas is [40]. In this case, we used Muecas to Mouth Anger AU4-AU7-AU17-AU23-AU24 Eyebrows-Eyelids perform the interaction with the user, while acquires the emotional information of the user and about the objects in the TABLE V environment. Given that this information from the user and M OVEMENTS OF THE ROBOTIC HEAD Muecas, AND MOBILE ELEMENTS the objects, are the basis for the use of affordances. Finally, ASSOCIATED WITH EACH EMOTIONAL STATE (TABLE OBTAINED FROM Figure 14 shows an example of a complex interaction that THE PUBLICATION : [3]). includes facial expressions, and robotic manipulators. VI. CONCLUSIONS In the case of multimodal localization systems, it is The robotic heads need a process of development and working to integrate the tracking system of the model of constant evolution, which will enable them to improve and mesh Candide-3 [2] with a localization system based on adapt in a better way to users. Either through non-invasive audio, as the submitted by [39]. methods, interactions based on the natural language or forms more user-friendly. Currently, the robotic heads do not in- A. Learning by imitation in robotic heads corporate certain fundamentals as psychological theories of One of the main attributes of the systems for recognition the emotions or the effect McGurck that seek to support of emotions associated with this platform, is the property the compression of the human behavior through the analysis of self-training with the robotic head Muecas.This process and observation. However, this work uses these theories uses TTS and ASR systems (explained in [3]), to gener- related to emotions and how the human beings communicate ate a database of information of learning for a system of to describe new aspects that are relevant to the design recognition of facial expressions based on Candide-3 [2], of anthropomorphic robotic heads, oriented to the human- without the need for a supervisor or a third party during robot interaction. Describing what are the enhancements and FinE-R 2015 Page 33 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 considerations that were obtained after working with the first [11] K. Berns; J. Hirth. Control of facial expressions of the humanoid version of Muecas. robot head ROMAN. In Proc. of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3119-3124, 2006. In relation to other robotic heads, Muecas can be consid- [12] T. Asfour; et al. The Karlsruhe Humanoid Head, In 8th IEEE-RAS ered a robot designed for the human-robot interaction and International Conference on Humanoid Robots, Humanoids 2008, pp. the development of learning by imitation based on natural 447 - 453, 2008. [13] M. Zecca; T. Chaminade; M.A. Umilta; K. Itoh; M. Saito; N. Endo. language. Due to the different types of aspects evaluated Emotional Expression Humanoid Robot WE-4RII- Evaluation of the in this document, have as goal replicate the movement and perception of facial emotional expressions by using fMRI, In Robotics emotions in a different way to humans, but that is easily and Mechatronics Conference ROBOMEC2007, pp. 2A1-O10, 2007. [14] T. Hashimoto; S. Hitramatsu; T. Tsuji and H. Kobayashi. Development recognizable by the user. On the one hand, this platform of the Face Robot SAYA for Rich Facial Expressions. In Proc. of 2006 focuses on the systems to improve the interaction through SICE-ICASE International Joint Conference, pp. 5423-5428, 2006. behaviors and actions similar to the human based on learning [15] T. Hashimoto and H. Kobayashi. Study on Natural Head Motion in Waiting State with Receptionist Robot SAYA that has human-like by imitation. On the other hand, the hardware focuses on Appearance. In IEEE Workshop on Robotic Intelligence in Informa- elements such as the movements of the neck and eyes that tionally Structured Space (RIISS ’09) , pp. 93 - 98, 2009. allow a user tracking by cameras (within the eyes), avoiding [16] T.P. Spexard; M. Hanheide; G. Sagerer. Human-Oriented Interaction With an Anthropomorphic Robot. In IEEE Transaction on Robotics, sudden movements or jumps by means of linear motors. 5, pp 852-862, 2007. In this paper, one of the main topics that were evaluated [17] I. Lütkebohle; F. Hegel; S. Schulz; M. Hackel; B. Wrede; S. was the incorporation of new degrees of freedom in elements, Wachsmuth; and G. Sagerer; The Bielefeld Anthropomorphic Robot Head ”Flobi”, In Proceedings of the IEEE International Conference such as: neck, eyelids and mouth. Besides, the integration of on Robotics and Automation (ICRA), 2010. tracking systems and a new external appearance based on [18] M. Mori; Bukimi no tani - The uncanny valley. In Energy,7(4), pp. covers, aim to improve the interaction through communica- 33-35, 1970. [19] F. Cid, A.J. Palomino and P. Núñez, ” A new Paradigm for Learning tion. Affective Behavior: Emotional Affordances in Human Robot Interac- Future work is aimed at implementing the considerations tion”, In Proc. of the XIV Workshop on Physical Agents - WAF2013, described in this paper, through the integration of an evalu- pp. 47–52, 2013. [20] J. Seyama, and R. Nagayama; ”The Uncanny Valley: Effect of Realism ation process based on interaction with non-trained users in on the Impression of Artificial Human Faces”, In Journal Presence: uncontrolled environments. Teleoperators and Virtual Environments, 16(4), pp. 337-351, 2007. [21] P. Ekman, R. Levenson, and W. Friesen. ”Autonomic Nervous System ACKNOWLEDGMENT Activity Distinguishes Among Emotions”. In Science, 221(4616), pp. 1208-1210, 1983. This work has been partially supported by the MICINN [22] J. Hirth and K. Berns, ”Concept for Behavior Generation for the Humanoid Robot Head ROMAN based on Habits of Interaction”, In Project TIN2012-38079-C03-01, and by the Institute of Elec- Proc. of International Conference on Humanoid Robots, pp. 360-365, trical and Electronics of the Universidad Austral de Chile. 2007. [23] T. Chen and R.Rao, ”Audio-Visual Integration in multimodal Com- R EFERENCES munication”. In Proceedings of the IEEE, 86(5), pp. 837–852, 1998. [24] A. van Breemen; X. Yan and B. Meerbeek; iCat: an animated user- [1] S.E. Guthrie, Anthropomorphism: A Definition and a Theory, in interface robot with personality; In AAMAS ’05: Proceedings of Mitchell R.W.; et al. Anthropomorphism, anecdotes, and animals. the fourth international joint conference on Autonomous agents and Albany: States University of New York Press. USA. multiagent systems, USA:ACM, pp. 143-144, 2005. [2] P. Romero, F. Cid and P. Núñez. ”A Novel Real Time Facial Expres- [25] J. Saldien; K. Goris; S. Yilmazyildiz; W. Verhelst and D. Lefeber. sion Recognition system based on Candide-3 Reconstruction Model”. On the Design of the Huggable Robot Probo, In Journal of Physical In Proceedings of the XIV Workshop on Physycal Agents (WAF- Agents, 2(2), pp. 3-11, 2008. 2013), 2013. [26] P. Bustos, et al. ”Multimodal interaction with Loki”, In Proc. of [3] F. Cid, J. Moreno, P. Bustos and P. Núñez. ”Muecas: A Multi-Sensor Workshop of Physical Agents, pp. 53–60, 2013. Robotic Head for Affective Human Robot Interaction and Imitation”. [27] J. Russell, ”A Circumplex Model of Affect”. In Journal of Personality In Sensors 2014, pp. 7711–7737, 2014. and Social Psychology, 39(6), pp. 1161-1178, 1980. [4] C. Doblado, E. Mogena, F. Cid, L. V. Calderita and P. Núñez. ”RGB- [28] R. Plutchik, and H. Kellerman. ”A general psychoevolutionary theory D Database for Affective Multimodal Human-Robot Interaction”. In of emotion”, In Emotion: Theory, research, and experience: Vol. 1. Proceedings of the XIV Workshop on Physycal Agents, 2013. Theories of emotion, 1(2), pp. 3-33, 1980. [5] F. Hegel; S. Krach; T. Kircher; B. Wrede; G. Sagerer. Theory of Mind [29] M. Arnold, Emotions and personality. New York: Columbia University (ToM) on Robots: A Functional Neuroimaging Study, In Proc. of the Press, 1960. 3rd ACM7IEEE International Conference on Human Robot Interaction [30] N. Frijda, The emotions. New York: Cambridge University Press, (HRI), pp. 335-342, 2008. 1986. [6] M. Siegel, C. Breazeal and M.I. Norton, Persuasive Robotics: The [31] J. Gray, The neuropsychology of anxiety. Oxford:Oxford University influence of robot gender on human behavior, In 2009 IEEE/RSJ Press, 1982. International Conference on Intellegent Robots and Systems, pp. 2563- [32] C. Izard, The face of emotion. Appleton-Century-Crofts, 1971. 2568, 2009. [33] W. James, What is an emotion? Mind, 9(34), pp. 188-205, 1884. [7] A. Pavia et al. Caring for Agents that Care: Building Empathic Rela- [34] W. McDougall, An introduction to social psychology. Boston:Luce, tions with Synthetic Agents. In Third International Joint Conference 1926. on Autonomous Agents and Multiagents Systems, pp. 194-201, 2004. [35] K. Oatley and P. Johnsin-Laird, Towards a cognitive theory of emo- [8] R. Beira et al. ”Design of the Robot-Cub (iCub) Head”. In Proceedings tions. Cognition & Emotion, 1(1):2950, 1987. 2006 IEEE International Conference on Robotics and Automation, [36] S. Tomkins, Affect theory. Approaches to emotion, pp. 163-195, 1984. ICRA2006, pp. 94 - 100, 2006. [37] A. Ortony, and T. Turner, Whats basic about basic emotions?, Psy- [9] C. Breazeal. Designing Sociable Robots. Cambridge, MA: MIT Press, chology Review, 97(2), pp. 315-331. 1990. 2002. [38] L. Manso, P. Bachiller, P. Bustos, P. Núñez, R. Cintas, and L. Calderita, [10] J. Hirth; N. Schmitz; K. Berns; Emotional Architecture for the Hu- ”Robocomp: a tool-based robotics framework”. In Simulation, Mod- manoid Robot Head ROMAN, In 2007 IEEE International Conference eling and Programming for Autonomous Robots, SIMPAR2010, pp. on Robotics and Automation, pp. 2150-2155, 2007. 251–262, 2010. FinE-R 2015 Page 34 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 [39] R. Viciana-Abad; R. Marfil; J.M. Perez-Lorenzo; J.P. Bandera; A. Romero-Garces; P. Reche-Lopez, ”Audio-Visual Perception System for a Humanoid Robotic Head”, In Sensors 2014, 14, pp. 9522–9545, 2014. [40] F. Cid and P. Núñez. ”Learning Emotional Affordances based on Affective Elements in Human-Robot Interaction Scenarios”. WAF 2014. XV Workshop of Pysical Agents,2014. FinE-R 2015 Page 35 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Lessons from the Design and Testing of a Novel Spring Powered Passive Robot Joint Joel Stephen Short1 , Aun Neow Poo2 , Chow Yin Lai3 , and Pey Yuen Tao3 , Marcelo H Ang Jr2 Abstract— The design, assembly, and testing of a new tor- by [1] and expanded by [2], there arose a need for an sional spring joint for use in underactuated robots is presented. underactuated robot, for testing of the method proposed The joint can use an array of spring sizes and is able to adjust in [3]. The motivation behind designing and building an the spring offset and preload independently. This work outlines the design process with details on the troubles faced and lessons underactuated robot was twofold, first it would provide a learned from multiple redesigns. experimental platform to test the theoretical system inversion method mentioned above, and second, it would give insight I. INTRODUCTION into the general capabilities and usefulness of such a robot. The design of new mechanical parts and assemblies is an The robot is required to perform cyclic(repeating) tasks integral part of robotics research. Even when an engineer’s and is made up of two linkages in a planar arrangement. research is mainly theoretical, it is typically expected that the There is an actuator at the first joint and the torsional spring theory will be tested in an experimental setup, often requiring mechanism at the second joint, see Figure 1 for a simplified the design of specialized pieces and devices, either for model of the robot. The actuator and passive joint placement testing by themselves or inclusion in a larger robotic setup. ensures that the robot is underactuated but not completely While there exist many design and testing methodologies uncontrollable, as backed up by the general serial-link robot for mechanical and mechatronic parts and assemblies, when analysis done in [4]. There are many reliable sources to use seeking to create a one-off prototype there is normally not enough time for these long processes. The engineer must try to quickly design, build and test an assembly, being efficient y and using only as much time as is necessary to ensure the design criteria is achieved. And this all must be done without τ1 running into dead ends or overly difficult problems during L1 any stage of the build-up. This work presents the design and build process of a m1 torsional spring joint with a special emphasis on the problems encountered and the lessons learned. A short background sets the stage for a discussion of the design goals and the resulting θ1 τs initial design. Then the assembly and testing are discussed L2 with a presentation of the problems, attempted solutions and final torsion joint layout. Lastly a discussion presents the key lessons learned from this experimental work and how they x m2 can contribute to prototype design in the future. A. Background θ2 While working on a stable system inversion method for the control of underactuated robots, a technique first investigated Fig. 1. 2DOF planar robot with torsional spring joint This work is supported by A*STAR, the Agency for Science Technology and Research, under the Ministry of Trade and Industry of Singapore 1 Joel Stephen Short studies at the National University of Singa- when working with springs and mechanical design, though pore and also a student member of the SIMTech-NUS Joint Lab [5] was consulted most often for this project. (Industrial Robotics), c/o Department of Mechanical Engineering, Na- The use of torsion springs to provide a passive torque, that tional University of Singapore, 9 Engineering Dr. 1, Singapore 117576 joel.stephen.short@u.nus.edu depends on the position and arrangement of the spring, is a 2 Aun Neow Poo and Marcelo H Ang Jr are with the Department of very old idea and most easily seen in the common clothespin, Mechanical Engineering, National University of Singapore, 9 Engineer- yet its use in robotics has been limited. An early study of ing Dr. 1, Singapore 117576 and also staff members of the SIMTech- torsional springs within the dynamics of the a generalized NUS Joint Lab (Industrial Robotics) mpepooan@nus.edu.sg; mpeangh@nus.edu.sg; robot framework can be seen in [6]. Other closely related 3 Lai Chow Yin and Tao Pey Yuen are with the Singapore Institute of work focuses on using springs in conjunction with actuators, Manufacturing Technology, Agency for Science, Technology and Research, normally classified as passive-compliant or variable stiffness Singapore 638075 and also a staff member of the SIMTech-NUS Joint Lab (Industrial Robotics), cylai@SIMTech.a-star.edu.sg; actuators. A useful survey of various passive-compliant ac- pytao@SIMTech.a-start.edu.sg tuators, where the springs basic properties are used without FinE-R 2015 Page 36 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 adjustment, is seen in [7]. Some variable stiffness actuators The flanges can be rotated independently, along the cham- use actively adjusted springs, as seen in [8], showing an fered slots, when the screws are loosened as seen in Figure additional connection to biomechanical design. 2. The flange slots allow both the preload and offset of the The design presented here is unique in two ways; first spring to be adjusted within a limited range of positions. The it is very versatile, capable of using many different size range of motion and possible adjustment is outlined in Table springs, second, it is highly adjustable, allowing the offset I while the setting ranges of the offset and preload are shown and preload to be set independently. Though experimental, in more detail in Figure 3. The graph helps show that as the this joint allows for greater investigation into the capabilities offset is adjusted the available offset range also changes, this and usefulness of torsional springs within the serial-link is due to the limits of the mechanical setup. robot framework. The spring sits around the joint axle with the second link mounted to the axle using a mini-bush clamp, this allows the II. DESIGN linkage to be placed higher or lower on the axle depending The design of the torsional joint was performed using the on the size of spring used. The axle is secured to the first traditional tools and methods of the mechanical engineer. joint with a single ball bearing. After developing a few possible ideas that led to sketches and drawings, the most promising one was built up in a computer aided drafting (CAD) program (Autodesk Inventor) with the creation and virtual assembly of the parts. The completed initial design of the prototype led to the manufacturing and assembling of the parts. The build, test, and redesign cycle was run through twice with the final prototype showing reliable performance in all important areas of the design. A. Design Goals Adapting the basic spring principles and capabilities for use in a torsional spring driven joint started with a review of what was needed from the joint. The design goals were created by reviewing the needs of the overall robot as well as the materials and space available. The goals are built around keeping the design simple and are listed below: Fig. 2. Linkage 2 flange attachments 1) Use a single torsion spring 2) Offset and preload angles must be adjustable The last design goal, allowing an encoder to read the 3) Spring body width must be adjustable angular position, was fulfilled by creating a joint axle with a 4) Only use the spring in compression small protruding extension at the bottom. An encoder could 5) Allow an optical encoder to read the angular position then be mounted on the underside joint, specifically on the The experimental nature of the joint drove the first and end cap spacer, with the optical wheel mounted at the end second goals, to allow for adjustment of the spring position of the axle. and initial torque. The use of different springs prompted the third goal. The fourth design goal was created after TABLE I investigating the proper use of torsional springs, they are T ORSIONAL SPRING JOINT PROPERTIES not made to be used repetitively in both tension and com- Parameter Stiffness Offset Preload Link 2 Motion pression. Most manufactures recommend only using them in Variable k θf θp θ2 Range (0.02, .01) ±50 (−20, +80) (−175, +270) compression. The last goal is due to the experimental nature Units Nm/rad degree degree degree of the mechanism and enables the angular position feedback from the joint to be recorded, allowing further study and evaluation of the robots motion in post processing. The overall design can be seen in Figure 4 with its related parts list in Figure 5. All of the design goals were achieved B. Implementation in the general design layout, though only by building the The simplest and most direct design uses only one spring torsional joint and testing it could the mechanism be deemed and two pairs of hook and flange subassemblies. Each hook successful. plate is attached to a flange that is stacked with another flange with both secured to the robot linkage. There are two III. ASSEMBLY AND TESTING flanges per link, one set has long flange arms and the other The prototype went through a cycle of assembly, testing, short flange arms. This hook hand-off design, with the two and redesign, twice before the arriving at the final setup. different flange arm lengths, ensures that the torsion spring Therefor are three designs, denoted alpha(original), beta, and is only used in compression, no matter if the linkage moves final. The difficulties encountered at each stage are discussed in the positive or negative radial direction. leading to the proposed solutions and redesign. FinE-R 2015 Page 37 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 1 2 D 3 18 7 4 5 6 9 10 14 15 7 11 8 16 12 17 13 SECTION D-D D Fig. 4. Overall design of the torsional joint, optical encoder to be mounted at the joint underside, to linkage 1 PARTS LIST 2 D ITEM A. AlphaPART QTY NUMBER results 120 1 1 Joint 2 Axle 100 2 1 TheMini-Bush parts Inner for 18 the torsional spring joint were sent out 3 for 1 manufacture Mini-Bush Outerat a local machine shop while the mini- 804 5 4 bush, 1 springs, Linkage 2 and hardware(bolts) were procured from local Preload(deg) 60 5 1 Upper Flange2 suppliers. Upon receiving the parts and assembling the joint a 6 6 1 Upper Flange1 40 major problem was observed; the bore in linkage 1, to house 9 7 4 M3x10 CSK 20 10 8 the 1 ball bearing was cut 1mm to short, causing the bearing to Torsion Spring 14 protrude from the housing. This was discouraging but before 0 15 9 1 Lower Flange1 10 sending 1 theFlange2 Lower part back to be finished properly, the rest of the 7 -20 11 11 assembly 1 Linkagewas1 constructed to check for other problems. -40 12 1 Additional Ball Bearinginvestigation proceeded despite the improper 16 13 1 -20 -10 0 13 10 20 fit of End theCap Spacer 17bearing and another major problem was found. 14 2 Long Hook Offset(deg) The ball bearing tolerances were far too loose and allowed D 15 2 Short Hook 16 the 2 axleWasher toPlate wobble from side to side. This caused the Fig. 3. Range of Torsion joint settings hook hand-off to sometimes miss and more importantly the 17 3 M2x6 18 8 opticalM3x6 encoder could not function reliably under such wide tolerances. After considering this major problem of axle PARTS LIST wobble, it was thought that by adding a roller bearing to the ITEM QTY PART NUMBER axle the problem could be fixed with the addition of only 1 1 Joint 2 Axle one new machined part, an extension spacer. All the original 2 1 Mini-Bush Inner 3 1 Mini-Bush Outer parts could still be used. This new design compensated for 4 1 Linkage 2 the previous machining error, a drawing of the new bearing 5 1 Upper Flange2 package can be seen in Figure 6. The tolerance limit of the 6 1 Upper Flange1 encoder was closely consulted but due to the lack of precise 7 4 M3x10 CSK bearing tolerances from the manufacturer the redesign had 8 1 Torsion Spring to rely on the best estimates of the engineer. 9 1 Lower Flange1 10 1 Lower Flange2 Lastly, as part of the hook hand-off difficulties, the short 11 1 Linkage 1 hook trough (where the spring sits on the hook) was found 12 1 Ball Bearing to be too close to the flange, making it difficult for the 13 1 End Cap Spacer long hook plate to grab the spring leg at the hand-off. The 14 2 Long Hook new hooks would be needed to allow for easy transition, a 15 2 Short Hook comparison of the old and new hooks is seen in Figure 7. 16 2 Washer Plate The hook plates lacked specific angular markings, so 17 3 M2x6 18 8 M3x6 preload and offset angles had to be estimated. In order to change out the torsion spring or adjust the preload or offset Fig. 5. Torsion joint parts list the second linkage had to be removed from its axle. This was not difficult due to the locking mini-bushings used, though FinE-R 2015 Page 38 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 1 3 8 4 8 2 6 B 2 6 5 C 5 4 7 1 6 3 SECTION C-C PARTS LIST 7 ITEM QTY PART NAME 1 1 Linkage 1 V2 2 1 Joint 2 Axle V2 SECTION B-B 3 1 Extension Spacer 4 1 End Cap Spacer PARTS LIST 5 1 Bearing Spacer B ITEM QTY PART NUMBER 6 2 Ball Bearing 1 1 Linkage 1 V2 7 3 M2x15 2 1 Joint 2 Axle V1 C 8 3 M2x6 3 1 End Cap Spacer 4 1 Extension Spacer Fig. 8. Final bearing package 5 1 Ball Bearing 6 1 Roller Bearing 7 3 M2x15 8 3 M2x7 it made adjustments a tedious affair. B. Beta results Fig. 6. Second torsion joint design (Beta) With an additional roller bearing, a new short hook, and the extension spacer the second assembly proved to still contain difficulties. The new short hook allowed the hook hand-off to proceed smoothly despite the fact that the joint axle wobble was still too great. The roller bearing did reduce the axial play (in terms of the wobbling) but not enough to allow for reliable readings from the optical encoder mounted on the bottom. It was at this point decided that an adjustable bearing package would be the best solution, then the tolerances of the ball bearings would not be an issue. The final bearing setup is seen in Figure 8. The final design required a new joint axle that was slightly longer as well as a bearing spacer for the axle. An additional ball bearing was also needed. The measurements sent to the machine shop, regarding the bearing package, were kept rough such that upon assembly the engineer could adjust the fit of the bearing package to allow an appropriate amount of play. If the bearings package is too tight and the axle won’t turn, the bearing spacer can be ground down, while if the package is too loose, the machined surface of the extension spacer (which sits against linkage 1) can be ground down. This is a common method for tuning the bearing clearances of large gearboxes. C. Final results The final build-up of the torsional joint can be seen in Figure 9. The tuning of the bearing package was done by hand; by using a hand file and a lathe the extension spacer was ground down progressively, bringing the outer races of the bearings closer together until the axle wobble was Fig. 7. Old hook (left) with new hook (right) eliminated, but it could still freely turn. The final design was completely successful in achieving all of the design goals. The optical encoder returned a reliable FinE-R 2015 Page 39 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 signal while the adjustability of the joint allowed for the use complex. The first redesign of the bearing setup only required of different springs and numerous different offset and preload one more machined part and one additional bearing, plus it setups. allowed the imperfection of the bearing bore in linkage 1 to be left alone. It was thought to be the most economical, IV. DISCUSSION yet there was little to no guarantee that it would solve the The design, assembly and testing of the torsional joint was axle wobble problem. The adjustable bearing package was completed as a prototype, for use in testing a theoretical slightly more complex but should have been used in the first control methodology and some important lessons can be redesign. learned from the process. The failures in design and the Lesson 3: Familiarity with standard engineering solutions route taken in redesigns reveals some beneficial as well as that are related to the current design is highly beneficial. detrimental decisions. These will each be discussed as they The design of a prototype lends itself to quick thinking relate to either the design of the mechanism or the testing of and the use of engineering solutions that “may” work or the assembled parts. “should” work. Though time is often of the essence and there A. Design Lessons is not time for an in depth analysis of the parts to ascertain if the part or assembly will meet the design goals exactly it Lesson 1: Bring all the design constraints together, ex- is critical that the engineer have a general understanding of plicitly listing how they need to be achieved. standard industry and engineering practices. Spending time The design goals of a mechatronic system typically involve to become familiar with the traditional solutions, relating to requirements from the mechanical side, such as bearings, the particular parts or assemblies under design, can save time fits and hardware, as well as from electrical parts, such and energy later in the process. This can be readily seen in as encoders, motors and other interface pieces. If details the example when considering the bearing setup for the joint are left out, they will often show up as trouble during axle. The first design turned out to be inadequate and only testing. The design goals in the example were clear when a half measure. Instead, the industry standard for bearing the mechanism was first drawn up, the needs of the torsional packages which need tight tolerances should have been used spring adjustments were straightforward to implement, but right away. the optical encoder requirements were not explicitly checked in the initial design. This lack of detail in the design goal B. Testing Lessions contributed to the problems found in the first design, leading Lesson 4: Test and investigate all aspects of a mechanisms the the first redesign. design, as able, before disassembly and redesign. Lesson 2: The simplest solution is not always the best, When working with the design, assembly and testing of choose the redesign solution that solves the problem most prototype, it is important not to get caught up with a single completely. problem such that it distracts from overall testing. This is When redesigning a part or assembly, the simplest and seen with regards to the machining mistake on the first most minimal design is often the most attractive but when linkage, where the bearing bore was to short. Instead of considering the complexity of the possible solutions, go with immediately sending the part back for correction and having the one most likely to solve the problem, even if it is more to wait before testing the overall mechanism, the engineer assembled the rest of the parts to examine the part interfaces, the hook hand-off. This additional testing revealed problems that were much more critical than the bore mistake. By testing and examining the assembly as much as possible before trying to fix the small mistake, time was saved and the redesign could include the altered dimensions. Lesson 5: Implement low risk redesigns early. Lastly, when working with and testing an assembly of parts that requires a redesign, take time to step back and examine the assembly as a whole, looking for small problems that can be improved with a low risk of affecting the overall working of the mechanism. Including these improvements in a first redesign can save time in later testing. An example of this is seen in the short hook redesign. Though the wobble of the joint axle, when supported by one ball bearing, contributed to an unreliable hook hand-off the engineer was able to identify a second problem area around the short hook. The hook trough was too close to the flange, in order for a successful spring leg hand-off the long hook was required to pass extremely close to the short hook flange. The hook Fig. 9. Final torsional joint setup (without encoder) was redesigned to allow for more space between the moving FinE-R 2015 Page 40 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 parts. This contributed to a smoother working hook hand-off of the spring, outside of the troubles with the bearings. V. CONCLUSIONS When designing, building and testing a prototype mecha- nism for robotics research there are often difficulties. The short time schedule forces an engineer to make certain assumptions and estimations, which can lead to trouble in the assembly and testing phase. This paper presented the experience of one researcher in designing, building and testing a torsional spring joint prototype. The process faced a few problems but through two redesigns the failures were solved, producing a successful mechanism that met all the required design goals. The lessons learned from this process were discussed in detail and connected to specific examples in the design and testing of the torsional spring joint. R EFERENCES [1] M. Benosman and G. Le Vey, “Stable inversion of siso nonminimum phase linear systems through output planning: An experimental applica- tion to the one-link flexible manipulator,” IEEE Transactions on Control Systems Technology, vol. 11, no. 4, pp. 588–597, 2003. [2] K. Graichen, V. Hagenmeyer, and M. Zeitz, “A new approach to inversion-based feedforward control design for nonlinear systems,” Automatica, vol. 41, no. 12, pp. 2033–2041, 2005. [3] J. S. Short, J. A. N. Poo, M. H. Ang Jr., C. Y. Lai, and P. Y. Tao, “A generalized underactuated robot system inversion method using hamiltonian formalism,” in IEEE/ASME International Conference on Advanced Intelligent Mechatronics, AIM, 2015. [4] M. Bergerman, C. Lee, and Y. Xu, “A dynamic coupling index for underactuated manipulators,” Journal of Robotic Systems, vol. 12(10), pp. 693–707, 1995. [5] P. Childs, Mechanical Design. Elsevier Butterworth-Heinemann, 2004. [6] T. Yamamoto and Y. Kuniyoshi, “Harnessing the robot’s body dynam- ics: A global dynamics approach,” in IEEE International Conference on Intelligent Robots and Systems, vol. 1, 2001, pp. 518–525. [7] B. Vanderborght, R. Van Ham, D. Lefeber, T. G. Sugar, and K. W. Hollander, “Comparison of mechanical design and energy consumption of adaptable, passive-compliant actuators,” The International Journal of Robotics Research, vol. 28, no. 1, pp. 90–103, 2009. [8] J. W. Hurst, J. E. Chestnutt, and A. A. Rizzi, “The actuator with mechanically adjustable series compliance,” IEEE Transactions on Robotics, vol. 26, no. 4, pp. 597–606, 2010. FinE-R 2015 Page 41 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Design, Simulation and Implementation of a 3-PUU Parallel Mechanism for a Macro/mini Manipulator Zheng Ma, Aun-Neow Poo, Marcelo H. Ang Jr , Geok-Soon Hong, and Feng Huo,  Abstract— Parallel mechanisms have the advantages of high involved continuous contact between the robot end-effector rigidity, high precision and fast movement in its workspace. It is and the workpiece, and the simultaneous control of the force at a most suitable mechanism to serve as the mini manipulator in a the point of contact. Such force/position controlled operations macro/mini manipulator as the mini manipulator needs to have include high-precision edge and surface finishing operations fast response and high resolution in positioning. In this paper, often encountered in the precision engineering, aerospace, and the design of a 3-PUU parallel mechanism to be used as such a marine industries. mini is presented. Failures are encountered during the process of simulation and implementation of the parallel mechanism. Since an adequate workspace and a sufficient Causes of the failures are analyzed and solutions are proposed to payload-carrying capacity are required in the performance of overcome these. Based on the lessons from building the first their tasks, industrial robots are often designed with long and prototype, improvements were made to the second prototype large arms. With its large mass and inertia [1], it is thus which effectively removed the shortcomings resulting in a mini difficult to control such a single robotic arm in applications which met the requirements for its intended application. which require position, force or force/position control and achieve high accuracy with a fast response simultaneously. I. INTRODUCTION A proposed solution is to implement a compact The development and application of robotics has made end-effector with a small limited workspace which can have a much progress since the first programmable industrial robotic high bandwidth and high accuracy in positioning and have this arm, the Unimate, was invented in 1961. Compared with carried by a larger but slower robotic arm. This configuration human operators, industrial robots have the advantages of high is commonly referred as a macro/mini manipulator, where the precision, repeatability and speed of motion, and high large robotic arm is referred to as the “macro”, and the smaller dexterity. They can also work in environments hazardous or and faster end-effector referred to as the “mini”. The unsuitable for human beings and, with large robots, are capable macro/mini manipulator has the advantages of a large of carrying and moving, with higher speeds and accuracy of workspace provided by the macro robotic arm, as well as a fast motion, heavy workpieces. In addition, except for downtime and high-accuracy response provided by the mini [2]. for maintenance, they are 24/7 workers who do not need rest or holiday leaves and can thus improve productivity and speed of Considerations which need to be taken in the design of a production. mini manipulator depend on what tasks it is being developed for. In this paper, a mini manipulator designed for polishing When used appropriately, industrial robots can reduce the and deburring tasks is discussed. The normal forces that need need, not only of unskilled labourers but also skilled workers, to be applied by the polishing or deburring tool on the in industry. As a result, they have found widespread workpiece are estimated at up to 100 N and a few Newtons for applications in repetitive operations such as material handling polishing and deburring respectively. The optimum exerted and assembly, welding and spray painting. To date, most of the force depends on the type of operation, the material of the applications of industrial robots are for non-continuous contact workpiece and the type of tool used. A rough type of operations, operations which do not require the robotic sanding/polishing operation using a sanding/polishing pad end-effector to be in continuous contact, and with a controlled which has a large area of contact with the workpiece surface level of contact force, with the workpieces. will require a large exerted force whereas a small exerted force Recent advances in robotics technology have allowed the will be needed for a fine finishing operation with a smaller development of robotic arms with increased speeds and polishing pad. precision of motion and with greater build-in intelligence. The profile of the surface of the workpiece that is to be There is now increasing interest in developing and employing operated on is assumed not to have sudden rapid changes such these devices for more challenging tasks, including those that a workspace in the form of a sphere with a diameter of labour-intensive and low-productivity operations which 40mm will be sufficient for the mini end-effector. During a polishing or deburring operation, the macro manipulator *Research supported by SIMTech-NUS Joint Laboratory and A*STAR. carries the mini manipulator (end-effector) along a desired Zheng Ma is with the Advanced Robotics Center and the SIMTech-NUS reference path parallel to and at a small distance away from the Joint Laboratory (Industrial Robotics), National University of Singapore, surface to be polished or from the edge of the workpiece to be Singapore, 117580. (Corresponding author. Phone: +65-91188562; Email: deburred. For optimum operation, the orientation of the mpemz@nus.edu.sg). end-effector should have a predefined orientation with respect Aun-Neow Poo, Marcelo H. Ang Jr, Geok-Soon Hong and Feng Huo are with the Department of Mechanical Engineering, National University of to the surface, or edge, of the workpiece. While being moved Singapore, Singapore, 117576. (Emails: mpepooan@nus.edu.sg, along this reference path by the macro, the mini moves in such mpeangh@nus.edu.sg, mpehgs@nus.edu.sg and huofeng@nus.edu.sg). FinE-R 2015 Page 42 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 a way as to exert the desired normal force on the workpiece. prismatic joints move in a direction perpendicular to the base Since the mini is always in contact with the surface or edge of platform and are attached symmetrically at 120 degrees apart at the workpiece, and as long as there are no sudden and large Ai , where i  1,2,3 , to the base platform. As shown in Fig. 1, change to the surface or edge of the workpiece, the workspace of the mini will not need to be large to perform the polishing or two universal joints (universal joints) connect the end of each deburring task. prismatic joint to the top platform. The axes of the two universal joints are parallel to each other and perpendicular to Based on the aforesaid considerations and using feedback the prismatic joint. According to the Chebychev-Grübler– from users with experience in polishing and deburring Kutzbach criterion [3], the number of degrees-of-freedom is operations, a 3-DOF PUU(Prismatic-Universal-Universal) given by: parallel mechanism, inspired by the Delta robot was selected j for the mini manipulator. This 3-DOF translational parallel  M  3(N 1 j)   fi   mechanism (TPM) has only pure translational motions and was i1 designed to have a cylindrical workspace with a diameter of where N is the total number of links, j the total number of 40mm and a height of 30mm. joints, and fi , ( i  1,2,3 ) the degrees of freedom of link i . For In the design process, solid models were first created to the mechanism shown in Fig. 1, the total number of links simulate and to analyze the motions, and to evaluate the (including the base link) is N  8 , the total number of joints stresses and deformations in the various links and components is j  9 , and the degree of freedom is fi  1 for the prismatic when it is subjected to the maximum design applied forces and joints and fi  2 for the universal joints. Thus torques. During the simulation study of its motions, unexpected motions with extra degrees of freedom were  M  3(8 1 9)  31 6  2  3   observed which caused the mini manipulator to take on and the mechanism shown in Fig. 1 has three postures in which the platform on the mini end-effector was degrees-of-freedom with all being translational motions as will not purely translated but was rotated from its starting position. be elaborated on in the next section. This ensures that the top A kinematic analysis based on the 3-DOF translational motion platform is always parallel to the base platform. fails to explain these unexpected motions since the assumptions made in the kinematic analysis does not hold top platform U‐ joint when the mechanism is not in parallel with its starting position. B3 B2 P3 O’ To reduce the overall cost and time, the universal joint U‐ joint P2 components are directly ordered off the shelf for implementation. The parallel mechanism appears to have B1 Prismatic notable backlash. The resulting precision of the mechanism is joint poor and cannot serve as the mini manipulator which supposed base platform P1 A3 A2 to have high accuracy in positioning. The mechanism is modified eventually to overcome the O backlash problem and retains the same kinematics as previously designed. As a result, the working range and mobility of the mechanism meets the requirement. Together with a proper control algorithm, the mechanism can be used to serve as the mini manipulator which has a fast response and A1 high precession in positioning. In this paper, the 3-PUU parallel mechanism is first Figure 1. Structure of the 3-PUU parallel mechanism. described and a standard kinematic analysis is derived under assumptions. Unexpected motions in simulations are shown, with a brief analysis of the reason why it happens. Problems of backlash and positioning accuracy encountered in implementation is discussed with an analysis of an off-the-shelf universal joint structure. Improvements of the mechanism architecture and joint options are presented which overcomes the failure from the simulation as well as the real implementation. II. MECHANISM DESCRIPTION AND KINEMATIC ANALYSIS A. A 3-PUU Parallel Mechanism The structure of the 3-PUU parallel mechanism designed is shown in Fig. 1 with three identical limbs connecting the base platform to the top platform. Fig. 2 shows the structure for one of the limbs. From the figures, it can be noted that the three Figure 2. One of the limbs of the 3-PUU parallel mechanism. FinE-R 2015 Page 43 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 B. Kinematic Analysis of 3-DOF Translational Motion  z i   L2  ( x  xi ) 2  ( y  yi ) 2  z   With knowledge of the 3-DOF translational mobility, the kinematic model of the parallel mechanism can be derived [4]. In the same way, the forward kinematics can be obtained The top view of the base and top platform is shown in Fig. 3, by applying the same constraint equation. where Ai and Bi are the locations where the prismatic joints and the universal joints are mounted to the base and the top III. FAILURES IN SIMULATION AND IMPLEMENTATION platform respectively. Coordinate Frame O and Frame O’ are With the kinematic model obtained, the parameters R, r and respectively attached at the centre of the base and the top L were chosen to meet the workspace criteria. Solid models platform. The distance from the center of the platforms to Ai were then established for motion and stress analysis, the and Bi are R and r respectively. Let the displacement of the ith former to confirm the translational motions of the top platform prismatic joint attached at Ai be zi. All the universal joints are within the specified workspace and the latter for sizing the passive. components for strength and stability. Since the parallel mechanism are constrained to have only translational motions, the transformation matrix for rotation During simulation, some unexpected results were observed from frame O’ to frame O is an identity matrix. Let the position when the top platform moved away from being parallel to the vector of Frame O’ in Frame O be base platform. Unacceptable motion performance was also obtained with the first prototype developed using off-the-shelf  [ c ]O  ( x y z )T   universal joints. These will be discussed in the following A2 sections. B2 A. Extra DOF observed in Simulation R Solid models of the parallel mechanism were created using r the software SolidWorks®. Motion studies were done A3 B3 simulating motion at the three prismatic joints. This caused the three lower universal joints, P1, P2, and P3 in Fig. 1, to move vertically. Various combinations of linear motions for the three prismatic joints were used to study the movement of the top platform relative to the base platform, as well as to verify the B1 size of workspace of the parallel mechanism. A1 The top platform was expected to remain parallel to the base platform at all times since the design of the mechanism Figure 3. Top view of base platform(left) and top platform(right). constrained it to have only 3-DOF translational motion. However, it was noted that for some motion combinations of According to the mechanism structure shown in Fig. 1 and the prismatic joints, the top platform does not always remain the geometric conditions shown in Fig. 3, the inverse and parallel to the base platform but moved into a non-parallel forward kinematics of the parallel mechanism can be obtained. mode of motion after remaining parallel for some time. Fig. 4 By assuming the top platform has only translational motion shows an example of how the roll-pitch-yaw angles of Frame with respect to the base platform, position vector Bi in frame O’ with respect to Frame O change with time for one such instance. From the figure, it can be seen that the top platform O’ is moves with only translational motion for about 11s after which [ Bi ]O '  (r cos i r sin  i 0)T , it has rotational motions.    1  30 , 2  150 , 3  90 Therefore the position vector Bi in frame O is  [ Bi ]O  (r cos i  x r sin  i  y z )T   and the position vector Pi in frame O is  [Pi ]O  (R cosi R sini zi )T   For all three limbs, if the distance between the two universal joints, Bi to Pi is L. The constraint equation can then be written as  [Bi  Pi ]O  L   After substituting Bi and Pi into (7), we have ( x  xi ) 2  ( y  yi ) 2  ( z  zi ) 2  L2 ,    xi  ( R  r ) cosi , yi  ( R  r ) sin i Figure 4. Roll-pitch-yaw angles oftop platform for non-paralle motion. The inverse kinematics thus can be obtained as FinE-R 2015 Page 44 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 To explain the unexpected rotational motion, the a hole to accommodate the external shaft and a dowel pin is assumption of pure translational motion was reviewed. A used to hold the shaft to the joint as shown in the figure. typical drawing of a universal joint is shown in Fig.5. Ux Uy Figure 6. Universal joint(double) [7]. Link side UzP Figure 7 shows the first prototype of the mini manipulator Platform UzL mechanism using these universal joints. Three linear actuators, side labeled with 0, 1 and 2, are used for the prismatic joints. Each link connecting the prismatic joint to the top platform is made up of a circular shaft with a universal joint at each end. The universal joint at one end of each link is fixed to a linear actuator and the other end to the top platform. Figure 5. Rotational axis of a universal joint. Consider one of the three universal joints attached to the top platform as shown in Fig. 5. With the other end, Pi, of the link fixed, there will be no rotation about the axis UzL, The universal joint can only rotate about the Ux and Uy axes, enabled by the cross component in the joint. With only two degrees-of-freedom, there will not be any rotation about the axis UzP, and thus no rotation of the platform [5]. Since there are three universal joints attached to the top platform, therefore no rotation of the platform is allowed about three axes. When these three axes are linearly independent in 3 , the top platform will lose all the rotational motion and Figure 7. Translational parallel mechanism using U-joints. its 3-DOF motions will be purely translational. Based on this analysis, the rotational motion of the top platform during When the three linear actuators are fixed in any position, simulation as shown in Fig. 4 is thus unexpected. i.e. not moving, the top platform should also remain in a fixed position parallel to the base platform. However, it was found This rotational motion observed in simulation is suspected that with the actuators fixed in their positions, the horizontal to be caused by the loss of independence among the three axes slack of the top platform was 4 to 5 mm, which is unacceptably UzPi. When two or more axes become linearly dependent, the large, together with unacceptably large angular rotations. parallel mechanism will be in a singular position. Unlike the Investigations showed that these unacceptably large motions, singularities in serial-link robots, instead of losing degrees of or “backlash”, are due to the clearances used in the mobility, a parallel mechanism gains extra degrees of freedom manufacture of the mechanical components used. While pure at a singular position [6]. translation motion of the top platform was observed in In Fig. 4, it is likely that the parallel mechanism reached a simulation for which perfect dimensions of the various singular position at about 11s, gained an extra degree of components are used in computation, such perfectly formed rotational mobility and the top platform became non-parallel to parts are not available in practice, thereby resulting in the the base platform. Thereafter, the motion of the mechanism unacceptable results. A close examination of the first prototype was no longer constrained to be purely translational. showed that the exhibited backlash phenomenon is due almost entirely to clearances in the off-the-shelf universal joints used. Referring to the Chebychev-Grübler–Kutzbach criterion, the mechanism should have three degrees-of-freedom when it The universal joint, also known as a Hooke's joint, is a joint is not in a singular position. It is likely that the motion of the or coupling which is commonly used to transmit rotary motion mechanism after passing through the singular position is a from one rigid shaft to another rigid shaft when the axes of the combination of three degrees of motion with both rotation and two shafts are at a small angle to each other. The rotary motion translation. Further investigation will be needed explain and to transmitted is usually in one direction only. Because there is no understand this unexpected simulation result. change in direction of the transmitted rotary motion, the small clearances designed into them for ease of manufacture does not B. Backlash in Implementation cause any backlash problem. The universal joints used in the construction of the first The universal joints used in the TPM mechanism in the prototype were off-the-shelf good quality joints the schematic work here serve a different purpose. They serve as joints of which is shown in Fig. 6. Each side of the universal joint has providing two degrees of freedom (rotary motion) constraining FinE-R 2015 Page 45 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 the motion of the parallel mechanism as required from the are used for their typical functions of transmitting rotary structure shown in Fig. 1. Referring to Fig. 5, the universal motion between two shafts. joints used should rotate only about axes Ux and Uy to cause The first prototype failed to meet the requirements for its the top platform of the TPM mechanism to move. There should intended application and a review of the design, and where it be no rotation about axis UzL or UzP. However, when one side failed, was carried out to come up with the second prototype. of the U joint, say the link side, is fixed and not allow to rotate about its axis UzL, it is observed that the other side has freedom IV. LEARNING FROM THE FAILURES to rotate, about axis UzP, to some significant degree. This is due to manufacturing clearances designed into the joints, in In the process of developing and building the first particular at the four ends of the cross component in the joint. prototype, two valuable lessons were learned. One is the The resulting free-play or backlash is accentuated due to the unexpected results during simulation studies and the other is short lengths of the two rods forming the cross component in the poor performance in the fabricated mechanism due to the joint. The off-the-shelf U joints thus did not have sufficient manufacturing clearances and backlash in the off-the-shelf stiffness along the Uz axes and are not suitable for the TPM universal joints used. mechanism. It is noted that that the top platform of the mechanism does Another significant cause of the free-play or backlash not remain parallel to the base platform under all problem in the motion of the TPM mechanism is due to circumstances. Rather, when starting from a parallel position, clearance applied during the fabrication of the mechanism. As the top platform may move into a mode, or region of its mentioned earlier and with reference to Fig. 6, dowel pins were workspace, where it gains rotational motions after passing used to connect the external shaft to each end of the U joints. through a singular position. This problem occurred during Ideally, the two holes in the U joint and the one in the shaft to simulation when it is put all possible motions within its total accommodate the dowel pin should all be of exactly the same workspace. In practice, this problem can easily be overcome diameter, corresponding to the diameter of the dowel pin, with by constraining the motions of the three actuators such that its their centers perfectly aligned. However, as the holes were workspace clearly does not contain any singular positions. drilled at different times, if they were to be made of the same diameter with very little clearance, the centers of the holes The first prototype has unacceptably poor accuracy in its need to be perfectly aligned in order for the dowel pin to be motion and positioning. The top platform has some degrees of inserted. Alignment of the holes, when drilled separately, is not mobility, of about 5 mm due to backlash when the actuators are easily done. As such, the fabricator introduce some clearance fixed in their positions. This mobility is not acceptable as the and made the hole in the shaft larger (Fig. 8) than that of the mini manipulator is required to have high stiffness and holes in the U joint, which is of the same diameter as the dowel precision. It is clear that this problem is caused by the pin. While this allowed for the insertion of the dowel pin even manufacturing clearances in the off-the-shelf universal joints if there is some slight misalignment of the holes during used. To overcome this problem, while still using lower-cost manufacture, it caused significant rotational free-play or off-the-shelf components, other type of joints which has the backlash between shaft and the universal joint. Here again, the same motion properties as universal joints but do not suffer rotational backlash is accentuated by the small diameter of the from the same backlash problem was investigated as shaft, and thus the length of the hole in it. replacements. The mechanical structure to replace the link with its pair of universal joints is shown in Fig. 9. It is composed of four ball joints connected in a way to form a parallelogram. B A D C Figure 8. Clearamce between dowel pin and the external shaft connected to the U-joint. Figure 9. Improved parallel mechanism with ball joints. The unsatisfactory motion of the first prototype of the According to the property of an ideal parallelogram, the mechanism is largely due to the clearances in the off-the-shelf opposite sides of the parallelogram will always be parallel. universal joints and the limited machining accuracy of the Therefore, the side AB will always be parallel to the side CD in fabricated parts. Information on clearances for off-the-shelf Fig. 9. Since the side CD is mounted parallel and fixed to the base platform, the side AB will also always be parallel to the universal joints are not readily available from manufacturers base platform. As there are three limbs in the TPM mechanism, as such information may not have been important when they FinE-R 2015 Page 46 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 there are three parallelogram with three sides AB attached to resolved. Further research will be done to determine the exact the top platform. cause of the rotational motions of the 3-PUU parallel These three parallelogram limbs are attached to the top mechanism during simulation. platform such that the three sides AB all lie in a plane and the Unacceptable free play and backlash was exhibited by the top platform is parallel to this plane. Since all the three sides first prototype. This was not evident in the simulation AB are parallel to the base platform, the plane formed by them experiments which are based on perfectly manufactured will be parallel to the base platform. Therefore, the top components. Investigations showed that this problem was due platform will also always be parallel to the base platform. With to inaccuracies in the dimensions of the components used. The the top platform constrained to be parallel to the base platform, main cause was the free play in the off-the-shelf universal and the base platform is fixed and immobile, the motion of the joints used for the first prototype. To overcome this problem top platform will be constrained to be translational only. the universal joints were replaced by off-the-shelf ball joints If there is free play or backlash in the ball joints at A, B, C, forming a parallelogram structure for the three limbs of the or D in Fig. 9, then the parallelogram formed will not be an mechanism. The kinematic model of the mechanism remains ideal parallelogram. In this case, the sides AB may become the same but the free play problem was effectively eliminated non-parallel to the side CD. The amount of non-parallelism and the second prototype exhibits high stiffness and depends on the amount of free play in the ball joints and the length of the sides AB and CD, the longer the sides are, the positioning accuracy. smaller the degree of non-parallelism. Lessons were learned from unexpected outcomes and failures during the simulation experiments and in For the typical applications they are intended for, good implementation. Properly designed simulation experiments quality ball joints have almost no free play or backlash. The may produce results not predicted by theoretical studies as length of the sides AB and CD of the parallelogram are also much longer than the length of the cross component in the these studies are normally based on certain simplifications universal joints. As such, the use of ball joints with a and assumptions, which cannot be completely replicated in parallelogram structure for the three limbs of the TPM simulation experiments. mechanism effectively eliminated the free play and backlash Furthermore, straightforward simulation experiments problem. The resulting second prototype is rigid and has high which are based on perfect physical properties of the precision in positioning. With the actuator fixed in their component parts may not show up possible inadequacies in positions, there is no measurable backlash in the top platform. the design. These inadequacies may show up only in the The backlash found in the first prototype had been effectively prototypes built due to unavoidable imperfections in the eliminated and this second prototype will be suitable as the physical components making up the whole system. mini in a macro-mini manipulator to be used for finishing and deburring applications for which both position and force/position control are required. Unlike a serial-link robot, ACKNOWLEDGMENT the parallel structure of this robotic device gives it the high The authors acknowledge the support from the Collabora- rigidity and thus the capability of exerting large forces on the tive Research Project under the SIMTech-NUS Joint Labora- workpiece in force-controlled polishing applications tory (Industrial Robotics). This work was also supported in part by the Science and Engineering Research Council V. CONCLUSIONS (SERC) A*STAR Industrial Robotics Program Grant 12251 A parallel mechanism, based on the structure of the Delta 00008. robot, was designed and implemented to serve as a mini REFERENCES manipulator, acting as an end-effector, in a macro-mini manipulator configuration for polishing and deburring [1] O. Khatib, "Inertial properties in robotic manipulation: an objective-level framework," The International Journal of Robotics Research, vol. 1, no. applications. 13, pp. 19-36, February 1995. Kinematic models of the mechanism were first obtained [2] Z. Ma, G. S. Hong, M. Ang and A. N. Poo, "Mid-ranging control of a and applied to fulfil the given criteria. Solid models were macro/mini manipulator," in IEEE International Conference on Advanced created to simulate and analyze the resulting motions and Intelligent Mechatronics, Busan, 2015. workspace of the mechanism which was design. Unexpected [3] J. Angeles and C. Truesdell, Rational Kinematics, Springer 1988, p. 78. and unacceptable motions of the top platform in the [4] Y. Li and Q. Xu, "Kinematic Analysis and Dynamic Control of a 3-PUU Parallel Manipulator for Cardiopulmonary Resuscitation," in 12th mechanism were observed during the simulation experiments. International Conference on Advanced Robotics, Seattle, 2005. The kinematic models failed to explain the motion since the [5] I. Bonev and D. Ziatanov, "The Mystery of the Singular SNU assumption of pure translational motion of the top platform Translational Parallel Robot," 12 June 2001. [Online]. Available: did not hold. It is likely that the non-parallel motions of the http://www.parallelmic.org. [Accessed 11 March 2014]. top platform in the mechanism was due to it passing through a [6] G. Gogu, Structural Synthesis of Parallel Robots, Dordrecht: Springer 2008, p. 249. singular position at which it gained extra degrees of freedom. [7] Misumi, "Universal Joints-Set Pin Type," Misumi, [Online]. Available: With the motion of the actuators in the mechanism https://sg.misumi-ec.com/asia/ItemDetail/10300127430.html. [Accessed constrained such that no singular positions lie within the 17 July 2015]. workspace, the problem of non-parallel motions can be FinE-R 2015 Page 47 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Intelligence Level Performance Standards Research for Autonomous Vehicles Roger B. Bostelman, Tsai H. Hong, and Elena Messina  Abstract— United States and European safety standards emergency response and military applications [4]. This have evolved to protect workers near Automatic Guided reference also discusses research challenges, test and Vehicles (AGV’s). However, performance standards for evaluations, and intelligent systems development programs AGV’s and mobile robots have only recently begun that can support advancement of industrial AGVs towards development. Lessons can be learned from research and attaining greater levels of intelligence. These other efforts also standards efforts for mobile robots applied to emergency provide useful standards development criteria for AGV response and military applications. Research challenges, performance test methods. Experiences and results in tests and evaluations, and programs to develop higher advanced mobility and intelligence for robotics will be intelligence levels for vehicles can also used to guide essential for AGV manufacturers and users to fully understand industrial AGV developments towards more adaptable and capabilities and specific applications of their autonomous intelligent systems. These other efforts also provide useful vehicle systems. standards development criteria for AGV performance test methods. Current standards areas being considered for Performance test methods for docking, navigation, (see AGVs are for docking, navigation, obstacle avoidance, and Figure 1) [5], and terminology standard work items have been the ground truth systems that measure performance. This initiated under the new ASTM Committee F45 on Driverless paper provides a look to the future with standards Automatic Guided Industrial Vehicles performance standard developments in both the performance of vehicles and the [6]. Standards for autonomous industrial vehicle obstacle dynamic perception systems that measure intelligent vehicle avoidance and protection, based on past research [7], performance. communication and integration, and environmental impacts I. INTRODUCTION are also being considered. Automatic Guided Vehicles (AGV’s) have typically been This paper will specifically discuss measurement of: used for industrial material handling since the 1950’s. Since vehicle navigation (e.g., commanded vs. actual AGV path- then, U.S. [1] and European [2] AGV safety standards have following deviation), vehicle docking (e.g., AGV stop point evolved to protect nearby workers. These standards have positioning vs. known facility points), and obstacle detection minimal test methods to describe how manufacturers and users and avoidance of standard test pieces (e.g., comparison of real- are to perform AGV safety measurements, resulting in time AGV path-planning and new path following vs. potential measurement differences across the industry. For commanded path) towards smart manufacturing applications, example, American National Standards Institute/Industrial such as assembly and unstructured environment navigation. Truck Safety Development Foundation (ANSI/ITSDF) Additionally, this paper will discuss a new ASTM Committee B56.5:2012 provides new language to generically handle a on 3D Imaging Systems E57.02 [8] standard work item for six situation when an object suddenly appears within the AGV degree-of-freedom (DOF) optical measurement of dynamic stop region. The stop region is the area surrounding the AGV systems (see Figure 2), which advances the existing static 6 in which the non-contact safety sensor detects obstacles and DOF standard [9]. The new standard is expected to be a critical stops the vehicle. The manufacturer must now prove that when component of performance measurement for current and the AGV detects an object closer than its stopping distance, future robotic systems that rely on advanced perception although collision with the object is perhaps imminent, the systems. AGV demonstrates a reduction in kinetic energy. However, II. PERFORMANCE STANDARDS THRUSTS there is no description of how manufacturers measure this situation, resulting in different measurement results across AGV navigation, docking, and obstacle detection and manufacturers. One test method was researched to handle this avoidance tests were conducted in support of future situation and is described in [3]. performance standard test methods and are described in this section. In some instances, typical industry practices were Recently AGV and mobile robot performance standards evaluated as well as the improved AGV performance tests. developments have begun to limit measurement method differences. Initial developments began with a review of other research and standards efforts for mobile robots as applied to R. V. Bostelman is with the National Institute of Standards and T. H. Hong, is with the National Institute of Standards and Technology, Technology, Gaithersburg, MD 20899, USA and with the IEM, Le2i, Gaithersburg, MD 20899, USA (phone: 301-975-3444; fax: 301-990-9688; Université de Bourgogne, BP 47870, 21078 Dijon, France (phone: 301-975- e-mail: tsai.hong@nist.gov). 3426; fax: 301-990-9688; e-mail: roger.bostelman@nist.gov). E. Messina is with the National Institute of Standards and Technology, Gaithersburg, MD 20899, USA (phone: 301-975-3510; fax: 301-990-9688; e-mail: elena.messina@nist.gov). FinE-R 2015 Page 48 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 A. Vehicle Navigation The most basic functions of mobile robots and AGV’s are navigation to and docking with equipment in the workspace. However, the description of how well the vehicle navigates (i.e., commanded vs. actual AGV path-following deviation) has certain ambiguities. For example, navigation implies that the vehicle measures its current position, plans a route to another location, and moves from the current location to planned location upon command. Most vehicle manufacturers don’t provide specifications for how uncertain the navigation performance is (i.e., the error bounds on position or velocity), other than perhaps radius of vehicle turns, maximum velocity, and maximum acceleration. The vehicle velocity sets limits on the allowable turn radius for particular vehicles. Some controllers [10], if not all, will not allow high velocities on relatively small radii to prevent Figure 1. Example reconfigurable apparatus for navigation tests for various unsafe vehicle conditions. These limitations are not typically AGV sizes. specified by AGV manufactures, causing AGV users To address AGV navigation uncertainty, with an eye difficulty in planning how many vehicles they may require for towards a potential test method for all automatic industrial moving their products within the facility to maintain a desired vehicles, tests were executed, both with an AGV prior to and throughput. after being calibrated. The uncalibrated AGV test is similar to Industrial vehicles may eventually become uncalibrated typical industry methods since not all AGVs can be frequently through regular use. An uncalibrated vehicle does not follow calibrated. An uncalibrated AGV was moved along a straight a commanded path or stop/dock at a commanded point with line path between two commanded points in an open area and minimal relative uncertainty (standard deviation of measured spaced approximately 5 m apart [5]. Figure 2 shows the results vs. ground truth) as does a calibrated vehicle. To correct this, amplified in the X direction 100 times to exaggerate vehicle vehicle manufacturers have calibration procedures for their performance. In the figure, the blue line is the commanded vehicles, although these procedures can be tedious, time- path between points 1 and 2. The green dots to the right and consuming, and may not be appropriate for all vehicles. For left of the line are uncalibrated AGV controller-traced position example, calibration of Ackerman steered vs. ‘crab’ steered data moving forward and reverse, respectively, between the (sometimes called quad) vehicles have different calibration points. The red dots are ground truth of the navigating AGV procedures. It is not always clear what will happen when a between points using an optical tracking system. This vehicle is uncalibrated nor when the vehicle becomes experiment demonstrated one AGV navigation performance uncalibrated. The effects of calibration on vehicle control and measurement method using a precision (0.2 mm standard uncertainty are typically not specified either. There is also deviation) six degree-of-freedom (DOF), optical measurement typically no specification describing how far from the system as a ground truth comparison to the onboard vehicle commanded path a vehicle navigates. This may be important tracking system. Path deviation was approximately 20 cm to users who have tight tolerance AGV paths (e.g., paths maximum. The AGV was then calibrated using the between infrastructure) that must be followed. A test can be manufacturer’s method. developed to uncover the effects of uncalibrated vs. calibrated vehicle navigation performance when commanded to move along a path, as shown as a dashed line in the example in Figure 1. Should objects be near the vehicle path, such as walls or obstacles, depicted in Figure 1 as bordering lines along the path, the vehicle may stop, slow, or worse, collide with the boundary object. A user would then be required to provide Pt 2 additional, perhaps unnecessary space for one manufacturers’ vehicle and not for another. How the vehicle handles (slow, stop, etc.) the event is also ambiguous. For example, some, but not all vehicles are equipped with obstacle detection based on non-contacting sensors that provide detection beyond the physical vehicle footprint. Pt 1 Figure 2. Ground Truth (red) and AGV (green) data of the straight line path tests. Scales for X and Y axes are in meters where the X axis shows only - 0.11 to -0.02 range to clearly show the AGV performance as compared to Ground Truth measurement. The blue line represents the commanded path from pt 1 to pt 2 and back. FinE-R 2015 Page 49 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Another test setup was tried, with an eye towards a A series of eight trials were completed with nearly all trials relatively less expensive test method that will allow all AGV including three or more runs each to demonstrate the systems to be measured, ideally, with an independent navigation test method concept. Ten or more runs are ideal for measurement method that doesn’t use AGV controller statistical analysis. The optical measurement system tracking, yet captures the full AGV configuration (i.e., mentioned earlier was used as an experimental ground truth including safety sensing). The AGV was commanded to drive (GT) to measure the barrier and vehicle position during back and forth between temporary barriers, along a straight experiments to further understand the test method and vehicle line defined by commanded points spaced approximately 10 m performance. The barriers and AGV were marked with apart. The goal of the experiment was to measure the AGV spherical reflectors (visible in Figure 4 (a, b, and c) detectable deviation from the commanded path. A critical AGV from the GT system. Figure 5 presents GT data plotted for navigation performance area is also deviation from the navigation tests showing ground truth data of: (a) test 8 vehicle commanded path after turns so a 90° turn was added to the end path and emergency stopped vehicle (red circle) when a wall of the straight path beyond the barriers to measure the vehicle was detected, (b) test 1 path, and (c) test 1 path data from (b) navigation uncertainty when moving from/to a straight path zoomed in to show data points of three runs. to/from a turn. Figure 3 shows the test setup and Figure 4 shows (a) a B56.5 test piece being used to define the safety laser stop field edges, (b) the barriers and lines to which barriers are moved between trials, and (c) the AGV emergency-stopped upon detection of the barriers. The safety laser, stop field edges were marked on the floor, as a ground truth, zero-tolerance spacing that the vehicle can navigate, when the vehicle was at position 1 and again at position 3, shown in Figure 3, for both left and right vehicle sides. The barrier position lines were measured from the edge line using a ruler and marked at 2 cm increments from the edge up to 10 blue barrier- cm away from the edge line. Smaller spacing between lines position lines (e.g., 1 cm) could also be used for finer uncertainty measurement. For each test trial, the barriers were moved a b c towards the AGV to the next line beginning at 10 cm for trial Figure 4. (a) B56.5 test piece (black cylinder) used to define safety laser edge 1, 8 cm for trial 2, and so forth until the navigating vehicle (note red emergency stop light (within the red circles) is on), (b) barrier detected a barrier, and emergency-stopped the AGV, thus (black) painted wood panel, blue lines spaced at 2 cm, and spherical reflector completing the test run. from ground truth system, (c) AGV emergency stopped, as noted by the red light, upon detection of barriers during a test. Experimental results from the barriers demonstrated a path uncertainty of between 6 cm and 8 cm maximum when the vehicle detected the boundaries at nearly the center of the straight line path and when moving at either 0.25 m/s or 0.50 m/s. The navigation test method using barriers is simple and cost-effective for manufacturers and users to employ, as compared to the higher accuracy, but more expensive ground truth visual tracking system used for test method development. A simple straight line with one turn was tested. However, more complex test configurations, such as shown in Figure 1, could be set up using B56.5 test pieces instead of larger, physical barriers as were used in this research. Figure 3. AGV navigation test setup. Figure 5. Example graphical results of navigation tests showing ground truth data of: (a) test 8 vehicle path and emergency stopped vehicle (red circle) FinE-R 2015 Page 50 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 when a wall was detected, (b) test 1 path, and (c) test 1 path data from (b) point, whereas the visual tracking system can track multiple zoomed in to show (red, green and blue) data points from three runs. point markers and can computer orientation from them. Both A working document that addresses quantifying vehicle GT systems can measure relatively high-precision navigation uncertainty is being developed as an initial step displacement between two points, as compared to an AGV towards a performance standard for ASTM F45.02 docking. subcommittee on Docking and Navigation. Based on An experiment using an uncalibrated AGV that was consensus of the task group developing this standard, as was programmed to stop at various points yielded an uncertainty tested at NIST, the simple path-bounding test method using range of approximately 1 mm to 50 mm. Figure 7 (a) shows temporary reconfigurable barriers made from readily- the vehicle paths and Figure 7 (b) shows average errors for five available, off-the-shelf materials is being proposed. runs at stop or dock points. The vehicle position was measured B. Vehicle Docking using a laser tracking GT system which provided high- precision measurement of AGV stop points. [13] However, in Vehicle docking is another common application of mobile several experiments, laser tracker positioning was critical as robots and AGVs. Unit load (tray, pallet, or cabinet carrying), the laser beam was continuously interrupted by onboard AGV tugger (cart pulling), and fork/clamp (pallet or box hardware. This prompted a switch to using an optical tracking load/unloading) are typical industrial style vehicles that system for GT measurements. require different docking uncertainties. For example, a unit load vehicle that places/retrieves platters during wafer A 6 DoF optical tracking GT system was used instead to manufacturing would no doubt require less uncertainty than a measure AGV docking. Docking was measured again after the fork style vehicle that places/retrieves pallets. As robotics AGV was calibrated using the manufacturer’s procedures. The advances, current and potential users are requesting mobile AGV approached similar dock locations and after AGV manipulators to perform tasks such as unloading trucks. calibration, provided consistent 5 mm uncertainty. Standards Eventually, it is expected that mobile manipulators will be development for optical tracking systems is also underway and used for smart manufacturing assembly applications [11, 12]. is discussed in section 2 D, 6 DOF Optical Measurement of Dynamic Systems. Similar to navigation, there are no performance measurement test methods that define how manufacturers and users characterize their vehicle’s docking capabilities. Figure 6 (a) shows an example method for docking for any style vehicle. A vehicle approaches and makes contact with ‘a’ and/or ‘b’ docking points dependent upon the vehicle type. Relative displacement from each of the points would be measured to determine vehicle docking uncertainty. A fork- type AGV is shown docked with a test apparatus in Figure 6 (b). The fork tips are marked with yellow points. (a) (a) (b) Figure 6. (a) Example docking test method using various AGVs (e.g., 1 and 2 for AGV unit load tray table docking, 3 for fork and tugger AGV docking). “a” and “b” are fixed points in space (e.g., contact or non-contact sensor Docking points locations in space). Approach vectors and sensor point spacing and locations are variable. (b) Fork-type AGV docking with a docking apparatus. (b) Two experiments were simultaneously performed: AGV Figure7. (a) Commanded paths and stop points and (b) stop point errors of a docking relative to known facility locations and GT system use single AGV point for each location in (a) averaged over 5 runs. for measuring AGV docking. Two different GT measurement Additional AGV equipment docking experiments were systems were used to measure AGV performance: a laser also performed using a mobile manipulator and a tracking GT with an uncertainty of approximately 10 µm [13] reconfigurable mobile manipulator artifact (RMMA) and an optical tracking system with uncertainty of 0.2 mm in developed at NIST (see Figure 8). [14] The mobile position uncertainty and 0.13° in angle uncertainty as manipulator, with uncalibrated AGV, repeatedly moved next measured at NIST. The laser tracker tracks position of a single to the artifact from a starting point. Although uncalibrated, the FinE-R 2015 Page 51 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 AGV provided relatively low repeatability uncertainty (e.g., +/-5 mm) although more than 10 mm from the commanded docking points. This manipulator could reach the commanded points on the RMMA even with 10 mm uncertainty in AGV position. The mobile manipulator corrected for the position uncertainty after being taught the actual RMMA locations. At the RMMA, the manipulator, wielding a laser retroreflector, was commanded to move in a spiral pattern to detect 6 mm Figure 9. Graphical output of path planner, starting footprint of the AGV is in white, the goal position is a dark grey rectangle. Yellow rectangles show the diameter reflectors. The reflectors provide non-contact area swept out as the AGV would travel, blue curve shows the resulting alignment detection of the tool point position and orientation. spline, and orange circles represent obstacles. The experiment provided results demonstrating that this relatively inexpensive ground truth measurement method was The navigation performance measurement experiment sufficient for measuring docking accuracy. As the reflector discussed previously in section II A. Vehicle Navigation can based measurement system is inexpensive compared to the be similarly applied for obstacle detection and avoidance. In optical tracking-based GT, it may prove ideal for use as a fact, the ASTM F45.02 subcommittee navigation and docking precision vehicle/mobile manipulator docking test method that task groups have discussed the potentially overlapping nature both manufacturers and users can replicate. of the two vehicle capabilities. The ASTM F45.03 Obstacle Detection and Protection subcommittee is currently in the process of considering standards in this area. Questions have been raised regarding standards development as follows: Manipulator 1. How well does the AGV react to situations? For example: RMMA  Obstacles appearing in the path  Potential obstacles headed towards the path  Unstructured (i.e., changing obstacle locations) AGV areas not on the original planned path or that rapidly change 2. How far off the commanded navigation path can an AGV be, and at what speeds, before it violates the path and causes a stop? For example, due to environmental factors such as:  Offset-pitched/rolled AGV can’t see guidance markers, such as reflectors, magnets, wire, etc. Figure 8. Docking performance measurement of a mobile manipulator with a  Guidance or boundary-marking tape is worn or reconfigurable mobile manipulator artifact (RMMA). broken  Terrain causes “bouncing” or moving laser or other C. Obstacle Detection and Avoidance navigation sensors Obstacle detection and avoidance (ODA) research is well 3. How well does the vehicle react when a human is documented in the literature for mobile robots. However, detected and how should the human be represented? For there are few citations for AGVs perhaps due to the relatively example: closed nature of commercially available AGV controllers and  By test pieces, mannequins, humans because ODA is not often implemented on AGVs deployed in large manufacturing facilities. In [5], it was discussed that for  With what coverings? (i.e., what clothes should be large facilities, ODA could occur in ‘buffer zones’ (i.e., zones worn?) where AGVs would be allowed to pass other vehicles). For 4. How to interact with manual equipment (e.g., forklifts, small and medium manufacturing facilities, however, ODA machines) may be necessary due to more limited floor space and less- 5. How to standardize communication of vehicle controlled environments. NIST has developed an algorithm, intelligence for obstacle detection and avoidance? For detailed in [5], and measured the performance of an AGV with example: added ODA capability. The algorithm is also suitable for  Contextual autonomy levels [4] navigating an unstructured environment although it is  Situation awareness (e.g. LASSO) [14]: currently limited by the use of facility-mounted (sensors not mounted on the AGV) obstacle detection with obstacle Experiments to support ODA performance test method avoidance adapted to an AGV with a controller with limited development will be performed based on forthcoming ability to integrate external algorithms. Figure 9 shows a guidance from the ASTM F45 subcommittee. However, a snapshot of the ODA algorithm planning a path through prototype safety test method that has been developed to multiple obstacles. evaluate a vehicle’s response to obstacles in its path and within its stop zone, as noted in the Introduction, can be considered a first step towards full ODA standard test methods. ASTM F45 is meant to dovetail with safety standards such as ANSI/ITSDF B56.5. Therefore, providing an initial test FinE-R 2015 Page 52 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 method for detection of obstacles is ideal as a starting point for maximize metrology bar movement during dynamic F45.03. The ‘Grid-Video’ detection method [3] provides a measurements. simple-to-implement test method that measures positional accuracy of the dynamic test piece relative to the vehicle position when the obstacle enters the vehicle path. D. 6 DOF Optical Measurement of Dynamic Systems ASTM’s draft Standard for the Performance of Optical Tracking Systems that Measure Static and Dynamic Six Degrees of Freedom (6DOF) Pose (see Figure 10) is the next step beyond the static case covered by ASTM E2919-14 [8]. Optical tracking is being used for robot and autonomous vehicle GT measurement, as discussed in this paper. Optical tracking measurement systems [15] are used in a wide range of fields, including video gaming, filming, neuroscience, biomechanics, flight/medical/industrial training, simulation, Figure 9. (a) Proposed metrology bar, (b) Example frame used to move the metrology bar. and robotics. ASTM WK49831 is a working document that is considering both static and dynamic measurements of systems Most optical tracking systems have at least a 30 Hz data under test. The scope of the draft standard test method is to collection rate. Therefore, a minimum of 5 min of data needs provide metrics and procedures to determine the performance to be collected. The workspace is uniformly divided by the of a rigid object tracking system in measuring the dynamic artifact length. The artifact is moved using at least the pose (position and orientation) of an object. Optical minimum and maximum motion capture velocity specified for measurement systems may use the test method to establish the the system. performance for their 6 DOF rigid body tracking pose The static test procedure for measuring the performance measurement systems. The test method will also provide a of the optical tracking system is to divide the test space into a uniform way to report the statistical errors and the pose grid and place the artifact at intersections of the grid and at measurement capability of the system, making it possible to various orientations. The dynamic test procedure also divides compare the performance of different systems. So all the the test space into a grid where the metrology bar is moved in measurements can be traced to the standard. a raster scan pattern forward-to-back and left-to-right Ground truth throughout the space. cameras The metrology bar maintains a constant separation and orientation of the two marker clusters along all the paths and can be rigidly attached to and moved using a wheeled frame as illustrated in Figure 9 (b) that is pushed/pulled by a human, a mobile robot, or other mover to closely follow the path. The metrology bar is moved at the maximum specified AGV velocity of the optical tracking. Pose error measurement and reporting methods are also described in the ASTM WK49831 [8] working document. III. CONCLUSION The AGV standards development process has been limited for many years to considering only safety standards. Starting in late 2014, ASTM F45 Driverless Automatic Guided Industrial Vehicles performance standards are being Figure 10. (top) autonomous vehicle test lab and (bottom) screenshot of the developed to include navigation, docking, terminology and perception ground truth system space showing cameras and vehicle rigid several other key areas for AGV’s, mobile robots, and mobile body. manipulators. As discussed in this paper, standard test In the initial test procedure, measurements with methods for measuring vehicle performance are being uncertainties were computed using an artifact – namely a developed so that manufacturers and users of these systems metrology bar as shown in Figure 9 (a). Current optical can easily replicate the measurements in their own facilities tracking systems utilize a three-marker metrology bar with all and at minimal cost and effort. More AGV and mobile robot markers in a line which does not provide 6 DOF system systems, instead of just the one AGV used in these performance measurement. A metrology bar made of carbon experiments, would ideally validate the generic test method fiber with length 620 mm and with five reflective markers proposed. attached on each end was used as the 6 DOF artifact. A carbon A comparison of GT measurement systems was also made fiber bar is used since it limits the effects of thermal to support the test method development. It was determined expansion. The metrology bar markers on each end form a that for dynamic AGV measurement, an optical tracking constant relative 6 DOF pose between the two ends. A shorter system provided a suitable ground truth measurement. At the bar length should be used for smaller space measurements to same time, a standard for these dynamic measurement FinE-R 2015 Page 53 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 systems is also being developed. The standard will allow vehicle and robot performance standards developers to use the systems as ground truth with known measurement uncertainty. Optical tracking systems users and manufacturers can replicate the same test methods with similar tracking systems and use the results to compare their performance at dynamic tracking tasks. ACKNOWLEDGMENT The authors would like to thank the ASTM F45.02 subcommittee navigation task group and Omar Y. Aboul- Enein, Salisbury University student, for their recent input to navigation test method development and experimentation. Also, we thank Sebti Foufou, Qatar University, Doha, Qatar, for his guidance on the mobile manipulator docking performance measurement research. REFERENCES [1] ANSI/ITSDF B56.5:2012, Safety Standard for Driverless, Automatic Guided Industrial Vehicles and Automated Functions of Manned Industrial Vehicles, www.itsdf.org, Nov 2012. [2] British Standard Safety of Industrial Trucks - Driverless Trucks and their Systems. Technical Report BS EN 1525, 1998. [3] Bostelman, Roger, Will Shackleford, Geraldine Cheok, and Kamel Saidi. "Safe Control of Manufacturing Vehicle Research Towards Standard Test Methods." In Proc. International Material Handling Research Colloquium, pp. 25-28. 2012. [4] Roger Bostelman, Elena Messina, “Towards Development of an Automatic Guided Vehicle Intelligence Level Performance Standard”, Autonomous Industrial Vehicles: From the Laboratory to the Factory Floor, Chap. 1, ASTM International, to be published 2015. [5] Roger Bostelman, Tsai Hong, and Geraldine Cheok, “Navigation Performance Evaluation for Automatic Guided Vehicles”, IEEE International Conference on Technologies for Practical Robot Applications (TEPRA), Boston, MA, April 2015. [6] ASTM International, Committee F45 on Driverless Automatic Guided Industrial Vehicles, www.astm.org/COMMITTEE/F45.htm, 2014. [7] Roger Bostelman, Will Shackleford, "Obstacle Detection and Avoidance from an Industrial Automatic Guided Vehicle," IROS 2014. [8] ASTM International, E57.02 Standard Test Method for Evaluating the Performance of Rigid Body Tracking Systems that Measure Dynamic, Six Degrees of Freedom (6DOF), Pose, Work Item #WK49831, June 2015. [9] ASTM International E2919, Standard Test Method for Evaluating the Performance of Systems that Measure Static, Six Degrees of Freedom (6DOF) Pose, http://www.astm.org/Standards/ E2919-14.htm [10] NDC 8 jAGV Control System, http://www.kollmorgen.com/en- us/products/vehicle-controls/electrical-vehicle-controls/ndc8/, 2015. [11] Hvilshøj, Mads, and Simon Bøgh. "" Little Helper"-An Autonomous Industrial Mobile Manipulator Concept." International Journal of Advanced Robotic Systems 8, no. 2 (2011). [12] Roger Bostelman, Tsai Hong, Jeremy Marvel, “Performance Measurement of Mobile Manipulators,” Proceedings SPIE DDS 2015, Baltimore, MD, April 2015. [13] Burge, James H., Peng Su, Chunyu Zhao, and Tom Zobrist. "Use of a commercial laser tracker for optical alignment." In Optical Engineering+ Applications, pp. 66760E-66760E. International Society for Optics and Photonics, 2007. [14] Drury, Jill L., Brenden Keyes, and Holly Yanco. "LASSOing HRI: analyzing situation awareness in map-centric and video-centric interfaces." In Human-Robot Interaction (HRI), 2nd ACM/IEEE International Conference, pp. 279-286. IEEE, 2007. [15] “Motion Capture Software Developers in the US: Market Research Report,” IBISWorld 2014. FinE-R 2015 Page 54 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Gualzru’s path to the Advertisement World F. Fernández, M. Martı́nez I. Garcı́a-Varea, J. Martı́nez-Gómez J.M. Pérez-Lorenzo, R. Viciana Planning and Learning Group SIMD M2P University Carlos III Madrid University Castilla-La Mancha University of Jaén Email: {ffernand}@inf.uc3m.es Email: {ismael.garcia,jesus.martinez}@uclm.es Email: {jmperez, rviciana}@ujaen.es P. Bustos, L.J. Manso, L. Calderita, A. Bandera, A. Romero-Garcés, M. Gutiérrez, P. Núñez J.P. Bandera, R. Marfil RoboLab Ingenierı́a de Sistemas Integrados University of Extremadura University of Málaga Email: {pbustos, lmanso, lvcalderita, marcog, pnuntru}@unex.es Email: {ajbandera, argarces, jpbandera, rebeca}@uma.es Abstract—This paper describes the genesis of Gualzru, a robot first ’robotic’ consortium was composed by the Universities of commissioned by a large Spanish technological company to Málaga, Extremadura and Carlos III of Madrid. From 2010, provide advertisement services in open public spaces. Gualzru the first two groups were working together on the definition has to stand by at an interactive panel observing the people passing by and, at some point, select a promising candidate and of a software framework for robotics, which could be used for approach her to initiate a conversation. After a small verbal the project. The University of Extremadura would also build interaction, the robot is supposed to convince the passerby to the platform and would be the responsible of endowing the walk back to the panel, leaving the rest of the selling task to robot with the abilities for autonomous navigation and facial an interactive software embedded in it. The whole design and emotion detection. The University of Málaga would address building process took less than three years of team composed of five groups at different geographical locations. We describe here the rest of vision-based problems (e.g. use facial descriptors the lessons learned during this period of time, from different to estimate the gender or age) and help with the navigation points of view including the hardware, software, architectural modules. Finally, the University of Carlos III of Madrid would decisions and team collaboration issues. be in charge of the high-level planning and learning modules. I. I NTRODUCTION Everyone agreed in using this project as a test for the initial proposal from the University of Extremadura: to organize Gualzru is a social robot built as an advertisement tool for the whole software architecture around a centralized internal a consortium of technological and digital media companies model of the outer world. Such representation is accessed by within the ADAPTA1 project. The core of this project is an all software components to keep them informed about the interactive panel able to provide personalized advertisement current world state. They can also update it as the result of according to the preferences of the user. To achieve this goal, processing the data from the sensors. The ADAPTA project the consortium includes advertising companies, media asset will provide a controlled but realistic scenario for testing the management, software developers, technological consultants idea. and software infrastructure providers, coordinated by the Soft- One major requirement of the proposal that the robot ware Labs group from the Spanish Indra company. The idea initially lacked was the ability to dialogue with people. To of using a robot as a more personal way of bringing people’s solve this problem the SIMD group from the University of attention was suggested in order to endow the panel with Castilla-La Mancha, summing a large expertise on automatic the ability to recognize the emotional state of the user and speech recognition and natural language processing, joined the to classify her according to the estimated age and gender. Consortium. Furthermore, the human-robot interaction ability After agreeing on creating a new social robot, it was decided was strengthen with the incorporation of the researchers from that Gualzru, would team up with the interactive panel and the University of Jaén. Six researchers from all groups were boost the advertisement potential of the platform. The project contracted during different periods of time to work on the started in May 2012 and this paper describes our experiences, project, however a larger group of researchers was always successes and failures, during the three-year process. involved on the project. II. T HE TEAM A. Team coordination and sharing of resources From the early analysis of the problem we knew that the project needed the expertise from different research groups. The coordination of this large group was supported by the So, once a solid group of complementary researchers was use of collaborative tools. However, we soon understood that agreed, we accepted to join the project’s consortium. Our the only way to make a steady progress in the development of a large and complex project like ADAPTA, was by sharing 1 See Acknowledgments section at the end. a common code base and by scheduling periodic hackathons FinE-R 2015 Page 55 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 of the project. It was a relatively new process for us with many steps that were out of our direct control. Going in Table I from top to botton and from left to right, we can rapidly summarize the manufacturing steps: 1) Gualzru’s initial 3D design. To come up with a nice robot image we set up a public design contest among all Spanish universities and people and companies in the design business. One person from Cádiz, Spain, was selected among more than 30 proposals with a poll among a selected resolution committee. 2) The 3D drawings were sent to a company specialized Fig. 1. Gualzru the robot in manufacturing expanded polystyrene molds using industrial CNCs. We learned that the choice of prices and qualities here are apparently important, since the in which the members of all the teams could seat together final quality of the surface of the cover and the number for a week and fight a specific, common battle. This strategy of hours spent by the sculptor in fixing the small naturally led to the division of the project in well defined imperfections generated in the machining process were milestones, consisting on system features to be integrated and closely related. It is important to assure the final quality tested during the one-week period. Coordination was therefore level in this early stage. subtended on: 3) An external coating over the mold is necessary to • A unique robotic prototype, available from month zero in facilitate the unmolding process. The mold is split in a robotics simulator. two halves. • A common programming framework, RoboComp [17], 4) A thick silicone layer is manually applied on the mold used in several previous projects. All the software de- with additives to avoid sagging. This layer is called veloped for the project had to qualify as a RoboComp negative. component, meeting the established quality standards, 5) On top of the silicone a resin with fiber glass layer is and had to be uploaded to a common git repository. applied to create a rigid external cover called mother. • A common cognitive architecture, RoboCog, available for 6) Both layers are unmolded. all researchers and where individual modules could be 7) The silicone mold after being separated. inserted and tested minimizing the knowledge required 8) A positive mold is finally built by applying resin and about the rest of the architecture fiber glass inside the negative. After drying, the cover • The organization of several intensive working weeks - is unmolded from the silicone and both parts are glued hackathons- coinciding with the project milestones. These together. A final polish work is done to obtain a nice meetings were intense and dedicated to integrate and texture. debug specific target functionalities. 9) A solid and reliable differential base is built as the To maintain the global view of the project and of the specific mechanical core of the robot. requirements, all the members should meet for each milestone 10) The cover is fit on the base. Additional holes and slits and have always access to an open document storing this in- have to be carved to allow for laser, camera, fastening, formation. The document was edited online by all researchers etc. and also served as a battlefield to discuss technical issues. We 11) Sensors are incorporated to the robot. The tactile screen did not always coincide about how to do things but we agreed is placed after a final coating is ordered to a car painting that the digital arena was the right place to fight. workshop. III. G UALZRU 12) Gualzru at the University of Málaga in a public event with the University’s Provost. Gualzru, a phonetic transcription of the English phrasal verb ”walk through” pronounced by a native speaker of As a summary of the experience it is evident that the process Extremadura, is a 1.60m. tall robot with an external cover is slow, expensive in working hours and almost impossible built of resin and fiber glass, and a differential base with to rectify if a new idea comes by. The whole process took two powered wheels and two casters. It includes gel lead us many more months than expected and we had to use a batteries that provide an autonomy of three hours and all the replacement Nomad 200 robot while the robot was being built. necessary power electronics, recharging and power supplies In summary, it is a valid solution to the cover problem but for the sensors and processors. The complete fabrication of with the arrival of 3D printing technology, all chances are that the robot was custom made by the groups of the consortium. future robot covers will be divided in pieces small enough Table I shows the complex handcrafting process of to be printed in a modern 3D printer, and then assembled Gualzru’s external cover. This step was one of the most together. There are also new small companies starting to offer exasperating and time-consuming in the overall development these kind of services. FinE-R 2015 Page 56 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 TABLE I A SERIES OF SNAPSHOTS OF THE BUILDING PROCESS OF G UALZRU . S EE TEXT FOR DETAILS ON EACH STEP AND THE CONCLUSIONS OBTAINED AFTER IT WAS FINISHED . IV. ROBO C OMP • RoboComp’s component model has been evolving since its beginning and has the necessary complexity for our As commented in section II-A, one of the few things that needs. Not more. were already clear when the project started, was the need of • The current communications middleware, Ice by ZeroC a common code base. [14], is extremely robust. No complaints and a big thank A big part of the group had been already working in you to an excellent open source project. • New middlewares could appear in the future with some previous projects together and sometimes with other partners. From these works we learned that one of the main causes game-braking features. In that case, if you control the that prevented the formation of a cohesive, long lasting group framework you can define a reduced set of communica- with a common goal was the fact that each one was coding tion primitives, like the ones proposed by Schlegel in his their own programs on different frameworks or without one PhD thesis [1] and a set of data types, and write some at all. There are many robotics labs around, still unable to interface code that makes you framework middleware organize and create a coherent code base that grows from the independent. • A code generator is mandatory so the generic part of the accumulated work of dozens of researchers. After some tough negotiations involving the different frameworks that the groups components is always the same, compiles without errors were using or planning to use, we agreed to use RoboComp. and keeps the required quality levels. Code re-generation We believe there are several reasons that, in the hindsight, might be trickier but there are several techniques. Robo- justify this decision: Comp splits the working part of the component in two using inheritance. The inherited part is always generated • We keep the control of the core and thus, we decide when and the part that inherits is generated only the first time. to change and when to hold. It looks like a contradiction A lesson learned here is that it is easier if all tools and but when some complex open source software is very technology in the framework use the same development soon used by thousands of people, its evolution freezes language and environment. Better if it is the one that most or slows down almost immediately. The reason is that of the users are familiar with. Otherwise the no common the core decisions made at the very beginning cannot be specific tool becomes a bottle-neck that might delay and easily changed without generating compatibility problems affect other parts of the framework. and versions nightmares. As an example you can look at Our initial code generator was created with the Eclipse the widely expanded Microsoft’s operating system (Win- ecosystem using the existing tools it provides for DSL dows) and the relatively slow addition of new features designs. This tool turned out not to be easily adjustable by with each release (mainly nothing on the core changes). developers (since they mostly develop C++ and Python) This does not mean that good software cannot be used by and a heavy environment that would not exactly match the many people, but that complex software that deals with team needs. Therefore, we ended up rewriting a lighter new, changing, not very well defined sort of things, takes code generator in Python using pyparse and COG so its time to settle down. FinE-R 2015 Page 57 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 everybody could collaborate in the natural evolution of This now familiar scheme was synthesized by Erann Gat as the tool. Now, RoboComp’s code generator generates the three-layered architectures [2], probably the most extended also Python components, that are becoming more a more approach to build deliberative-reactive agent control systems popular due to their simplicity. today. However, when making decisions that directly involve • We have developed all the tools we needed, although human users, the domain of HRI, these architectures present there are always tools that we would like to have but we some limitations. The most important one, from our point of have not had time to code them. RCIS deserves a special view, is the need of a shared representation among all agents mention, RoboComp’s simulator, that it has been there including metric and symbolic information, making each of almost since the beginning of the framework. By the time them more aware of what was going on in the rest of the RoboComp started the only existing open source simula- agents. For example, if a navigation module is driving the tor was Gazebo and it was in its early versions. If you are robot to a target place, and a person appears somewhere building a robotics framework and have already decided close to the planned path, how does the navigation agent on the communications middleware, the chances are that differentiate between and obstacle and the person, so different you want a simulator that speaks the same language avoiding (social) behaviors can be elicited? Or how does a as the components of the framework. Only doing so, conversational module knows that the person the robot is the simulator would behave like a component or several talking to, is not paying attention anymore, and thus a change components with all the advantages that come with that. in the discourse is advisable? Therefore we wrote RCIS using Open Scene Graph [20] To us, it looks like that the good engineering practice of de- and an initial scene specification language that we named coupling the problem in parts of infinite impedance, took away InnerModel. Later on, we discovered with great joy that a crucial element, context. It is sometimes argued that context having our own simulator would immediately provide us is somehow coded in the interactions between agents [3] or in with an emulator. That is, a simulator that could be run the dynamics of coupled differential equations [4]. We decided inside the architecture computing in super real-time future to take here the more classical path of building an explicit courses of action and predictions. That is now part of our shared representation for the context and face the problems to new architecture CORTEX, which is still in development. come. To start, we already had a representation of the robot V. ROBO C OG and its close environment in the form of a scene-graph, called InnerModel. This simple DSL served to initialize our Initially, we addressed the ADAPTA project from a very RoboComp’s 3D simulator, RCIS. Therefore, RoboComp’s specific point-of-view. That is, giving the use case, we trans- InnerModel was the perfect starting point to develop the idea lated it to a finite state machine and assigned tasks to software of a shared representation of the robot, the environment and components or groups of them that we call agents. The the people in it. The initial scene graph specification language idea of using a finite state machine to manage the whole was gradually extended to include more types of objects. Also use case was soon unbearable. The number of states and a C++ class was written to hold in memory the graph and allow transitions grew with every bit of reality added to scenario. an easy and safe access to all the handy functionalities that Even modern hierarchical and concurrent formalizations of this structure provided, such as coordinate transformations, state-machines [8] and ready to go implementations such as measuring, insertion, modification and removing of nodes, the Qt StateMachine Framework [9] did not offer enough perspective changing, frustum reachability, etc. flexibility and maintainability to risk a project with many A basic scene-graph is essentially a kinematic tree with potential implications for our future. some add-ons. We had to incorporate all the symbolic in- We thus decided to take the hard way to a fully fledged formation needed by the deliberative elements of the ar- symbolic planning system, in charge of the automatic gen- chitecture. The requirements were that the perception-action eration of those huge state machines. The Planning and related agents could update a fixed set of symbolic attributes Learning Group at the University Carlos III of Madrid had and predicates, and that the selected representation could be a very long trajectory in these disciplines and was the perfect efficiently translated to PDDL, so a specialized planning, match to provide the needed technology. The use case was executing and monitoring framework like PELEA [5] could be translated into a PDDL domain specification [18] and several used. Our election on how to proceed in that situation was not planning algorithms were tested for that domain. A separated exactly a mistake but it was certainly close to it. We decided interface was clearly defined between high and low level to build a second graph, this time a graph rather than a tree, to domains. High-level being the domain of logic attributes and hold this symbolic data and leave to a less stressed moment predicates, and low-level the domain of behavior agents that the problem of how to integrate both structures. It was not receive parametrized calls to act and provide metric values a mistake in the sense that the solution worked well and the for relevant variables of the world state. The interface layer robot managed to complete the use case. It was a mistake translates between high and low level, so both worlds are kept in the sense that now, months after the end of the project, communicated. Of course, it is also the main cause of the so we are hurrying to finish the integration of both structures called, symbol grounding problem [11]. because the separation is already generating many problems. FinE-R 2015 Page 58 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 The current solution we are working on is the embedding of may be some limited capability of planning or sequencing sub the kinematic tree inside the symbolic graph, and the code tasks, e.g., maintain the interest of the person through dialog, necessary to efficiently extract and insert the tree in a format monitor the correct execution of a gesture, recognize her facial that can be used by the many components that were written emotions, etc. The most basic components are in charge of before the integration. It is hard to evaluate if the other choice sensor motor loops and normally execute their commands would have permitted us to finish the robot on time, saving the without interaction with other components. posterior integration step. Software developing time is really VI. T HE USE CASE hard to estimate, specially when robots are in the loop. The new graph was named AGM, for Active Grammar- We now describe the use case that constituted the main based Model [6], and besides fulfilling both requirements, it goal of the project. It was defined in the ADAPTA’s kick-off was also an experiment on planning with a variable number meeting. The first version of this use case is depicted in Figure of symbols. In HRI, the perception component is getting 3. It can be textually described as, more and more important. When interacting with a human Gualzru is waiting in the Waiting area. It is now in domestic or service environments, there are references to ready to start one of its tedious working days. The objects and places that may not be known beforehand. Of Waiting area is at the middle of an uncluttered course, planning in an open world takes you out of the comfort corridor in a large shopping center. People usually zone of algorithms, where you know that the program will enter this side of the mall from the left side of the finish. In open worlds, there is always the possibility of adding corridor, crossing in front of the Panel area before a new symbol if the solution does not arrive. entering the shops. People going out the mall also AGM and InnerModel where independent structures and cross in front of the Panel area, but walking toward their coordination was managed by ad-hoc procedures, but the left part of the corridor. The objective of Gualzru AGM was finished on time and will maintain a complemen- is to offer products and services to all these people. tary, symbolic representation of the robot and the objects in In fact, its aim is to drive potential consumers to the scene. For example, a human in front of the robot being an advertising panel, in which these products and detected and represented as a skeleton inside InnerModel, services will be displayed. As there are products for could now be tagged happy, focused or woman in AGM. With everybody, it can choose any person in the corridor. AGM we had the missing part of the architecture and all the When it chooses a target, it moves from the Waiting groups could start to meet in hackatons and reach, one by one, area following an intersecting trajectory with the the urgent remaining milestones of the project. In summary, person’s heading direction. This displacement is very the main purpose of this dual representation was to provide short (2-3 meters maximum) and allows Gualzru to both, a local description that could be updated and used wait for the person in a static pose, facing her at by the different agents for their computations, and a shared comfortable social distance (1,5-2 meters minimum). context that is propagated among them to carry information Therefore, Gualzru can say ’hello’ to the person that otherwise would remain hidden. without scaring her even if she is not very used to As a result of the graphs occupying their places with the interact with a moving robot. agents, the overall idea of RoboCog started to change and If the person engages with him in this first contact, we started to move from the three-tier original model to a Gualzru will classify her into a group -using gender non-hierarchical disposition in which all agents gather around and age parameters- and will choose a product these shared graphs and interact among them by reading, topic to offer. Product topics provide Gualzru an writing and propagating the changes. The abstraction axis is specific theme of conversation before inviting her hidden inside the agents and defines what parts of the graph to walk back to the Waiting area. During this short are accessible by its internal components. This is discussed conversation, Gualzru will be always ready to say in recent works by the group [7], [12]. Figure 2 shows a goodbye to the user if she shows the intention of schema of the RoboCog architecture by the time of the final leaving the conversation or if the presented product demonstrations of the project. topic is not interesting to her. On the other hand, When a mission is assigned to the robot it is internally Gualzru must also check its batteries level to say re-coded as a desired state in an AGM graph, which could goodbye and move to the Charging area if this level include the whole world or just the symbols needed to satisfy is under a minimum value. The Charging area is the mission. The Executive module is the one in charge close to the Waiting area. In fact, when Gualzru of achieving it. The steps needed to transform the current arrives to the Waiting area, he will home itself to AGM graph are provided to the Executive by the Task-based the Charging area. If the person agrees on going Planning and Monitoring module PELEA as a sequence of with Gualzru to the Waiting area, both move there tasks that are injected back in AGM. At this point, AGM holds and the robot says goodbye to her. Then, it returns the current belief about the world and the current desire about to the Waiting area and waits for some time, which how the world should be. Agents scan the graph and find is the expected time that the person is going to be tasks that can be performed by them. Inside each agent there at the panel, before starting the process to select a FinE-R 2015 Page 59 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Fig. 2. An overview of the RoboCog architecture (from [16]) new target. As before, if batteries level are under a in the way. Hackatons have turned out to be an effective way certain value, Gualzru moves to the Charging area to code, debug, test, share, make progress and build a team for a reload. spirit. A. May 2012. Kick-off The project initiated with a kick-off meeting at Málaga where the overall strategy was discussed and the periodicity of the meetings was set. B. December 2012. The ”WORST” workshop at Cáceres The main objective of the first hackathon was to explain and establish RoboComp as the common code base. All groups on the consortium had certain degree of knowledge about RoboComp, but it was considered mandatory to organize a workshop where simple examples could be programmed by all researchers under the supervision of experts from the Universities of Extremadura and Málaga. Fifteen people from all research groups and some more and some from Indra Software Labs met at Cáceres. C. May 2013. First public demonstration at Málaga Fig. 3. The ADAPTA use case For the first public demonstration of the whole project we had a simple prototype of Gualzru (Figure 4). Two autonomous VII. A N ANNOTATED DIARY behaviors were tested: the reactive navigation and a face As was mentioned before, the coordination of the project detector. The AGM graph and the kinematic tree were able to was based on periodic hackathons. We think this decision internalize the perceived information. The seed of the architec- was a real success. We have already noticed the difficulty to ture was planted. Obviously, not everything worked properly. integrate complex software and reach milestones without a real The algorithms underneath both behaviors were changed in motivation from the people working in different labs. Many the final version, for example. But this fact was rather usual times, the global objective or the potential implications of the during the project. Other issues were more time consuming work are not correctly perceived. Other, personal relations get as expected. During 2013 we tried to replace the laser by FinE-R 2015 Page 60 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Fig. 5. Playing with the ’empty-boxes’ architecture. The dashboard shows different panels: the one on the right shows the graph models that encodes (a) the goal to achieve -target model plan, and (b) the symbolic view of the outer world. It also includes the current action of the plan (’approachperson’ in this Fig. 4. The initial internal skeleton of Gualzru figure). The panel on the left shows a visualization of the kinematic tree -up- and of the virtual world -down. We did not endowed the virtual robot with virtual sensors. The person is automatically detected if she is in front of the robot. an array of RGBD sensors, arranged in a configuration that provided a wide field of view and theoretically gave good 3D coverage. Researchers from the Universities of Extremadura interaction stage. The RGBD sensor was used to detect the face and Málaga were involved on achieving this goal. The sensor of a real person during the conversation. It was an intensive had limitations to perceive at short distances and we had to integration task. connect them to embedded computers like the Raspberry Pi of One of the major successes of this architecture was the the time to liberate the USB ports in the main computers of development of the triangle, high-level decision maker - exec- the robot, Intel’s NUC. We could not make the Asus’ Xtion to utive - symbolic graph model. We were now able to translate run reliably with the available Raspbian. A few months after to PDDL the information stored and updated in the symbolic the end of the project, we succeeded with another board, the graph (AGM) to the PELEA framework at the deliberative ODroid C1 [13], and now we can create hard-components that level. Furthermore, the Executive module was able to publish are cheap and provide real-time performance. We spent a few the graph to all software components on the architecture months trying to make that device work because we though it when a change was introduced. These components were ar- should work. Clearly, it was not the time. You have to choose ranged on networks, connected to the Executive through one the right battles. distinguished component, the so-called agents. These agents were the responsible of maintaining the coherence of the D. November 2013. First evaluation of the architecture at information stored in the inner model, since they update the Albacete graph-model and, simultaneously, the geometric information One of the major goals achieved after the meeting at Málaga of the kinematic tree. This second route was not supervised by was the development of a complete architecture able to work the Executive. For the first time, the new definition of agent, with a simulated robot in a virtual environment. The so-called included formally in the RoboComp component model and ’empty boxes’ architecture took this name from the fact that code generator, allowed all participants to share the graphs it included a complete version of the architecture RoboCog, using the same interface. Our shared global representation on although some of the components only had the public interface the state was now real and working. -IDL file- and the structure inside. Nevertheless, it included We were able to launch more than fifteen software com- the two inner graphs -the symbolic and the geometric-, the ponents. From this point of view, we were able to modify or conversational module, reactive navigation, person detector add new components over a full-integrated architecture. Each and high-level planning, executing and monitoring. To play successive meeting would imply a refinement of the previous with it we did not need the physical robot, but a computer proposal. While waiting for the robot Gualzru, see Section III with a RGBD sensor, speakers and a joystick. The simulated for reasons explaining the long wait, we set up an old Nomad robot operated as an autonomous agent and we were able to 200 robot with the RoboCog architecture and organized a new move a virtual person in the simulator using the joystick. When meeting at Albacete. See Figure 6. robot and person were at interaction distance, the robot tried At Albacete we evaluated for the first time the robot’s to convince the person to accompany it to the advertisement behavior through questionnaires filled by the people interacting panel using his conversational skills. The speakers and the with the robot. The questionnaire is designed as a Likert microphones on the RGBD sensor were used to support this scale, although it uses six levels, from 0 to 5, to remove the FinE-R 2015 Page 61 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 TABLE II Q UESTIONNAIRE RESULTS (50 TESTS ) Question x̄ σ 1.1 Do you feel safe when the robot approaches you? 4.31 0.95 1.2 Does the robot invade your personal space? 0.96 1.37 1.3 Do you think robot movements are natural? 2.62 1.23 1.4 Have you stepped away from the robot? 0.96 1.46 2.1 Have you understood the robot? 3.57 1.28 2.2 Has the robot understood you? 2.70 1.30 2.3 Was the conversation coherent? 2.96 1.38 2.4 Do you like the voice of the robot? 3.13 1.29 3.1 Did the robot get blocked? 1.39 1.72 3.2 Was the interaction natural? 3.11 1.11 3.3 Was the conversation fluent? 2.85 1.22 3.4 Did the robot seem to be tele-operated? 0.87 1.44 4.1 Did you enjoy the experiment? 4.31 0.88 4.2 Do you think the exp. was not interesting? 0.70 1.32 4.3 Would you like to repeat? 4.28 1.32 4.4 Would you recommend it to other people? 4.52 0.86 Fig. 6. The old Nomad performing through the use case at Albacete did this integration without time to test the whole system. Also, we put the emphasis on collecting a larger collection of questionnaires. And to worsen things even more, the meeting neutral option -middle point. It is similar to that employed was not set as the other previous hackathons: the goal did not by Joosse et al. [15] to generate the database BEHAVE-II. focus on solving technical problems that were really there, Its main difference is that it has been created not from the but on showing a prototype that, at the end of the day, we point of view of the person observing the behavior of the user should have known that it would not do the job. And the against the presence of the robot, but from the point of view results were not good. During the demonstration the robot was of the same user that interacts with the robot. In this sense, able to interact with a person but it showed its limitations: the we can consider that it collects influences of questionnaires odometry alone was not able to correctly solve the localization of the Almere original model for man-machine interaction. In problem, speech recognition had problems to work on crowed particular, the questionnaire includes a collection of questions environments, the person classification module blocked the full arranged in four blocks: navigation, conversation, interaction use case until a good image of a face was taken, and so on. and general sensations. The user fills the questionnaire giving Another significant lesson was learned and never forgotten: a value for each response between 5 (completely agree) and you can not say that your robot will succeed in one trial until 0 (completely disagree). These questions are listed in Table you have tested it for at least hundreds of times. There is a II. From that point on we would use that tool to evaluate the saying that can be applied here: ”let’s rehearsal so hard that evolution of the project. Although at Albacete we were able the show looks like a rest”. to run the software in a real robot, we only could finish 12 use cases with different users. The number of questionnaires F. June 2014. Hackathon within the Workshop on Physical were also reduced to take hard decisions based on it, however Agents the results were promising. In June 2014, all groups had talks within the Workshop on Physical Agents (WAF2014) to be held in León (Spain). E. May 2014. Second public demonstration at Málaga We asked the local organizers to facilitate us a working space The Nomad 200 was moved from Albacete to Málaga and to set up another hackathon during the week before. Vicente we continued testing specific problems related to the naviga- Matellán, the conference director gave us a cordial welcome tion module, speech generation and recognition, and person and provided an excellent place for testing. classification based on age and gender. The communication Before the hackathon, we discussed and organized the prob- among components was forbidden and all information was lems to solve there and when we arrived to León everybody transmitted through the inner representation. Things still did knew what to do and joined in groups for a long week. not work as we needed and the causes were not clear. Then, The result was a real success: the dialog module was largely the robot Gualzru arrived and all efforts were translated to improved and tested, the localization problem was solved using getting it ready. AprilTags landmarks, and so on. The use case was repeated For the second public demonstration Gualzru was already and repeated, and for the first time we detected real bugs and running RoboCog. Probably, this was not a mistake, but we problems to deal with. After several days of intense work, FinE-R 2015 Page 62 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 Gualzru was able to do its job relentlessly until the battery was the robot. This situation is more common than expected due off! The robot spoke with all of us in the Lab and accompanied to the interest the robot produces. Additional issues such as us to the panel. We had the impression that all ours problems different accents, voice volumes, etc. add more difficulties were solved. We were happy for the moment... to the scenario. Despite these limited conversational skills, But we were not going to enjoy the success for a long Gualzru achieved its main objective, to capture the attention of time. After discovering that the learning module would make people. Most of them enjoyed the experiment and also would the robot avoid people that always answered: ’No, I do not recommend the experience to friends or would like to repeat want to go with you to the panel’, our host asked us to move it. Comparing these results with the ones collected in the first Gualzru to the large hall where the conference was about to experiments, revealed that successive updates in the robot have start and to have it welcoming the assistants. It looked like a made it more robust and its conversational abilities, while still good scenario for our use case. We accepted. constrained, have been significantly improved. The moment we moved to the hall new problems appeared. The robot was unable to talk to people because nobody, not H. Last stage: refining the HRI even humans, could hear what the other was saying. The space The conversational abilities represented a severe drawback. was wide open and we could not find a good place for the Despite our efforts, only 50 % of the people that interacted AprilTags landmarks. Light conditions were changing and the with the robot in these real scenarios thought that it was able algorithms in charge of the RGBD camera did not always to maintain a coherent conversation. This was not enough for run correctly. For our younger researchers the experience was a robust, useful robot. But if you cannot solve a problem, really hard, as they passed from the complete success to a perhaps is a good option to totally change the way to solve glaring failure in a short time. Nevertheless, we were now in it. The speech recognition issue is hard to solve in noisy and the final scenario. A spacious environment where the robot crowded environments, where even humans find difficulties must interact with a specific person while other people are in understanding each other. Therefore, our idea was to look speaking and moving around. The failure had an aftertaste for alternative methods to allow people communicate with the of an approaching victory. We still were able to close some robot. Speech recognition was reinforced with the incorpora- use cases in this challenging scenario and a new time for tion of a tactile screen installed on the chest of the robot. The improvements had started. verbalized phrases were now displayed on this screen and it was possible for the person to answer the robot by touching G. December 2014. Large evaluation test at Málaga it. This way, Gualzru retained its conversational abilities but After a new demonstration at Ingenia (Málaga), in an the new interfaces increased its robustness and reliability. environment very similar to the hall at León where we could Following this modification, a new set of questionnaires were capture new questionnaires, we returned to the Lab. The array collected on the same scenario at the University of Málaga. of microphones of the Kinect sensor was intensively tested These questionnaires showed us that the mean values related and, finally, we decided to change it for a shotgun microphone. to questions 2.1 and 2.2 (Table II) improved from 3.57 to 4.27 As it is described with more detail in [12], other minor issues and from 2.7 to 3.72, respectively. Additional changes on the were also solved. whole architecture allowed the robot to successfully close 93 On December, 2014, the current version of Gualzru was % of the started use cases (on December 2014, this rate was 81 tested in a real working scenario. The system was deployed %). Furthermore, the unfinished use cases were always caused in the hall of the Escuela de Ingenierı́as at the University of by the large amount of people in the place that would prevent Málaga. The area where the robot was operating was about 70 Gualzru from reaching the panel. square meters. Fixed obstacles included a column and some tables, but most of the area was free for the robot to move. VIII. C ONCLUSIONS The hall was populated by students and the trials lasted two In this paper we have presented the long process of creation half-days. The robot worked without human intervention and of Gualzru, the salesman robot built for the ADAPTA research engaged with people passing nearby. These people had no a project. Looking at the starting requirements, we can firstly priori knowledge about the robot, nor its functionality. We conclude that the final version of the robot conforms with collected a large set of questionnaires. The results are shown the goals and expectations that we and the companies in on Table II. the consortium initially had. But it is also true that even This data showed that the conversational system remained more rewarding than Ursus has been the whole process of as the weak point of the robot. Some people did not correctly collaboration and the knowledge distilled during these years. understood the robot due to the environmental noise and the It is not that common that basic research is taken close voice of the robot was perceived as not particularly pleasant. to the production line while all the intermediate steps are But the most important issue was related to the understanding registered and analyzed as a means to improve both, the capabilities of Gualzru. Even when using the shotgun micro- forthcoming research and the methods and ways to generate phone these capabilities were strongly limited. The system is reliable technology, given a limited amount of resources. For too sensitive to environmental noise and echos and it gets us, it has been a productive experience, both personally and also confused when there are several people speaking around professionally, and the capacity of the group to approach FinE-R 2015 Page 63 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 new technological challenges has increased notably. We have to be very important to improve the people’s attitude towards learned something useful in every step of the project, from the robot, as well as to reveal the most urgent weaknesses the handcraft manufacturing materials and steps, to the way in preliminary stages. It is a valuable lesson to be kept humans are starting to look at the (social) robots. that periodic tests and surveys are an important part of HRI From the point of view of the technology that has been research, although they are often seen by roboticits as a dull, created and used in the project, we reaffirm the initial idea questionable use of the scarce human resources available. of the need for a common code base that brings together the Another important source of feedback are the robotic contests work of all researchers. We still need some adjustments in the like RoboCup@Home2 or RoCKin3 , that put all teams in the protocols and some refinements in the technology, and even track of a common goal, and where real performances are more conviction by some doubters, but at the end of the day we evaluated in front of expert judges. might well be in the right track. The cost of maintaining and The ADAPTA project officially finished on May 2015 with improving a framework like RoboComp is compensated by the a final public demonstration in Málaga. There Gualzru was flexibility of adapting it to your needs. Making good choices able to interact with many people and successfully closed in this field, where Robotics meets Software Engineering, is several difficult use cases. All partners in the consortium were not easy at all but once the software reaches a certain point satisfied and the robot will be serving from now on at the of maturity, the leverage is undeniable. In the near future, we headquarters of Indra in Madrid. The research groups are now believe that these frameworks will play a crucial role in the involved in more collaborative projects that hopefully will fund evolution of intelligent robots. A role much more important the construction of new social robots. We hope that the next that it is given today. It is needed the confluence of interested generation will be capable of providing a better service to people from Software Engineering to gradually introduce new humans. advances in DSLs, meta-models, model-based design and com- munication middlewares. From the recent evolution of social ACKNOWLEDGMENTS robot software, it looks like to us that the near future will bring This paper has been partially supported by the Spanish larger and finer-grained networks of components, hundreds Ministerio de Economı́a y Competitividad TIN2012-38079 and within the next years, that will demand more efficient software FEDER funds, and by the Innterconecta Programme 2011 communications, self-diagnosis and repair, and sophisticated project ITC-20111030 ADAPTA. monitoring and deployment systems. Maybe classic, coarse- grained architectures will meet fine-grained ones at a point R EFERENCES where interaction dynamics play a relevant role. [1] C. Schlegel, “Communication patterns as key towards component-based The cognitive robotics architecture, RoboCog, is a much robotics,” Advanced Robotic, vol. 3, no. 1, pp. 049–054, 2006. more experimental and uncertain piece of handcraft. We [2] E. Gat, “On three-layer architectures,” in Artificial Intelligence and Mobile Robots, D. Kortenkamp, R. P. Bonnaso, and R. Murphy, Eds. started with a standard three-tier schema and managed to AAAI Press, 1998. integrate symbolic planning with a fair amount of perception [3] R. Brooks, “Intelligence without representation,” Artificial intelligence, and action. To get there we re-introduced the idea of a shared vol. 47, no. September 1987, 1991. [4] C. ELIASMITH, “The third contender: a critical examination of the dy- representation among modules playing the role of an explicit namicist theory of cognition,” in Mind Readings: Introductory Selections context. It was implemented as two graphs, one geometric and on Cognitive Science, P. Thagard, Ed. MIT Press, 1999. one symbolic, and it proved enough for the required adver- [5] C. Guzman, V. Alcazar, D. Prior, E. Onaidia, D. Borrajo, J. Fdez- Olivares, and E. Quintero, “Pelea: a domain-independent architecture for tisement scenarios. Also, the introduction from the beginning planning, execution and learning,” Proceedings of ICAPS’12 Scheduling of symbolic planning and learning technology in the project and Planning Applications woRKshop (SPARK), pp. 38–45, 2012. has proven a huge success. The initial idea of a using a flat [6] L. J. Manso, “Perception as stochastic sampling on dynamic graph spaces,” Ph.D. dissertation, 2013. PDDL description of the domain with a standard planner has [7] R. Marfil, L. V. Calderita, J. P. Bandera, L. J. Manso, and A. Bandera, evolved now into HRI specialized schemes, where hierarchical “Toward social cognition in robotics : Extracting and internalizing mean- planners take care of quotidian, repetitive tasks and flat ones of ing from perception,” Waf2014, no. June, pp. 1–12, 2014. the fine details and contingencies that might occur [19]. But, [8] http://www.w3.org/TR/scxml/ [9] http://www.qt.io/ each solution takes to the next problem and before the end [10] http://www.ros.org/ of the project, we were already working on integrating both [11] Harnad, S. The Symbol Grounding Problem. Physica D 42: 335-346. graphs, reordering the classical hierarchies into more versatile 1990 [12] A. Romero, L. Calderita, J.P. Bandera, J. Martnez, R. Marfil, L.J. Manso, organizations, infiltrate lifelong learning into all crevices of the P. Bustos, A. Bandera. The Cognitive Architecture of a Robotic Salesman, system or use domain specific symbolic planning in classical in review. low-level modules like navigation or object recognition. This [13] http://www.hardkernel.com/main/products/prdt info.php [14] http://www.zeroc.com issue, dealing with the overall organization of robotic intel- [15] M. Joosse and A. Sardar and M. Lohse and V. Evers, BEHAVE-II: ligence, is undoubtedly the hardest one but projects like this The revised set of measures to assess users’ attitudinal and behavioral motivate, and ultimately enforce, the search for new theoretical responses to a social robot, International Journal of Social Robotics 5(3): 379-388. 2013 perspectives. Other crucial part of the global Gualzru experience has been 2 http://www.robocup.org the use of evaluation metrics. User questionnaires turned out 3 http://rockinrobotchallenge.eu/ FinE-R 2015 Page 64 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015 [16] Calderita LV, Manso LJ, Bustos P, Surez-Mejas C, Fernndez F, Bandera A. THERAPIST: Towards an Autonomous Socially Interactive Robot for Motor and Neurorehabilitation Therapies for Children JMIR Rehabil Assist Technol 2014;1(1):e1 [17] Gutirrez, Marco A., A. Romero-Garcs, P. Bustos, and J. Martnez. ”Recent Advances in RoboComp.” Journal of Physical Agents 7.1 (2013): 38-47. [18] McDermott, D. (2000). The 1998 AI planning systems competition. AI Magazine,21 (2). [19] J. C. Pulido, J. C. Gonzlez, A. Gonzlez, J. Garca, F. Fernndez, A. Bandera, P. Bustos, and C. Surez, Goal-directed Generation of Exercise Sets for Upper-Limb Rehabilitation, in Proceedings of the 5th Workshop on Knowledge Engineering for Planning and Scheduling (KEPS), ICAPS conference, 2014, pp. 3845. [20] http://www.openscenegraph.com FinE-R 2015 Page 65 IROS 2015, Hamburg - Germany The path to success: Failures in Real Robots October 2, 2015