Recent advances in the eld of robotics contributed to the development of several kinds of robots able to express social communication skills using a number of interactive modalities such as facial expressions, gestures, gaze, motion, and color. Despite this progress, the multimodal expression capabilities of robots are still far behind the intuitiveness and naturalness that is required to allow uninformed people to interact in their everyday life with naturally communicative robots. The workshop on "Social Interaction and Multimodal Expression for Socially Intelligent Robots" is held in conjunction with the IEEE International Symposium on Robot and Human Interactive Communication. Its topics cover interdisciplinary research on understanding, designing, and evaluating multimodal communication skills for socially intelligent robots that have the potential to enhance our daily lives in our homes and at work. This years workshop consists of four talks covering topics such as verbal and non-verbal interaction with robots, design of a ective expressions and applications for multimodal interaction with social robots. Moreover, it includes three keynote talks by Ana Paiva, Andrea Bonarini and Jorge Dias and a brainstorming discussion.
In the last few years, a growing interest has been seen in the development of autonomous interactive robots to enhance our daily lives in our homes and at work. Applications are plentiful: companions for children or elderly people, partners in industries, guides in public or personal spaces, educational tutors at school and so on. Robots able to face such broad range of social situations require social-cognitive capabilities that promote uid and e ective interactions. These include the automatic understanding of the users actions, behaviors, and mental and emotional states and the coherent production of multimodal, verbal and non-verbal communication skills.
Designing and developing multimodal communication skills for robots in order to provide more natural and powerful interactive experiences is a signi cant challenge in practice due to limitations both in technology and in our understanding of how di erent modalities must work together to convey human-like levels of social intelligence. The area of social interaction and multimodal expression for socially intelligent robots remains very much an active research area with many challenges and open research questions. For example, to what degree, and how precisely, di erent modalities (e.g., eye gaze, touch, vocal, body, and facial expressions) might be involved in human interaction with intelligent robots remains largely unknown. Likewise, whether combining multimodal elements of emotional expressions result in enhanced recognition of the emotion is still an open question. Furthermore, current research is predominantly focused on visual and auditory senses, neglecting other modalities such as touch that are an integral part of how we experience the real world around us.
The scope of this workshop is to present rigorous scienti c advances on social interaction and multimodal expression for socially intelligent robots. Previous research ndings show that this challenge cannot be solely approached from a pure engineering perspective. Human sciences, social sciences, and cognitive sciences play a primary role in the development and the enhancement of social interaction skills for socially intelligent robots. This workshop fosters interdisciplinary collaboration between researchers on the domain, addressing both the study of human-human interactions as well as human-machine interactions. The analysis of human-human interactions o ers researchers the opportunity to understand how humans interact with the world and with one another multimodally, through both parallel and sequential use of multiple modalities (e.g., eye gaze, touch, vocal, body, and facial expressions), and to develop guidelines on how to design robot behaviors. On the other hand, results achieved by researchers studying human-robot interactions are particularly important to understand how the multimodal communication skills developed for social robots are perceived by uninformed interaction partners (e.g., children, elderly) and how they in uence the interaction process (e.g., regarding usability and acceptance). 2
In order to enable a diverse program, the workshop invites a variety of submission categories in the call for papers. Topics included, but where not limited to: (a) Contributions of fundamental nature
{ Psychophysical studies and empirical research about multimodality (b) Technical contributions on multimodal interaction { Novel strategies of multimodal human-robot interactions { Dialogue management using multimodal output { Work focusing on novel modalities (e.g., touch) (c) Multimodal interaction evaluation { Evaluation and benchmarking of multimodal human-robot interactions { Empirical HRI studies with (partial) functional systems { Methodologies for the recording, annotation, and analysis of multimodal interactions (d) Applications for multimodal interaction with social robots
{ Novel application domains for multimodal interaction (e) Position papers and reviews of the state-of-the-art and ongoing research 3
The workshop consists of three invited talks and four talks based on paper submissions covering topics such as verbal and non-verbal interaction with robots, design of a ective expressions and applications for multimodal interaction with social robots. The talks were followed by a brainstorming discussion. Table 1 presents an outline of the program. 3.1
Invited Speakers Ana Paiva - Instituto Superior Tcnico, Technical University of Lisbon
Short Bio: Ana Paiva is an Associate Professor in the Department of Computer Science and Engineering (Departamento de Engenharia Informtica) of Instituto Superior Tcnico from the Technical University of Lisbon (Universidade Tcnica de Lisboa). She is also the group leader of GAIPS (Grupo de Agentes Inteligentes e Personagens Sintticas), a research group on agents and synthetic characters at INESC-ID. Her main scienti c interests lay in the area of Autonomous Agents, Embodied Conversational Agents and Robots and Multiagent Simulation Systems. Prof. Ana Paiva has been researching in the area of arti cial intelligence for the past twenty years, having also taught at IST during that period.
Talk: Robots that listen to the users heart: The role of emotions in multimodal communication with social robots
Andrea Bonarini - Department of Electronics, Information, and Bioengineering, Politecnico di Milano
Short Bio: Andrea Bonarini is full professor and coordinator of the AI and Robotics Lab at Politecnico di Milano. His current research interests are on machine learning and emotional human-robot interaction, with special focus on robotic games and toys for entertainment and social, psychological and physical development of people with and without special needs. Since 1989, he has realized with his collaborators and students more than 60 autonomous robots. In 2015, he co-funded Nova Labs, to share experience and tools to produce professional robots in a short time at low cost.
Talk: Speechless Social Robotic Toys: Emotional Relations Without Words Abstract: More and more sophisticated toys that can perceive signals from the world and physically act in consequence, i.e. robotic toys, are entering the market and are initiating a new way of playing: they cannot any longer treated as props (as traditional plushes and dolls are), but they are agents that can decide their actions, to interact with. This arrangement opens challenges concerning believability and appropriateness of the robots behavior, which cannot really exploit speech because of both technological and privacy reasons. Multimodal interaction is relevant for this application, and should keep into account many dimensions, from psychology, to technology, and cost.
Jorge Dias - Institute of Systems and Robotics, University of Coimbra
Short Bio: Jorge Dias has been Associated Professor at the University of Coimbra with activities in the Department of Electrical Engineering and Computers and the Institute of Systems and Robotics.His research activities are in the area of Computer Vision and Robotics and has contributions on the eld since 1984. Jorge Dias coordinates the research group for Arti cial Perception for Intelligent Systems and Robotics of Institute of Systems and Robotics from University of Coimbra and the Laboratory of Systems and Automation of the Instituto Pedro Nunes. Since July 2011, Jorge Dias is acting as Faculty from ECE/Robotics at Khalifa University in Abu Dhabi.
Talk: BUM - Bayesian User Model for Distributed Social Robots Abstract: In this work we present a Bayesian User Model for inferring the characteristics and inter-personal relationships of a population users. The model can receive evidence gathered by various interactive devices, such as social robots or wearable devices. The system is modular, with each module being responsible for gathering information and observations from persons present in the systems operation scenario. This information enables each module to determine a single characteristic of the person. New observations and measurements received by the system are fused with previous knowledge by a sub-process based on an information theory technique. This allows the system to be implemented in diverse heterogeneous distributed system topologies, extending beyond robotics. We have conducted experiments involving a simulated team of social robots and user population with four sets of person types. Our experiments have shown that the system is able to learn and classify the persons characteristics, and to nd relevant user groups via clustering. This system can potentially be used to gather information on a large set of persons, as well as to be an information source for user-adaptive applications in areas such as Robotics, AAL and Internet of Things. 4
Christiana Tsiourti is PhD student at the Institute of Service Science of the University of Geneva (Switzerland), and a member of the Swiss Doctoral School in A ective Sciences, at the Swiss Center for A ective Science. In November 2016, she has been awarded a Doc.Mobility Fellowship by the Swiss National Science Foundation, allowing her to spend one year as a visiting researcher at the Vision4Robotics group at the ACIN Institute of Automation and Control at Vienna University of Technology (Austria). Christiana holds a Masters (2011) and Bachelor (2009) in Computer Science from the University of Cyprus (Cyprus). Between 2011 and 2013, she worked as a researcher at the Institute of Systems and Robotics of the University of Coimbra. Christianas research is focused on the development and evaluation of a ective socially intelligent agents (robots and avatars) that autonomously integrate into our daily life environments and possess social skills, such as the automatic understanding of the users actions and emotional states, and the production of coherent emotional feedback.
Jorge Dias has a Habilitation degree and a Ph.D. on Electrical Engineering by the University of Coimbra, with specialization in Control and Instrumentation. Jorge has been Associated Professor at the University of Coimbra (UC)5 with activities in the Department of Electrical Engineering and Computers6 and the Institute of Systems and Robotics (ISR)7.Jorge does research in the area of Computer Vision and Robotics and has contributions in the eld since 1984, with several publications in international journals, books, and conferences. Jorge has been teaching several courses on Computer Vision, Robotics, Automation and Electrical Engineering and Computer Science and supervised several Ph.D. and Master students in the eld of Computer Vision and Robotics. He has been principal investigator in several research international projects. Jorge coordinated the Arti cial Perception Laboratory at the ISR and the Laboratory of Systems and Automation (LAS)8 at the Instituto Pedro Nunes (IPN)9, and was Vice-President of IPN from June 2008 to June 2011. Jorge is currently on a leave of absence from the UC and acting as Faculty in the Electrical and Computer Engineering/Robotics Department at Khalifa University in Abu Dhabi.
Astrid Weiss is a postdoctoral research fellow in HRI at the Vision4Robotics group at the ACIN Institute of Automation and Control at Vienna University of Technology (Austria). She holds a masters degree in sociology and a PhD in social sciences from the University of Salzburg. Her current research focuses on 5 www.uc.pt 6 www.deec.uc.pt 7 www.isr.uc.pt 8 http://las.ipn.pt 9 www.ipn.pt user-centered design and evaluation studies for Human-Robot Interaction, with a special interest in the impact technology has on the everyday life and what makes people accept or reject technology. She co-organized workshops on a variety of HRI-related topics at the following conferences: RO-MAN2008, HRI2009, HRI2011, ICSR2013, and HRI2014. Moreover, she is regularly member of Program and Organizing Committees related to HRI research.
Sten Hanke has a Master in electrical engineering and a PhD in medical science. He is a researcher for the Austrian Institue of Technology (AIT) biomedical engineering in smart biomedical systems. He joined AIT in the year 2005. He is working on European and national projects in the eld of ambient assisted living, social robotics, interaction with ICT systems, smart systems as well as psycho-physiological assessments since more than 10 years. At the moment is he technical manager of the FP7 Miraculous-Life project10 dealing with development of a virtual support partner as well as workpackage leader of the H2020 GrowMeUp project11 developing an ICT service robot for ambient assisted living environments. He is a working member of the international ISO/IEEE 11073, European ISO/TC 215 and CEN/TC 251 health care and medical device communication working group as well as the national ON-AG238.4 group. Currently he is in the Ambient Assisted Living Open Association (AALOA) Governing Board.
Julian Angel-Fernandez is a postdoctoral research at the Visrion4Robotics at the Institute of Automation and Control (ACIN) at University of Technology (Austria). His interests twofold in representation and selection of emotions in robotics, and study of methodologies to increase the interest of students in STEM careers. Currently, he is working as scienti c coordinator in the European project ER4STEM, which aims to maintain childrens curiosity in technology through the use of robotics. He holds a European PhD with merit in Information and Technology (2016), from Politecnico di Milano Italy. He also holds a bachelor in Electronic Engineering (2008); Bachelor in Computer Science (2009); Master in Computer Science (2010) from Universidad de los Andes, Bogota Colombia. He has worked as temporal professor at Poti cia Universidad Javeriana at the department of computer engineering (Colombia). During this period, he was part of the research group Takina and taught: programming courses (I and II), as programming languages and introduction to arti cial intelligence.
Luis Santos holds a PhD from the University of Coimbra (UC), Portugal. He is currently an Invited Researcher at University of Coimbra and is acting as a Project and Technical Manager for the H2020 GrowMeUp Project (GA 643647 12. He held a postdoctoral position at the Institute of Systems and Robotics of the University of Coimbra from October 2014 to March 2015. He acted as a seconded researcher from the UC to a company, Citard Services Ltd., within the scope of the Marie Curie IAPP Project Social Robot (GA 28587013, till 10 www.miraculous-life.eu 11 www.growmeup.eu 12 www.growmeup.eu 13 http://mrl.isr.uc.pt/projects/socialrobot the 30th of September, 2014. He holds a Pre-Bologna M.Sc. degree in Electrical and Computer Engineering, specialization in Robotics and Automation by the Faculty of Science and Technology from the University of Coimbra. He worked as researcher in the area of human-robot interaction and action recognition using Laban Movement Analysis, within the BACS European Project from November 2007 to February 2010. His current research topics mainly focus on machine learning methods for autonomous cognitive systems and human motion analysis towards end-user applications.
In summary, the rst workshop on "Social Interaction and Multimodal Expression for Socially Intelligent Robots" brings together a multidisciplinary audience to discuss open questions, address di cult challenges and elaborate on novel ways to advance research in the eld of mutlimodal interaction, based on theories from human-human interaction and on empirical ndings validated human-robot interaction studies.
Advancing the development of socially intelligent robots opens new social and economic opportunities for the application of robots in our daily lives. We hope that the papers and the outcomes of the brainstorming discussion will contribute towards the establishment of a set of standard guidelines for the production of coherent multimodal, verbal and non-verbal communication skills for robots.
This work is supported by the Horizon 2020 project GrowMeUp (Grant No. 643647) as well as the Horizon 2020 MSCA ITN ACROSSING project (Grant No. 616757).
Copyright c 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. 09:00 - 09:05 09:05 - 09:50 09:50 - 10:30
Organizers' introduction and overview of workshop goals Invited Talk 1: Jorge Dias (University of Coimbra) "BUM - Bayesian User Model for Distributed Social Robots" Bob Schadenberg, Dirk Heylen and Vanessa Evers: A ect bursts to constrain the meaning of the facial expression of the humanoid robot Zeno Raymond Cuijpers: Turn-taking cue delays in human-robot communication 10:30 - 10:50 Co ee break 12:15 - 14:00 Lunch 14:00 - 14:05
Organizers welcome to the afternoon session Invited Talk 2: Andrea Bonarini (Politecnico di Milano) Speechless social robotic toys: emotional relations without words Alexis Block and Katherine Kuchenbecker: Physical and Behavioral Factors Improve Robot Hug Quality Lightning Round I: Summarized challenges of contributed talks Invited Talk 3: Ana Paiva (University of Lisbon) Robots that listen to the users heart: The role of emotions in multi-modal communication with social robots S ren Tranberg Hansen, Anders Krogsager and Jakob Fredslund; A Multimodal Robot Game for Seniors Lightning Round II: Summarized challenges of contributed talks Brainstorming discussion including keynote speakers and presenters Presentation of brainstorming results (What did we learn, what are the next questions to follow) 10:50 - 11:35 11:35 - 11:55 11:55 - 12:15 14:05 - 14:50 14:50 - 15:10 15:10 - 15:30 15:50 - 16:20 16:20 - 16:35