Introduction

Social robotics is a rapidly developing field. Thanks to advancements in hardware and software, encountering a robot is becoming an increasingly common event: from hospitals, where they interact with both children and older people, to museums as guides, and even in restaurants to serve customers [1,2]. Once placed in these unsupervised scenarios, the likelihood that a robot may make errors increases, in particular when robots need to interpret the interactions with humans via multiple signals.

Disruptions such as ambient noise, poor lighting, or the higher dynamism of a real-world context often lead to contrasting or uncertain signals while, as a consequence, produce social failures during a human-robot interaction (HRI). Social failures are errors that violate social norms and can degrade the perception of the robot's social and affective abilities [3]: not listening to the interlocutor, interrupting while they are speaking, or changing the subject without reason are just a few examples of social failures. In the field of HRI, it has been necessary to study behavioural techniques to mitigate the problem. One of these techniques is the use of humour by the robot, inspired by human-human interactions. Humour is pervasive in social relationships, being one of the most common ways to produce a positive influence on others: it has been shown that the use of spontaneous humour makes individuals more likeable and attractive in the eyes of others [4,5], making them more friendly and improving the trust conveyed [6]. In situations free from specific tasks [7], such as making icebreaker jokes, telling puns and then apologizing, contrasting serious topics with jokes to ease tension, and being self-deprecating, humour generate laughs and empathy towards the agent. Even in more structured scenarios, such as vaccination in a hospital or reception in a hotel, there have been advantages noted in personalities endowed with humour compared to neutral ones, specifically in terms of engagement, likeability, ease of interaction and empathy [8,9]. Exploring the vast field of humour, we particularly focus on irony and sarcasm. Some HRI researchers [10] have suggested that these can bring benefits to interaction, but it is not easy to understand how to effectively incorporate them into the robot's personality, since not everyone has the same humour or sarcasm. In this work we decided to adopt the Incongruity Theory [11], according to which humour is described as a process related to the experience of inconsistency, focusing on unexpectedness and inappropriateness. The approach proposed in this study is based on handling a delicate management of episodes where multimodal user feedback is deemed unreliable. In these cases, the robot reacts sarcastically by contrasting verbal cue polarity (i.e., the spoken phrase) with non-verbal cues, which include voice pitch and speed, facial expression (i.e., colour of LEDs), and gestures. The goal is to elicit a positive, particularly amused, reaction from the user, avoiding the unpleasant scenario where the interlocutor realizes that the robot did not actually understand, leading to a poorer perception of the robot's social and affective abilities.

The Scenario

The incongruity-based behaviour approach has been integrated into BRILLO (Bartending Robot for Interactive Long Lasting Operations), a three-year national project aimed at creating an autonomous robotic system capable of performing bartender tasks and interacting naturally with customers. The typical BRILLO scenario involves a user and three interaction systems: a kiosk where the user authenticates/registers to order a cocktail, the bartender robot that prepares the drinks, and optionally a waiter robot tasked with serving the customer if they are at a table. Focusing on the bartender robot, which is part of the system equipped with the proposed approach, it consists physically of a head, represented by Furhat, a torso, and two robotic arms for drink preparation. From an interaction standpoint, a key element is personalization and recommendation of both the drinks and the interaction. The robot adapts its behaviour based on the context, classifying the current customer into one of several profiles (e.g., with a worker on a lunch break, the bartender will converse in a way that relaxes them, while with a curious person, it will try to discuss various topics).

The Use Case

In this section, we present the acquisition and processing of user input, the decision algorithm for the behaviour to be adopted, and non-verbal signals configurations, and the results of use case scenario will be presented.

Input Acquisition and Processing

During the dialogue, two types of input are taken from the robot's sensors: voice and face. The first input is immediately processed by the robot's speech-to-text module, which transmits the phrase to the cloud service LUIS1 (Language Understanding Intelligent Service) to perform intent recognition, necessary to understand the user's will, highlighting the involved entities, and sentiment analysis to calculate the sentence emotional polarity (positive, neutral or negative). The second input, in the form of video, is sent to the Affectiva tool2 to recognize, by processing frame by frame on a cloud-based Docker container, the facial expression (positive: happiness, surprise; neutral; negative: sadness, anger, contempt, disgust and fear).

Incoherent Behaviour Decision

Once the inputs are processed, the core of the process begins: the decision on which behaviour the robot should adopt, based on the user feedback polarity. Given the user's facial expression and speech polarities:

• If the user's facial expression polarity is neutral, the robot will interact coherently, using verbal and non-verbal signals with the same polarity as the user's speech.

• If the user's facial expression polarity is not neutral, it is compared with the user's speech polarity:

-If the two user polarities are the same, a coherent behaviour will be chosen.

-If the two user polarities are different, an incoherent behaviour will be chosen: the robot's non-verbal signals will be opposite to its speech, simulating sarcasm.

We observed that the LUIS output was more robust compared to that of Affectiva SDK. Example of Pepper non-verbal cues are shown in Figure 1, while the full detailed non-verbal cues defined by polarity are reported in Table 1.

Experimental Design

To evaluate the proposed approach, an online study was conducted as a within-subject counterbalanced, repeated measures study. The study was organised in three phases. Firstly, we collected demographic information (i.e., age, gender, nationality, and previous experiences with robots), then we asked them to complete the Italian adaptation of the Humor Styles Questionnaire [12], with the aim of finding the participant humour style closest to the among:

• Affiliative: focuses on everyday life events, creating a sense of bonding with the listener.

• Self-enhancing: involves laughing at oneself and one's abilities, often being perceived as humble.

• Aggressive: includes insults and anything aimed at putting someone else down, typical of bullying.

• Self-defeating: involves putting oneself down aggressively, often ending up ridiculing oneself.

Participants were tested either with Coherent or Incoherent robot's behaviours. We asked participants to watch two videos of Pepper welcoming a customer and entertaining them in small talk, such as asking how they were doing. In both videos, the robot adopts the user sentence polarity for its verbal reply; in the first video it is positive, in the second one it is negative. The distinction between the two groups of participants occurs in the non-verbal signals: in the two videos shown, half of the participants witness the robot's coherent behaviour (non-verbal polarity matches verbal polarity), and the other half see an incoherent behaviour (non-verbal polarity opposite to verbal polarity), hence sarcastic. At the end of each video, participants are asked to rate the robot using the Short Form Bem Sex Role Inventory questionnaire [13], to evaluate its character traits, such as kindness, understanding, aggressiveness. We used a 5-point Likert scale (from 1 -totally disagree, to 5 -totally agree).

Results

We collected responses of 63 respondents (50 male, 10 female, no non-binary), average age was 24.3 years. We analysed only 60 questionnaires, with 3 discarded due to incorrect completion. Only 23% had previous experiences with robots, mostly as observers. Regarding humour style: 83.3% fell into the Affiliative category, 10% into Self-enhancing, and 6.7% into Self-defeating. The two behaviours, coherent and incoherent, were compared given a certain polarity of the customer's response, positive or negative. Full means are reported in Figure 2. Regarding the positive incoherent behaviours, we observed higher results in terms of Defence of one's beliefs (2.70 (3.43 vs. 3.03). In both polarities, there was a tendency to perceive the incoherent robot as more self-confident, probably because participants felt more surprised and "threatened" by the sarcastic reaction. For the incoherent configuration group alone, the evaluations between the two polarities were also compared. No significant differences were noted. A T-Test was conducted to check for statistically significant differences between the two behaviours for each item of the Short Form BSRI with relevant variance. We did not find any statistically significant difference.

Conclusion

In this study, we investigated people's perceptions of a robot's sarcastic behaviour, which have been created by contrasting incoherent behaviours. The incoherent behaviours have been presented with verbal and non-verbal cues communicating positive and negative affective expressions. Our results showed that the robot's incoherent behaviour was perceived as more self-confident and assertive compared to the coherent modality, which was rated as more warm and gentle. The differences were probably determined by the contrasts between the two attitudes. However, these results were not supported by statistical significance. Future works will test this incoherent approach in a real-bar interaction, by developing incongruency through facial emotions, and by exploring different humour styles.