1. Introduction
In the past years, numerous migrant children, often
unaccompanied, applied for international protection in EU
countries. EU has the responsibility of supporting these
children and, at the same time, the chance to nurture
their potential to enhance their contribution to our
societies. Even if, in this period of the pandemic, the number
of migrants decreased, it is necessary to design solutions
not only to ensure their access to care and education
but also to foster their social inclusion [
1
]. In this
context, is of particular importance to teach the language of
the country in which they are hosted and also how to
interpret and use culture-related gestures since human
communication is multimodal [
2
] and, according to some
studies and in some cultures, the vocabulary of hand
gestures is very rich especially in Italy [
3
]. Gestures are
used to convey the meaning of a message [
4
]. There are
several types of gestures: metaphoric gestures (i.e., those
that explain a concept), deictic gestures (i.e., pointing
ever, many gestures are culture-dependent and do not
have a unique meaning and symbolism. The same
gesture can mean something quite nasty and disrespectful to
a person from a diferent cultural background. Hand
gestures are a very important part when learning a foreign
NaoKi, that used the Nao robot for teaching Italian and
culture-dependent gestures to migrants [
11
]. Based on
system, we updated the developed solution for running
on the Pepper robot. Pepper, due to its characteristics,
is suitable for the application context for several
reasons, one of these is represented by the possibility to
synchronize what the robot says with the display of
useful information of the tablet (feedback, images, examples)
for making the comprehension of the explanation easier.
As in the previous approach, to recognize gestures in
real-time, we used Kinect, a device able to detect and
the user’s skeleton, that has been integrated with the
Pepper robot. An important component of the
application is represented by the gesture database that has been
movements), and iconic gestures among others. How- the results of a study made with this prototype of the
designed using the formalization proposed by [
3
]. Even respectively. At this stage, a social robot could play an
if, due to the COVID-19 emergency, it was not possible important supporting role of cultural mediator, who have
to perform a large study with migrant children, results the important task of trying to understand the child’s
obtained with the use of Pepper are in line with those expectations and lived experience, to transmit
informaobtained with Nao. However, from the observation of the tion about integration into Italian society, then acting as
interaction and the answers to some of the questions in a bridge between the two cultures. However, one of the
the post-experiment interview, the user experience and major dificulties of the operators consists in attracting
the believability of the interaction with the robot seem and keeping the attention of the children that most of the
to be better evaluated in the case of Pepper than of the times, is very dificult since they are scared and may not
Nao robot. The paper is structured as follows. Section 2 accept physical contact. Most of the young migrants have
of the paper briefly describes the background and moti- their gaze down, indicating their emotional situation and
vations of using a social robot in assistive domains, with their lack of self-awareness, they speak with a low voice
emphasis on issues concerning the reception of migrant and have dificulty in looking at operators. The robot
children. Then, we briefly explain the structure of the could be of great help both for the operators so that they
Italian Gestionary and then, in Section 4, how it has been can establish a relationship with the minors through the
used in the PepperKi application. Then we report the interactions with the robot as a ”game”. In this way,
miresults of a preliminary evaluation study in Section 5. nors could learn the first necessary information to start
Conclusions and future work directions are discussed in the integration process that, initially is based primarily
the last section. on the knowledge of the Italian language, which must
already be learned in the first reception center. A social
robot is a physically embodied, autonomous agent that
2. Background and Motivation communicates and interacts with humans at a social and
emotional level. The robot should be able to interpret
In the last years Italy, along with other European coun- properly human behavior, to react to changes during the
tries, has become the landing place for numerous migrant interaction in a socially plausible manner. The use of
sochildren often unaccompanied (Unaccompanied Foreign cial robots in education has been shown to be successful
Minors - UFM -, isolated children, separated children). in diverse contexts [
8, 12
]. In particular, the use of robots
After the first reception phase in communities for minors may increase attention, engagement, and compliance,
without a family, the second level reception of the Pro- which are critical components of successful learning [
13
].
tection System SIPROIMI – Protection System for Inter- Social robots are employed in education with particular
national Protection Holders and Unaccompanied Foreign success with children with special needs. For instance,
Minors - has the aim of ensuring to children the living social robots are widely applied to teach basic social skills
conditions appropriate to their age, with interventions to children with autism, since they resemble humans but
aimed at social integration and autonomy. Then, the are less complex, seem to be able to manage these
isminor will be assigned, through the Juvenile Court, to a sues successfully [
14, 15, 16
]. As far as the eficacy of
tutor. Due to their age, these children live a double status: using social robots in the teaching of a foreign language
that of a minor and a migrant, experiencing both the dif- and culture is concerned, recent research shows that it
ifculties related to the abrupt and rapid passage to adult- may lead to interesting results [
17
]. A social robot was
hood and the integration into a new and diferent society employed successfully as an assistant to teach English
vo[
6
]. They have profound traumas due to many dificulties cabulary to Iranian students [
10
]. Also, the robot-assisted
encountered along the way to reach Italy (repeated beat- group showed improved retention of the acquired
vocabings, threats, abuses, hunger and thirst). Moreover, they ulary. Other studies suggest that the sociality of the
sufer of the cultural shock and abandonment of parental robot improved the learning outcomes [
18, 19, 20
]. Then,
ifgures of reference, essential in the life of any child. For robots open up new possibilities in teaching that were
these reasons, the minors may show distrust towards previously unavailable, leaving space to explore novel
the operators and any physical contact, even a simple aspects of language learning, as culture-dependent
geshand on the shoulder, can generate a sense of anxiety in tures. Gesture recognition in Human-Robot Interaction
the child. Reception activities, specifically designed by has been proposed in [
21
] to allow people, especially
the government, are implemented to ofer psychological those with physical limitations, to give instructions to
support following the disastrous journey. Among the the robot easily and intuitively. The system uses a Kinect
various activities, whose objectives have been defined for gesture detection, and recognition is performed using
based on the characteristic of the target group, there are a Microsoft software, Visual Gesture Builder. Research,
some (the BLUE and GREY activities) aiming to share similar to the one described in this paper, has been
conkey words about greetings and moods (in Italian) and to ducted on children with autism spectrum disorders [
22
].
know customs and cultural habits of the country of origin, Since these children have delayed gestural development,
a social robot was used to teach them to recognize and
produce eight pantomime gestures that expressed
feelings and needs. This study reports that children in the
intervention group were able to recognize more gestures
and generalize the acquired gestural recognition skills
to human-to-human interaction. Also, in [
23
] the social
robot Nao has been used in conjunction with Kinect for
developing a serious game for sign language tutoring.
All these studies report how a social robot represents a
successful interface for teaching a second language and,
in teaching gestures to children, especially to those with
special needs. Looking at the activities planned for
migrant children, the robot, through games, images, and
sounds, can facilitate the acquisition of the first words
in Italian, useful to be understood in Italy. This could
be integrated with lessons from the robot on the Italian
culture and on that of the diferent countries of origin,
to analyze the two cultures, highlighting diferences and
elements in common. Finally, in this phase, the robot can
be fundamental to facilitate the process of
understanding the gestures typical of Italian culture that favor their
social integration.
Hands open with palms touching
Fingers pointing towards the chin
Neutral space
United hands moving
slowly forward
and backward
always keeping
your fingers
pointing upwards
pointing with the index finger);
2. Iconic, which depicts, in the air, the form or
imitate the movements of an object, an animal, a
person;
3. Batonic, in which the hands move rhythmically
from the top to the bottom to scan and highlight
the accented syllables in a sentence;
4. Symbolic, gestures that, in a given culture, have
a shared meaning, and thus culture-dependent
since they are learned from an early age by
observing them daily, then they are often
incomprehensible to people of diferent cultures. This
latter typology has particular relevance in the
application context of this work.
When we talk about communicative gestures, which cor
respond to a signal-meaning pair, it is necessary, for those
who have diferent cultures, to have a ”dictionary” of
such gestures, to allow a better translation of the
communicative act. To this aim, [
3
] proposed the ”Italian
Gestionary”, a useful resource in which each gesture is
presented through a picture, a description of the
movement, and the corresponding verbal description (Table
1). Gestures can be considered as a piece of semantic
3. The Gesture Dictionary information present in the mind of those who want to
communicate. It is possible to extract the meaning and
Communication is an intentional process of sending, re- use of a gesture by analyzing the Gestionary that, for
ceiving and interpreting messages conveyed through ver- each gesture, provides:
bal or nonverbal signals [
24
]. In verbal communication,
the goal of communicating can be almost intentional,
instead, when implemented with other types of non-verbal
signals such as, for example, gestures, the goal can more
likely, than in verbal case, defined as unconscious or
tacit. The correspondences between signals of our body
(gestures, expressions) and meanings are diferent from
culture to culture even if, in many communication
systems, such as facial expressions and gaze, they are almost
universal [
25
]. We define ‘gesture’ any movement made
with hands, arms and/or shoulders. In particular, the
hands play a significant role in communication and can
be classified in [
4, 26, 27
]:
a. a verbal formulation, the associated verbal
meaning, expressed using a real sentence, for example,
the gesture of applauding can be paraphrased as
”compliments!”;
b. a context, that can contribute to the core meaning
[
24
];
c. a meaning is the core definition across diferent
contexts, similar to that of word dictionary;
d. a grammatical classification , in terms of
”gesturessentences”, called holophrastic, from ”word
gestures”, called articulated, depending on that have
the meaning of a whole sentence or only a part;
e. a pragmatic classification that concerns only the
”phrase-gestures” which are also classified
accord1. Deictic, which indicates an object or a person (i.e.
the skeletal tracking function) of one or more people (up
to 6). The Kinect SDK 2.0 is a library that is essential for
ing to their specific performative just like the ges- recording the video of the gesture to be recognized and,
ture of praise (the applause, ”Congratulations!”); then, converting and analyzing it with the Visual Gesture
f. a semantic classification that provides informa- Builder. Using this technology, the application allows the
tion on the world, like among others the times creation of a database of gestures that can be recognized
(”yesterday”, ”after”) or quantities (”two”) or also and the degree of correctness of the recognized gesture
information about the mind, emotions, and de- with respect to the selected one.
gree of knowledge of the person with whom you The second component, the Gesture Detection
Interare communicating; face, allows connecting the application to the Pepper
g. a rhetorical meaning, that is a rhetorical use of robot that is used, in this case, to interact with the
chilthe gesture diferent from the literal one. dren in the learning task of gestures. To do so we
enriched the robot gesture database with the coordinates
For example, Poggi [
3
] describes the gesture of hitting of the gesture to allow the robot to play it. To do so, we
the chest with the hand, with the palm facing downwards used the functionality of the robot programming
enviand the fingers touching by providing as meaning ”I do ronment (Choregraphe) that allows to memorize a new
not digest it”, but in a rhetorical way because what is movement using the ”Timeline” scripts (Figure 2). Whit
not digested is not a food but a person who metaphor- this functionality just moving the robot’s body parts and
ically ”can’t stand”. In a gesture identification and rep- then clicking on ”save” it is possible to save the gesture
resentation is important to classify not only the verbal and the behavior and then re-run it when selected.
formulation but also their description in terms of shape, The database is running in the background of both
orientation in the space and type of movement since components and contains the description of the gestures.
they synchronously contribute to the final meaning as It has been structured according to the Gestionary
prerepresented in Table 1 [
3
]. sented in Section 3 with, in addition, the coordinates of
the gestures recorded by the Kinect SDK and the
description of the gesture as programmed in Pepper. In
partic4. The Application ular, we stored the movement coordinates of
static/discrete (i.e. hand on the forehead) and continuous/dynamic
gestures (i.e. waiving to say hello). Another important
functionality regards the check of the correctness of the
performed gesture. To allow this, we need to query the
”.gbd” database (DB Robot Gesture in Figure 1) containing
the coordinates of the gestures recognized by the Kinect;
the coordinates are searched based on the gesture and
then the curve of correctness and finally, the evaluation
is done. In the database, together with the information
about gesture execution, we linked its description and a
In order to execute, recognize and explain the meaning
of the gestures using a social robot, we integrated the
Pepper robot with Microsoft Kinect through the PepperKi
application (Figure 1). The proposed architecture has
been described in [
11
] in which we used Nao as social
robot. In this case we used the same architecture but
with Pepper. As a first step, Kinect is used to build the
database of gestures to be successively recognized during
the learning session with Pepper. Kinect for Windows is
a device that allows to recognize and track the body (via
scenario to be used by the robot to better explain the
context in which the gesture is used. Gestures implemented
so far are used to express:
• cognitive states like “I am thinking” (hand and
index finger touching the chin or the forehead), “I
don’t believe it” (hand with the dorso toward the
forehead and the head looking down), “I don’t
understand what you are saying” (pinched fingers);
• emotional states like “I pray you” (hold hands
together moving them slowly forward and
backward and with the head down, see Table 1),“I love
you/kissing”, (hold one or both hands close the
mouth, and move them in the direction of the
interlocutor with the palm raised);
• regulation of the interaction with an
interlocutor like “Greeting/Hello” (waving the hand), “Ask
to speak” (rising the right hand).
(a) “I pray you” – Looking at the gesture
(b) “I pray you” – Repeating the gesture
The application ofers two learning modalities in which
the interaction may be initiated by the robot or the child. Figure 3: (a) Pepper shows the child how to perform “I pray
In the first modality, the robot shows a gesture to the you”, (b) the child repeats the gesture.
child, explains the meaning of the gesture, by taking it
from the database, and asks the child to reproduce it. In
the example in Figure 3, Pepper teaches a gesture to a interaction - if easy and engaging - in general, some
othchild. In this case, the robot is teaching the gesture “I ers were specifically concerning gesture execution and
pray you”. First, the robot shows how to execute the recognition. The statements were the following:
gesture and then asks the child to reproduce it. In the
second modality, the child asks the robot to perform a • Q1: I was able to interact with the robot
specific gesture or a gesture conveying a specific meaning. • Q2: It was easy to understand how to perform
Moreover, the child may ask for an explanation about the gestures
gesture’s meaning. When the child performs the gesture, • Q3: It was easy to understand the feedback
the robot gives him feedback that is generated according • Q4: The system had an adequate response time
to the correctness score calculated by Kinect. In case
of correct execution, Pepper enjoys and shows a green • Q5: It had a low number of misidentified gestures
star on the tablet otherwise it encourages the child to try • Q6: Interacting with the robot was engaging
again, shows again the gesture and asks to repeat it.
As far as task completion is concerned, each subject
was able to complete the tasks and to perform the
re5. Evaluation quired gesture. Figure 4 shows the results of the survey
compared to the previous study with Nao.
We performed the same exploratory experiment with The descriptive means are higher than 4 and thus they
the new version of the game with 10 Italian children indicate that the interaction was quite satisfactory both
with an age from 6 to 10 y.o. Unluckily, due to the pan- from the engagement and gesture learning points of view,
demic, it was not possible to involve migrant children. mainly because they are easy to understand and the
parWe asked each subject to learn and perform the following ticipants were Italian. Even if the study was conducted in
gestures three times: “I pray you”, “Hello”, “Kissing”, “I both cases with a limited number of participants and was
don’t understand”. Before the experiment, children gave not conducted with migrants, the results with Pepper
their verbal informed consent and their parents provided are higher on average compared to those obtained in the
written informed consent for participation in the study. previous evaluation with Nao. To confirm this, we
perAt the end of the interaction, we asked subjects to an- formed the Mann-Whitney Test and the resulting p-value
swer a simple survey about their experience. The survey was 0.00058 which is significant at p < .05, thus showing
was composed of six statements and participants were that the interaction with Pepper was better evaluated
asked to evaluate each of them on a Likert scale from 1 than the one with Nao for the learning gesture task.
to 5. Some statements concerned the evaluation of the