Collaborative AI to approach or to accompany people using social robots will be a fundamental robotics field in the near future. If we desire to share and to collaborate with social robots during the development of our daily task, social robots should be able to develop Collaborative AI task, such us accompanying or approaching people. In this article, we will present the robot-people accompaniment and approaching missions through the four levels of abstractions of Collaborative AI systems and describe the main Collaborative AI functionalities that are needed for these missions. We will also show the system that we have developed for accompany one or two pedestrians and approaching on person by a robot in an urban space.
Researchers stated robots will share humans’ environments, will
perform tasks together with humans, and will assist and help humans
in their daily tasks. Robots should behave in a social way and have
to be accepted by people, and for these reasons, robots must
understand the spatio-temporal situation, must understand humans’
behaviors and their intentions, and must take into account the goal that both
pursue in a collaborative task. There are many tasks, where robot and
people share a collaborative task, but in this article we will focus in
the well-known “companion robot”, which is defined as a robot
moving in a human crowd environment while accompanying one or more
pedestrians (e.g. for assisting them [
We will focus in the collaborative task of accompany one or two
persons and approaching a person by a robot in a dynamic
environment which have static obstacles (for example, walls, buildings,
urban furniture, etc.) and dynamic obstacles, for example, moving
pedestrians, bicycles, etc., see Fig.1. When humans walk in these
environments, they follow specific formations, for example two
people groups typically walk in a side-by-side formation; three people
groups usually walk in a V-formation [
Collaborative task while robots walk in groups is not at all a
trivial problem, as it involves different collaborative levels of
abstraction -for example reactive sensori-motor, spatio-temporal situational,
operational (task oriented), cognitive (knowledge oriented)
collaborational levels-; diverse functionalities working on-line (multimodal
perception, multimodal actions, decision making, etc.); and complex
computations at real time. In a typical accompany task the robot has
to infer the final destination and the best path to go through; to take
into account the orientation of the movement of the group; to adapt
their desired velocity to the changes of people’s velocity
(accelerating, decelerating and even stopping when necessary); to maintain
the formation and to be able to change their position in the group if
people change their positions; to always detect their companions or
at least include a behavior to deal with people’s occlusions by other
members of the group; and, finally, to anticipate the behavior of all
pedestrians to avoid collisions in advance. In this work, we explain
the human-robot accompaniment task through Collaborative AI
issues, however we are not explaining the details of the methods,
neither the experiments done due to the lack of space. These methods
and experiments can be found in [
In the remainder of the paper, we will explain briefly the following issues: in Section 3, the four levels of abstraction of Collaborative AI applied to the accompany task; in Section 4, the Social Force Model and other techniques used for the human-robot accompany and approaching of people; in Section 5, the functionalities involved in these missions; and in Section 6, the conclusions. 2
Researchers have developed techniques for robot guiding and
following people. A context-aware following behaviour was developed
by [
Recently, researchers have developed more complex strategies in
their work on social robots [
While previous studies only discussed the challenge of navigating around the person in a fixed side-by-side formation, our algorithm allows a more dynamic positioning around the human partner. This is, the method allows the robot to position itself at front, at lateral and at back of the person who accompanies depending on the situation. Then, if no obstacle interferes with the group’s path, the robot accompanies the person in a side-by-side, but if any obstacle interferes with the group’s path, the robot changes its position around the person to avoid it.
Another innovation that sets our approach apart from others is that our method is able to render a real-time prediction of the dynamic movements of the partner, as well as that of other people, in a horizon time. This kind of prediction performed within a determined time window allows the robot to anticipate people’s navigation and react accordingly.
The Human-Robot approach is an important collaborative task that
takes place between humans and robots in order to generate
interaction; central to this task is the ability to recognise and predict the
intentions of the other party’s movements. In the past, researchers have
used different control strategies for approaching a moving target by
either pursuing it, or trying to intercept it [
Fajen et al. [
In contrast to the previous approaches, our work employs a prediction module based on the social force model, which includes humanlike behaviours for navigating within dynamic environments, and for mapping out the best path for the robot to take towards a moving destination. We are also able to compute the best meeting point between parties by considering the status of the group, the state of the target person, and the target person’s future position. 3
We can describe the collaboration through four levels [
perception sensors and actuators required in a collaborative task to react in the environment.
the spatio-temporal situation assessment in a collaborative task.
the collaboration from the point of view of the task to be developed.
ented to the all collaborative cognitive issues required to the reach the goal.
Anyway, these levels are interconnected and they share information among them.
In order to detail these levels for the accompany and approaching missions, we will first explain the mission of accompany people side-by-side by a robot. The robot has to accompany one or two persons in an urban environment, where there are buildings, walls, urban furniture, etc. that are static obstacles; and people or bicycles moving, that are dynamic objects. The goal is to reach from an origin a destination, without colliding with the static or dynamic obstacles, following some criteria (for example, the minimum time) and taking into account that the robot has to behave human-aware (e.g. the robot navigation and planning has to be socially acceptable and minimizing the people paths disturbances). In the present system (accompaniment mission), we assume that the robot always follows the people, that means that the robot does not know where the people wants to go, and also we assume that the robots knows the actual urban map. It is clear that the behaviour of the system will be different if the person has to follow the robot (guiding mission). For the approaching mission we also assume that the robot knows the actual map.
Let us going to explain these levels for the case of robot-people accompany and approaching missions.
Reactive sensori-motor collaboration: The robot uses the perception systems to detect the person or people that are accompanied (or the person that has to approach) and the static and dynamic obstacles, and also uses this system to localize the robot and the accompanied persons in the urban map. The perception system includes several range-laser (Lidar) and one stereo-vision camera. The Lidar is also used to detect the velocity and acceleration of any moving object in the scene. Moreover, the system uses microphones to listen the voice of the persons. The system uses as actuators, the motors of the mobile platform to navigate, and the speaker to have a dialogue with the people. Although in the present work we do not use the people gaze people tracking, this information is important to know where the are looking to infer where they want to go. Moreover, the system is not detecting, neither identifying sidewalk signals, restaurants, bars, shop, etc.
Spatio-temporal situational collaboration: The robot monitors its poses (position and orientation) and velocity, the poses of all the static obstacles and the poses and velocity of all the moving persons and objects. Using this information the robot always knows the sideby-side position and orientation of the accompanied people, which is used to know how well the robot accompanies the people. Moreover, using the paths followed by the nearby pedestrians and other moving objects, the system is able to predict where they will be after some time and if it is going to be a collision. Then the robot creates several plans, select the best one and sends commands to the Sensori-motor collaboration to adapt the robot poses to the people, to maintain the best accompaniment formation and to avoid collisions. Because our path plans are human-aware, our system always adapts to path people modifications and in this way maintain an implicit agreement with pedestrians to not bother them.
route planning and navigation, providing the best route to the final destination, and the alternatives when the route is closed or there is a too narrow route or the route is too busy. In our system, although this route is always computed, the local path always depends on the accompanied person decision, since the robot follows the person path. The robot can also help in providing information of the upcoming restaurants, shops and other services in this level, but again, this functionality has not been implemented in the present system.
Cognitive (knowledge-oriented) collaboration: The robot can share with the person the goal destination and the alternatives routes to reach the destination in the shortest time. However, in our system this has not been implemented due what we mentioned before, the robot always follows the people path. Moreover, in our system, the robot generates a dialogue with the two persons (in two persons sideby-side accompaniment), in order to maintain them together side-byside while they are navigating. 4
In any robot accompaniment mission, the first thing that the robot
has to do is inferring the final destination of the accompanied
people in order to do it efficiently. Also, if the robot has to approach
to a person, it needs to know the final person’s destination to meet
him/her at some point. Finally, for the rest of people of the
environment, the robot needs to know to which destination they are going
to, in order to make the navigation human-aware and avoid
collisions. To know all people’s destinations, we use the Bayesian
Human Motion Intentionally Predictor (BHMIP) method [
Once the robot knows all the final accompaniment destination and
the rest of the pedestrian and other moving objects destinations, the
robot computes the best path to reach the destination and avoid
collision with the pedestrians and the static objects. Our navigation
system is based on the the Social Force Model [
Once the robot knows all people’s behaviours, the robot has to
plan its collaborative behavior with respect to the people it
accompanies or with respect to the people it will approach. To plan the
accompaniment of the robot with respect one accompanied person,
we use the Adaptive Side-by-side Accompaniment of One Person
(ASSAOP) [
Fr = fr;gdoal(Dnd) + Pi2Pci frg;pocail(Dpci goal) + (Frped + Pi2Pci Fppceid) + (Frobs + Pi2Pci Fpocbis); where fr;gdoal(Dn) is the attractive force until the final desd tination. In the accompaniment case this final destination is inferred using the direction of movement of the accompanied people. Also, this final destination can be a physical static destination inside the environment,Dnd (a door, street, passageway, etc), or other person in the environment in the case of the approaching, Dndg. Pi2Pci frg;pocail(Dpci goal) are the attractive forces to maintain the side-by-side formation with each i companion of the robot. Frped and Frobs are the repulsive forces respect to other people and obstacles. Pi2Pci Fppceid and Pi2Pci Fpocbis are the repulsive forces that the accompanied people feel from all the other people and obstacles applied to the robot, to be able to do a more effective accompaniment. For better explanation of what forces are used for each method, the reader is directed to the cited papers of accompaniment and approaching in the current section.
Once the robot has computed all the paths to accompany the group
or approach to one person, the robot has to select the best one. The
evaluation of these paths is done using a multi-cost function that
considers several sub-cost related to some characteristics of the paths,
Eq. 1. These sub-costs evaluate: the distance between the robot and
the final dynamic destination of the group (Jd); the orientation of
the robot respect to the orientation to arrive to the final destination
(Jor); the attractive force to control the robot (Jr); and the repulsive
interaction forces respect to people (Jp) and obstacles (Jo), and the
accompaniment cost (Jc), respectively. The first five costs were
introduced in [
J(S; sgoal; U ) = [Jd; Jor; Jr; Jp; Jo; Jc] (1)
Finally, the computation of the cost needs three steps. First, the
robot computes each individual cost in each step of the path. Second,
to avoid the scaling effect of the weighted sum method, each cost
function is normalized between ( 1; 1) using the mean and variance
of an erf function, that are calculated after the computation of all the
paths. Third, a projection via weighted sum J : Rn ! R is
obtained giving the weighted cost formula [
In this section, we include the main functionalities that center the research efforts in the Collaborative AI systems to accompany and/or approach people. The functionalities of the Collaborative AI systems to accompany and/or approach people are listed in Fig. 2, as well as the relations among them. 5.1
To interact in dynamic urban environments, robots must detect all
pedestrians and objects of the environment. In our case we use three
types of perception systems: a 360 range-laser range sensor (Lidar);
a video camera system; and a sterovision camera. The video camera
system [
The 360 range-laser range sensor allows to compute person position with high accuracy, high frequency and in large areas. These are important characteristics to do interactions in a real time. However, the Lidar does not allow to identify a specific person, and for this purpose, it is used the video camera.
The Lidar is also used for the adaptation of the robot in the accompaniment and approaching missions. It allows to keep the person in all the accompaniment process and to detect the person in the approaching process. Moreover, the Lidar is also used to detect the pedestrians position and orientation, and predict their paths. 5.2
Communication between the robot and the human is a main
functionality to allow an efficient accompaniment. Communication is
needed to reach common understanding about the environment that
surrounds the group (1-robot and 1 person or 2-people), to agree
on shared final destination, to share the perception of the
accompanied people or other people in the environment, to agree on common
plans to arrive until the destination and synchronize the execution of
these plans or more concretely paths until the destination. The
humans and the robot must communicate and coordinate among
themselves to fulfill a effective and efficient accompaniment or to avoid
collide among each other. During the accompaniment task this
communication can be verbal and non-verbal or low level (implicit or
explicit) [
Regarding the explicit verbal communication, the robot uses it for interacting with the accompanied person, the approached person or with other people using voice (robot speakers). In our accompaniment, this communication was done by speech dialogue between the robot and the human. For example, the robot communicates if it loses the target of the accompanied person. In the case of accompaniment of two persons, the robots makes an interactive dialogue with the persons, using a child game to create engagement between the persons and the robot (in our case we use the child game of discovering the name of an environment object), while walking towards a destination in the environment.
Regarding the implicit non-verbal communication of the accompaniment task, the communication is done through the range-laser, which gives information of the person with respect the robot. In any of the accompaniment missions, the robot knows in real time the position and orientation of the accompanied persons, and also the position, orientation and velocity of the pedestrians. The implicit communication is only in one direction, from person to robot, and when the robot needs to inform the person, from robot to person, it uses the explicit verbal communication. 5.3
To have an effective Collaborative-AI interaction between the robot
and the people, it is mandatory that the robot infers the people
intentionality. Then, in the case of accompaniment or approaching tasks,
the robot needs to predict the walking behaviour of all the people in
the environment. In our case, we use the the Bayesian Human
Motion Intentionality Predictor (BHMIP) [
Specifically, for the accompaniment task, the robot needs to predict the accompanied people behaviour, to anticipate their movements and improve the accompany task. In our case, we use it for maintaining a specific formation and inferring the final destination and the best path to arrive to the people destination. The intentionality is computed using the previous person path and the position of the goal.
For the approaching task, the robot has to predict where will be the position of the person that has to be approached. If the person stops in a specific location, then the prediction is simplified to a known destination. If the person is moving, then the robot using the BHMIP algorithm, computes where the person will be, and then modify its path to reach him or her. 5.4
In the accompaniment task there are mutual adaptation between the human and the robot. The robot is continuously adapting its path in order to fulfill a side-by-side accompaniment, and usually the person is also doing something similar. However, there are cases where for example the person stops without telling anything, in this case the robot modifies its trajectory to stop or to approach the person. When there are obstacles that have to be avoided, then the robot modifies its formation to allow the person to go ahead or behind the robot. If an obstacle implies that the side-by-side formation is broken, then the robot recovers the side-by-side formation after overcoming the obstacle.
In our experiments with inexpert people, we start explaining to the users the minimum information necessary to interact with the robot. This information includes: the destination where they will go together; the required time that needs the robot to start moving; and that the persons have to walk slowly in order that the robot can maintain the side-by-side formation. In addition, we explain that the robot has a safe distance, so they can not walk very close to it. Finally, for the accompaniment of two people, we also explain the child game that we be used.
To fulfill this accompaniment adaptation, the robot needs to detect, track and predict the behaviour of the accompanied people and also of other people or obstacles of the environment, to facilitate the group’s navigation in the dynamic environment.
In the case of the approaching mission, there is a mutual adaptation between the person and the robot. If both are moving, there is an adaptation between the speeds of both to approach and to stop in front of each other.
Furthermore, the accompaniment group must adapt to the dynamic
environment. This means that by detecting and predicting the people
and obstacles in the environment, the robot must avoid them in an
anticipatory way, while accompanying a person or a group of people.
In our case, the robot facilitates the navigation behaviour of the group
that accompanies, while at the same time, facilitates the navigation
behaviour of other people in the environment [
Interaction among robot and humans plays an important role in Collaborative-AI, where in our case, these interactions will be Master-Slave for the robot and Peer-to-peer for inexpert people that interact with the robot.
In our accompaniment case, we have two types of interactions. First, the robot interacts using the position of the person or persons being accompanied. The robot interacts with the accompanied people by approaching or moving away, depending in the type of formation, for example side-by-side or V formation. In the case of two persons being accompanied, the robot will interact in a different way if the robot is in between both persons, or if the robot is the lateral position. Moreover, in case that the robot has to break the formation, due for example to an obstacle, the robot will interact again with the persons to recover the previous side-by-side formation.
Second, the robot and humans can use direct communication among them. The direct communication is done through the robot speaker, for example by telling to the people that the robot can not move because there are too many people blocking its path or for maintaining the group formation using a child game. This game establishes a dialogue of questions-answers, where the accompanied people have to be near the robot and to follow side-by-side formation to maintain the dialogue.
In the case of Collaborative approaching, we use only the interactions regarding the position among the humans of the environment. First, the robot and the approached human can interact using position in two different situations, where only the robot approaches the human or where both approach each other. Second, the robot interact with other people of the environment, by avoiding them. 5.6
There are always agreement between robots and humans who collaborate to do accompaniment or approaching tasks. These agreements are in the shared goals, shared plans of actions and action execution. They can negotiate verbally these shared behaviours or in some cases, they can negotiate implicitly, for example using the distance between them. The negotiation exist and both of them have to agree on what has to be the next action. In most of the cases, the robot has to anticipate what the human will do, in order to facilitate the accompaniment or the approaching.
In the accompaniment tasks there are several agreements between the robot and the human. First, the group must agree on the final destination to go. In our case, the person decides to which destination, of all the possible environment destinations, he/she prefers to go and the robot infers this destination from the person’s navigation behavior. In the case of two people, the robot infers the most likely destination for the group taking into account the behavior of both people, and in the case that they separate, it will take into account the behavior of the closest person. Furthermore, the final destination can be static (an environment destination: door, stairs, passageway, etc) or a dynamic destination, for example other person position in the environment. Then, the group need to agree to which person they want to reach. Second, they must agree on which path they must follow to reach the final destination. In our case, the robot takes into account the behavior of humans by evaluating different costs in the possible computed paths and selects the best of them to reach the destination. Third, in our case they must agree in the adaptive formation when they overpass people or obstacles. Then, to avoid other people in the environment, the robot changes its position around the person to allow the group to avoid easily other people or static obstacles. As the robot is usually slower than the person, for security reasons, it has been decided that the robot goes always behind. And as the robot changes its group’s position in advance to avoid static obstacles and other people, the people in the group can adapt and understand that the robot prefers to go behind of him/her, to overcome obstacles. Fourth, for the accompaniment of groups, the members of the group must decide in which central or lateral position they will be within this group formation, and that position within the group can change dynamically for reasons of comfort and / or the environment.
For a robot approaching to a person, but without accompanying any one, the robot and the approached person may have to agree in: if both will approach at the same time; if it is the robot that approaches the person; in which way the robot has to approach the person; or whether the person really wants that the robot approaches him/her. 5.7
When human and robot collaborate doing a specific task, they need to
share some decision making, reasoning and planning through direct
or indirect communication by using nonverbal cues. In the
companion and approaching cases, we achieve these issues using a social
human-aware navigation [
We have done a number of real-life experiments, for accompaniment
of one person, two persons and approaching a person. In all the
experiments, we have used different groups of people that they did not
know robots before. We have set up the parameters of the models
doing experiments only with people, without robots, and other
experiments with people and a tele-operated robot. With these
parameters, we have complete our models and then tested the models with
people and robots. Fig. 1 shows examples of accompany two people
by a robot. Fig. 3 shows approaching robot a person. We have not
included in this paper the experiments that we did, due to the lack of
space, but they can be found in [
We have described in this article, the basic collaborative AI multi levels required to do accompaniment and approaching of people by robots. We have explained each one of the collaborative AI functionalities to do these two missions and show some illustrative images of the experiments. Finally, we showed that robot accompaniment involves complex Collaborative AI issues.
Work supported by the Spanish Ministry of Science project ROCOTRANSP (PID2019-106702RB-C21-RAEI/FEDER EU) by Ministerio de Ciencia e Innovacio´ n , the EU AI4EU project (H2020-ICT2018-2-825619), by the Spanish State Research Agency through the Mar´ıa de Maeztu Seal of Excellence to IRI (MDM-2016-0656). Ely Repiso is suported by the FPI-grant, BES-2014-067713.