Researchers stated robots will share humans' environments, will perform tasks together with humans, and will assist and help humans in their daily tasks. Robots should behave in a social way and have to be accepted by people, and for these reasons, robots must understand the spatio-temporal situation, must understand humans' behaviors and their intentions, and must take into account the goal that both pursue in a collaborative task. There are many tasks, where robot and people share a collaborative task, but in this article we will focus in the well-known "companion robot", which is defined as a robot moving in a human crowd environment while accompanying one or more pedestrians (e.g. for assisting them [13], guiding them [10,14,17], or following them).
We will focus in the collaborative task of accompany one or two persons and approaching a person by a robot in a dynamic environment which have static obstacles (for example, walls, buildings, urban furniture, etc.) and dynamic obstacles, for example, moving pedestrians, bicycles, etc., see Fig. 1. When humans walk in these environments, they follow specific formations, for example two people groups typically walk in a side-by-side formation; three people groups usually walk in a V-formation [31]; etc.; and in any of these situations, they do the accompany in a social manner. The "companion robot" will have to behave in a similar way as the humans accompany other people, and also they have to navigate human-aware, and adapt to people in the different type of formations. Fig. 1 shows examples of accompany people by a robot.
Collaborative task while robots walk in groups is not at all a trivial problem, as it involves different collaborative levels of abstraction -for example reactive sensori-motor, spatio-temporal situational, Left: Our robot, named Tibi, accompanies one volunteer using the adaptive side-by-side [26]. Center: Tibi accompanies the volunteers at the lateral of the side-by-side [27]. Right: Tibi accompanies the volunteers at the middle of the Side-by-side formation [27].
operational (task oriented), cognitive (knowledge oriented) collaborational levels-; diverse functionalities working on-line (multimodal perception, multimodal actions, decision making, etc.); and complex computations at real time. In a typical accompany task the robot has to infer the final destination and the best path to go through; to take into account the orientation of the movement of the group; to adapt their desired velocity to the changes of people's velocity (accelerating, decelerating and even stopping when necessary); to maintain the formation and to be able to change their position in the group if people change their positions; to always detect their companions or at least include a behavior to deal with people's occlusions by other members of the group; and, finally, to anticipate the behavior of all pedestrians to avoid collisions in advance. In this work, we explain the human-robot accompaniment task through Collaborative AI issues, however we are not explaining the details of the methods, neither the experiments done due to the lack of space. These methods and experiments can be found in [9,26,28].
In the remainder of the paper, we will explain briefly the following issues: in Section 3, the four levels of abstraction of Collaborative AI applied to the accompany task; in Section 4, the Social Force Model and other techniques used for the human-robot accompany and approaching of people; in Section 5, the functionalities involved in these missions; and in Section 6, the conclusions.
Researchers have developed techniques for robot guiding and following people. A context-aware following behaviour was developed by [30]. Hybrid approaches combined following, guiding and accompany behaviours have been developed by [24] and [25]. A new technique of following behaviour that could be perceived by a non-expert as merely following someone, or as a guiding companion, has been developed by [16].
Recently, researchers have developed more complex strategies in their work on social robots [22]. Morales et al. [20] proposed a model of people walking side-by-side which could predict the partner's future position, and subsequently generate a plan for the robot's next position. Furthermore, [21] and more recently [18] did a side-byside method inferring the final goal of the person and also recorded a database of people walking in a fixed side-by-side formation that is different from our database, included in [26], which includes also situations of an adaptive side-by-side companion behaviour.
While previous studies only discussed the challenge of navigating around the person in a fixed side-by-side formation, our algorithm allows a more dynamic positioning around the human partner. This is, the method allows the robot to position itself at front, at lateral and at back of the person who accompanies depending on the situation. Then, if no obstacle interferes with the group's path, the robot accompanies the person in a side-by-side, but if any obstacle interferes with the group's path, the robot changes its position around the person to avoid it.
Another innovation that sets our approach apart from others is that our method is able to render a real-time prediction of the dynamic movements of the partner, as well as that of other people, in a horizon time. This kind of prediction performed within a determined time window allows the robot to anticipate people's navigation and react accordingly.
The Human-Robot approach is an important collaborative task that takes place between humans and robots in order to generate interaction; central to this task is the ability to recognise and predict the intentions of the other party's movements. In the past, researchers have used different control strategies for approaching a moving target by either pursuing it, or trying to intercept it [2].
Fajen et al. [3] presented different control strategies for approaching a moving target. Narayanan et al. [23] used a task-based control law to enable the robot to meet two standing persons and interact with them, by carefully considering their respective positions and orientations, and use that knowledge to calculate an optimal meeting point. Other researchers [1,19] studied human social behaviours in order to yield better results. [1] recorded the different trajectories that people took when approaching other persons. In [11], the authors used proxemics rules to define the approaching distance to the target person for teaching robots proactive behaviour.
In contrast to the previous approaches, our work employs a prediction module based on the social force model, which includes humanlike behaviours for navigating within dynamic environments, and for mapping out the best path for the robot to take towards a moving destination. We are also able to compute the best meeting point between parties by considering the status of the group, the state of the target person, and the target person's future position.
We can describe the collaboration through four levels [29]:
• Reactive sensori-motor collaboration: This level involves all the perception sensors and actuators required in a collaborative task to react in the environment. • Spatio-temporal situational collaboration: This level includes the spatio-temporal situation assessment in a collaborative task. • Operational (task-oriented) collaboration: This level includes the collaboration from the point of view of the task to be developed.
• Cognitive (knowledge-oriented) collaboration: This level is oriented to the all collaborative cognitive issues required to the reach the goal.
Anyway, these levels are interconnected and they share information among them.
In order to detail these levels for the accompany and approaching missions, we will first explain the mission of accompany people side-by-side by a robot. The robot has to accompany one or two persons in an urban environment, where there are buildings, walls, urban furniture, etc. that are static obstacles; and people or bicycles moving, that are dynamic objects. The goal is to reach from an origin a destination, without colliding with the static or dynamic obstacles, following some criteria (for example, the minimum time) and taking into account that the robot has to behave human-aware (e.g. the robot navigation and planning has to be socially acceptable and minimizing the people paths disturbances). In the present system (accompaniment mission), we assume that the robot always follows the people, that means that the robot does not know where the people wants to go, and also we assume that the robots knows the actual urban map. It is clear that the behaviour of the system will be different if the person has to follow the robot (guiding mission). For the approaching mission we also assume that the robot knows the actual map.
Let us going to explain these levels for the case of robot-people accompany and approaching missions.
Reactive sensori-motor collaboration: The robot uses the perception systems to detect the person or people that are accompanied (or the person that has to approach) and the static and dynamic obstacles, and also uses this system to localize the robot and the accompanied persons in the urban map. The perception system includes several range-laser (Lidar) and one stereo-vision camera. The Lidar is also used to detect the velocity and acceleration of any moving object in the scene. Moreover, the system uses microphones to listen the voice of the persons. The system uses as actuators, the motors of the mobile platform to navigate, and the speaker to have a dialogue with the people. Although in the present work we do not use the people gaze people tracking, this information is important to know where the are looking to infer where they want to go. Moreover, the system is not detecting, neither identifying sidewalk signals, restaurants, bars, shop, etc.
Spatio-temporal situational collaboration: The robot monitors its poses (position and orientation) and velocity, the poses of all the static obstacles and the poses and velocity of all the moving persons and objects. Using this information the robot always knows the sideby-side position and orientation of the accompanied people, which is used to know how well the robot accompanies the people. Moreover, using the paths followed by the nearby pedestrians and other moving objects, the system is able to predict where they will be after some time and if it is going to be a collision. Then the robot creates several plans, select the best one and sends commands to the Sensori-motor collaboration to adapt the robot poses to the people, to maintain the best accompaniment formation and to avoid collisions. Because our path plans are human-aware, our system always adapts to path people modifications and in this way maintain an implicit agreement with pedestrians to not bother them.
Operational (task-oriented) collaboration: The robot helps in route planning and navigation, providing the best route to the final destination, and the alternatives when the route is closed or there is a too narrow route or the route is too busy. In our system, although this route is always computed, the local path always depends on the accompanied person decision, since the robot follows the person path.
The robot can also help in providing information of the upcoming restaurants, shops and other services in this level, but again, this functionality has not been implemented in the present system.
Cognitive (knowledge-oriented) collaboration: The robot can share with the person the goal destination and the alternatives routes to reach the destination in the shortest time. However, in our system this has not been implemented due what we mentioned before, the robot always follows the people path. Moreover, in our system, the robot generates a dialogue with the two persons (in two persons sideby-side accompaniment), in order to maintain them together side-byside while they are navigating.
In any robot accompaniment mission, the first thing that the robot has to do is inferring the final destination of the accompanied people in order to do it efficiently. Also, if the robot has to approach to a person, it needs to know the final person's destination to meet him/her at some point. Finally, for the rest of people of the environment, the robot needs to know to which destination they are going to, in order to make the navigation human-aware and avoid collisions. To know all people's destinations, we use the Bayesian Human Motion Intentionally Predictor (BHMIP) method [5]. The BH-MIP uses a set of predefined known destinations of the environment, D = {D1, D2, ..., Dn, ..., Dm}, and a geometric-based long term prediction method that uses a Bayesian classifier to selects the best destination of the person. These predefined destinations are locations where people usually go, like entrances, exits or work places of the environment.
Once the robot knows all the final accompaniment destination and the rest of the pedestrian and other moving objects destinations, the robot computes the best path to reach the destination and avoid collision with the pedestrians and the static objects. Our navigation system is based on the the Social Force Model [15], and we have extended this model (ESFM -Extended Social Force Model) to include repulsion of static objects and of the robot itself [4]. Moreover, we have developed a dynamic path planner, that computes the best path to be followed by the robot, that computes all the paths to go to the final destination, taken into account the pedestrian paths. This model is denominated Anticipative Kinno-dynamic Planner (AKP) [6].
Once the robot knows all people's behaviours, the robot has to plan its collaborative behavior with respect to the people it accompanies or with respect to the people it will approach. To plan the accompaniment of the robot with respect one accompanied person, we use the Adaptive Side-by-side Accompaniment of One Person (ASSAOP) [26]. Also, this method was combined with an anticipate robot approaching behaviour that infers in advance the best encounter point and do an engagement with an accompanied person and one approached person, by using a triangle formation. In addition, to plan the accompaniment of the robot with respect a group of two accompanied people, we use the Adaptive Side-by-Side Accompaniment of Groups of People (ASSAGP) [27], which allows the robot to accompany the group in the central an lateral position of the group. Further, to do a robot's approaching to a person, we use the G 2 -Spline and ESFM to approach people [9]. All these methods have in common that uses the ESFM to plan the tree of paths for the robot to be able to fulfill all the tasks. The next equation includes all the attractive and repulsive forces necessary to carry out all these collaborative navigation's with humans:
is the attractive force until the final destination. In the accompaniment case this final destination is inferred using the direction of movement of the accompanied people. Also, this final destination can be a physical static destination inside the environment,D d n (a door, street, passageway, etc), or other person in the environment in the case of the approaching, D dg n .
i∈Pc i f goal r,pc i (Dp c i goal ) are the attractive forces to maintain the side-by-side formation with each i companion of the robot. F ped r and F obs r are the repulsive forces respect to other people and obstacles. i∈Pc i F ped pc i and i∈Pc i F obs pc i are the repulsive forces that the accompanied people feel from all the other people and obstacles applied to the robot, to be able to do a more effective accompaniment. For better explanation of what forces are used for each method, the reader is directed to the cited papers of accompaniment and approaching in the current section.
Once the robot has computed all the paths to accompany the group or approach to one person, the robot has to select the best one. The evaluation of these paths is done using a multi-cost function that considers several sub-cost related to some characteristics of the paths, Eq. 1. These sub-costs evaluate: the distance between the robot and the final dynamic destination of the group (J d ); the orientation of the robot respect to the orientation to arrive to the final destination (Jor); the attractive force to control the robot (Jr); and the repulsive interaction forces respect to people (Jp) and obstacles (Jo), and the accompaniment cost (Jc), respectively. The first five costs were introduced in [7] and the companion cost was introduced in [26].
Finally, the computation of the cost needs three steps. First, the robot computes each individual cost in each step of the path. Second, to avoid the scaling effect of the weighted sum method, each cost function is normalized between (−1, 1) using the mean and variance of an erf function, that are calculated after the computation of all the paths. Third, a projection via weighted sum J : R n → R is obtained giving the weighted cost formula [6]. Where n is the number of costs.
In this section, we include the main functionalities that center the research efforts in the Collaborative AI systems to accompany and/or approach people. The functionalities of the Collaborative AI systems to accompany and/or approach people are listed in Fig. 2, as well as the relations among them.
To interact in dynamic urban environments, robots must detect all pedestrians and objects of the environment. In our case we use three types of perception systems: a 360 • range-laser range sensor (Lidar); a video camera system; and a sterovision camera. The video camera system [12] is used for identifying specific people that we want to search for or track. The stereovision is used for tracking people in the accompaniment and approaching missions. The 360 • range-laser range sensor allows to compute person position with high accuracy, high frequency and in large areas. These are important characteristics to do interactions in a real time. However, the Lidar does not allow to identify a specific person, and for this purpose, it is used the video camera. The Lidar is also used for the adaptation of the robot in the accompaniment and approaching missions. It allows to keep the person in all the accompaniment process and to detect the person in the approaching process. Moreover, the Lidar is also used to detect the pedestrians position and orientation, and predict their paths.
Communication between the robot and the human is a main functionality to allow an efficient accompaniment. Communication is needed to reach common understanding about the environment that surrounds the group (1-robot and 1 person or 2-people), to agree on shared final destination, to share the perception of the accompanied people or other people in the environment, to agree on common plans to arrive until the destination and synchronize the execution of these plans or more concretely paths until the destination. The humans and the robot must communicate and coordinate among themselves to fulfill a effective and efficient accompaniment or to avoid collide among each other. During the accompaniment task this communication can be verbal and non-verbal or low level (implicit or explicit) [8].
Regarding the explicit verbal communication, the robot uses it for interacting with the accompanied person, the approached person or with other people using voice (robot speakers). In our accompaniment, this communication was done by speech dialogue between the robot and the human. For example, the robot communicates if it loses the target of the accompanied person. In the case of accompaniment of two persons, the robots makes an interactive dialogue with the persons, using a child game to create engagement between the persons and the robot (in our case we use the child game of discovering the name of an environment object), while walking towards a destination in the environment.
Regarding the implicit non-verbal communication of the accompaniment task, the communication is done through the range-laser, which gives information of the person with respect the robot. In any of the accompaniment missions, the robot knows in real time the position and orientation of the accompanied persons, and also the position, orientation and velocity of the pedestrians. The implicit communication is only in one direction, from person to robot, and when the robot needs to inform the person, from robot to person, it uses the explicit verbal communication.
To have an effective Collaborative-AI interaction between the robot and the people, it is mandatory that the robot infers the people intentionality. Then, in the case of accompaniment or approaching tasks, the robot needs to predict the walking behaviour of all the people in the environment. In our case, we use the the Bayesian Human Motion Intentionality Predictor (BHMIP) [5] to predict all the people walking behavior.
Specifically, for the accompaniment task, the robot needs to predict the accompanied people behaviour, to anticipate their movements and improve the accompany task. In our case, we use it for maintaining a specific formation and inferring the final destination and the best path to arrive to the people destination. The intentionality is computed using the previous person path and the position of the goal.
For the approaching task, the robot has to predict where will be the position of the person that has to be approached. If the person stops in a specific location, then the prediction is simplified to a known destination. If the person is moving, then the robot using the BHMIP algorithm, computes where the person will be, and then modify its path to reach him or her.
In the accompaniment task there are mutual adaptation between the human and the robot. The robot is continuously adapting its path in order to fulfill a side-by-side accompaniment, and usually the person is also doing something similar. However, there are cases where for example the person stops without telling anything, in this case the robot modifies its trajectory to stop or to approach the person. When there are obstacles that have to be avoided, then the robot modifies its formation to allow the person to go ahead or behind the robot. If an obstacle implies that the side-by-side formation is broken, then the robot recovers the side-by-side formation after overcoming the obstacle.
In our experiments with inexpert people, we start explaining to the users the minimum information necessary to interact with the robot. This information includes: the destination where they will go together; the required time that needs the robot to start moving; and that the persons have to walk slowly in order that the robot can maintain the side-by-side formation. In addition, we explain that the robot has a safe distance, so they can not walk very close to it. Finally, for the accompaniment of two people, we also explain the child game that we be used.
To fulfill this accompaniment adaptation, the robot needs to detect, track and predict the behaviour of the accompanied people and also of other people or obstacles of the environment, to facilitate the group's navigation in the dynamic environment.
In the case of the approaching mission, there is a mutual adaptation between the person and the robot. If both are moving, there is an adaptation between the speeds of both to approach and to stop in front of each other.
Furthermore, the accompaniment group must adapt to the dynamic environment. This means that by detecting and predicting the people and obstacles in the environment, the robot must avoid them in an anticipatory way, while accompanying a person or a group of people. In our case, the robot facilitates the navigation behaviour of the group that accompanies, while at the same time, facilitates the navigation behaviour of other people in the environment [26].
Interaction among robot and humans plays an important role in Collaborative-AI, where in our case, these interactions will be Master-Slave for the robot and Peer-to-peer for inexpert people that interact with the robot.
In our accompaniment case, we have two types of interactions. First, the robot interacts using the position of the person or persons being accompanied. The robot interacts with the accompanied people by approaching or moving away, depending in the type of formation, for example side-by-side or V formation. In the case of two persons being accompanied, the robot will interact in a different way if the robot is in between both persons, or if the robot is the lateral position. Moreover, in case that the robot has to break the formation, due for example to an obstacle, the robot will interact again with the persons to recover the previous side-by-side formation.
Second, the robot and humans can use direct communication among them. The direct communication is done through the robot speaker, for example by telling to the people that the robot can not move because there are too many people blocking its path or for maintaining the group formation using a child game. This game establishes a dialogue of questions-answers, where the accompanied people have to be near the robot and to follow side-by-side formation to maintain the dialogue.
In the case of Collaborative approaching, we use only the interactions regarding the position among the humans of the environment. First, the robot and the approached human can interact using position in two different situations, where only the robot approaches the human or where both approach each other. Second, the robot interact with other people of the environment, by avoiding them.
There are always agreement between robots and humans who collaborate to do accompaniment or approaching tasks. These agreements are in the shared goals, shared plans of actions and action execution. They can negotiate verbally these shared behaviours or in some cases, they can negotiate implicitly, for example using the distance between them. The negotiation exist and both of them have to agree on what has to be the next action. In most of the cases, the robot has to anticipate what the human will do, in order to facilitate the accompaniment or the approaching.
In the accompaniment tasks there are several agreements between the robot and the human. First, the group must agree on the final destination to go. In our case, the person decides to which destination, of all the possible environment destinations, he/she prefers to go and the robot infers this destination from the person's navigation behavior. In the case of two people, the robot infers the most likely destination for the group taking into account the behavior of both people, and in the case that they separate, it will take into account the behavior of the closest person. Furthermore, the final destination can be static (an environment destination: door, stairs, passageway, etc) or a dynamic destination, for example other person position in the environment. Then, the group need to agree to which person they want to reach. Second, they must agree on which path they must follow to reach the final destination. In our case, the robot takes into account the behavior of humans by evaluating different costs in the possible computed paths and selects the best of them to reach the destination. Third, in our case they must agree in the adaptive formation when they overpass people or obstacles. Then, to avoid other people in the environment, the robot changes its position around the person to allow the group to avoid easily other people or static obstacles. As the robot is usually slower than the person, for security reasons, it has been decided that the robot goes always behind. And as the robot changes its group's position in advance to avoid static obstacles and other people, the people in the group can adapt and understand that the robot prefers to go behind of him/her, to overcome obstacles. Fourth, for the accompaniment of groups, the members of the group must decide in which central or lateral position they will be within this group formation, and that position within the group can change dynamically for reasons of comfort and / or the environment.
For a robot approaching to a person, but without accompanying any one, the robot and the approached person may have to agree in: if both will approach at the same time; if it is the robot that approaches the person; in which way the robot has to approach the person; or whether the person really wants that the robot approaches him/her.
When human and robot collaborate doing a specific task, they need to share some decision making, reasoning and planning through direct or indirect communication by using nonverbal cues. In the companion and approaching cases, we achieve these issues using a social human-aware navigation [6] [7]. In addition, this navigation is accessed using the extended social force model (ESFM) based on the relative position between humans and people. The ESFM includes several interactions between the robot, the accompanied people, the approached people and other people in the environment. Using these interactions and the intentionality prediction of all people, our robot is able to infer a planning behaviour that allows the robot to accompany people or approach to people through a social accepted way.
We have done a number of real-life experiments, for accompaniment of one person, two persons and approaching a person. In all the experiments, we have used different groups of people that they did not know robots before. We have set up the parameters of the models doing experiments only with people, without robots, and other experiments with people and a tele-operated robot. With these parameters, we have complete our models and then tested the models with people and robots. Fig. 1 shows examples of accompany two people by a robot. Fig. 3 shows approaching robot a person. We have not included in this paper the experiments that we did, due to the lack of space, but they can be found in [26] [28] [9].
We have described in this article, the basic collaborative AI multi levels required to do accompaniment and approaching of people by robots. We have explained each one of the collaborative AI functionalities to do these two missions and show some illustrative images of the experiments. Finally, we showed that robot accompaniment involves complex Collaborative AI issues.
Work supported by the Spanish Ministry of Science project ROCO-TRANSP (PID2019-106702RB-C21-RAEI/FEDER EU) by Minis-