=Paper=
{{Paper
|id=Vol-2503/paper2_3
|storemode=property
|title=Integrating Humans in Decentralized Control Systems with Autonomous Transport Vehicles under the Premise and Use of Proxemic Distances
|pdfUrl=https://ceur-ws.org/Vol-2503/paper2_3.pdf
|volume=Vol-2503
|authors=Thomas Kirks,Jana Jost,Tim Uhlott
|dblpUrl=https://dblp.org/rec/conf/eics/KirksJU19
}}
==Integrating Humans in Decentralized Control Systems with Autonomous Transport Vehicles under the Premise and Use of Proxemic Distances==
https://ceur-ws.org/Vol-2503/paper2_3.pdf
Integrating Humans in Decentralized Control Systems
with Autonomous Transport Vehicles under the Premise
and Use of Proxemic Distances?
Thomas Kirks1, Jana Jost1, and Tim Uhlott1
Fraunhofer IML, Joseph-v.-Fraunhofer Str. 2-4, 44227 Dortmund Germany {thomas.kirks,
jana.jost, tim.uhlott}@iml.fraunhofer.de
http://www.iml.fraunhofer.de
Abstract. In future warehouses and production facilities co-working areas of hu-
mans and autonomous vehicles will play a more important role. There, humans
are confronted with no longer centralized controlled transport systems but de-
centralized ones, where humans can not easily comprehend the autonomous be-
havior of robots. Additionally, safety issues may interfere with process optimiza-
tion. Therefore, new ways of interaction between humans and robots have to be
investigated. This paper describes an experimental setup where humans and au-
tonomous transport vehicles share the same working environment. Using methods
from communication sciences and psychology an interactive way is introduced to
mitigate the problem of process interference. The integration of humans in multi-
agent systems of autonomous transport robots and the increased awareness using
augmented reality for the human are central purpose of the experiment.
Keywords: Human-Robot Interaction · Augmented Reality · Human Awareness
· User Interface.
1 Introduction
For humans and robots sharing the same workspace and collaborating, the functional
safety has to be analyzed [1]. Usually, expensive safety laser rangefinders for mobile
robots are applied to guarantee safe operations. In some cases mobile robots and hu-
mans on the shop floor are separated - hence by default the possibility of collision is
not given. In this case safety methods e.g. solid fences or laser barriers surrounding
the working area of the robots are deployed. An activation of e.g. a single scanner or
a simple breach of the fence will shut down the mobile robots. Here, the downtime of
the system increases the process costs. To mitigate the costs humans and robot have
to work together [9]. In decentralized control systems the same technologies can be
applied but there is still more to research and evaluate to integrate humans in such de-
centralized systems. For the development process, psychologists, computer scientists,
designers and mechanical engineers are involved. For usage, the system is designed in
? This project has received funding from the European Union’s Horizon 2020 research and in-
novation programme under grant agreement No 688117 (SafeLog).
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
2 T. Kirks et al.
way that by the implementation of a virtual representation of the human worker, it can
adjust to communication and properties for diverse needs of multiple individuals. The
first step is to raise awareness of humans and robots between each other. The following
experiment will show one way to overcome the investments for safety.
2 Background
2.1 Augmented Reality
Augmented Reality (AR) is not a new development the basic idea reaches back to the
year 1901. Then Baum described in his short story “The Master Key” a kind of glasses
with which the protagonist could get information about people directly on their fore-
heads [2]. Despite this early vision, it took almost 70 more years until Sutherland de-
veloped the first AR system [14]. For the first time the term “Augmented Reality” has
been defined by Caudell and Mizell during their work at the company Boeing in 1992.
Two years later, Rekimoto developed the first AR prototype for 6 degree of freedom
tracking.
In general, with AR the real world of the human and his natural way of interaction
will be enriched in real-time with computer generated information [6]. This leads to a
fusion of real and virtual world. Although the enhancement can be used for all five hu-
man wits, usually AR developments focus on the visual sense with its high information
transmission density. Besides AR, there exist other concepts for merging the real and
the virtual world together e.g. virtual reality (VR). [11]
To sum it up, through AR the interconnection between human and technology offers
significant potentials in various fields.
2.2 Proxemics
For a better integration of the human while collaborating with automated transport ve-
hicles (ATVs) or other robots, not only safety aspects have to be met, also social aspects
used in human-human-interaction have to be faced to increase the acceptance towards
the ATVs [13]. One important part of the interaction between humans is the theory of
proxemics introduced by Hall [7]. Depending on the kind of people one is interacting
with, they are allowed to approach closer. Each person has four distinct spaces they
maintain around themselves. The intimate space (0.15 m - 0.45 m) is used for partners
whereas the personal space (0.45 m - 1.2 m) is reserved for good friends. In the so-
cial zone (1.2 m - 3.6 m) acquaintances take place. The last distance is the public one
(>3.6 m). A problem occurs if a person enters a zone which he is not allowed to. The
other one might feel uncomfortable and tries to distance himself. [7] Also an ATV has
to stick to these rules depending if it is just passing by the human or has to interact with
him.
Different experiments have examined the effects of robot sizes (see e.g. [3]) and
velocities (see e.g. [12]) besides human factors. Those studies showed that people feel
more anxious and are keeping a greater distance when facing taller or faster robots. In
general, studies of proxemics in human-robot collaboration are conducted to maximize
the user experience and to enhance the productivity of the collaboration.
Title Suppressed Due to Excessive Length 3
Perception is one of the most important tasks in human-robot-interaction. The pre-
requisite is that humans as well as robots are aware of each other. Only than trust and
reliance on one another can be build. To ensure this, one has to find ways to raise the
awareness of the human towards the robot while considering information overflow and
mental workload.
3 Human Awareness in Decentralized Systems
To increase the acceptance of the human worker towards the ATV he has to feel safe
around it. In complex warehouses with a lot of racks, ATVs which drive underneath
those to pick them up, cannot always be seen by the human. Therefore, we use AR to
inform the human about ATVs close by.
3.1 Experimental Setup
The experimental setup is constructed in a way that we can test occlusion of an ATV
in a known area by the use of an obstacle and the surrounding fence itself. The human
wearing AR glasses is observing the inside area from different views and will eventually
see the real ATV in some situations and in others not. To realize the overall system used
in the experiment we implemented a multi-agent system following [8]. For being able
to communicate between the human and the ATV we followed [10]. In that way it is
possible to adapt to the human’s individual zone, resulting from proxemics research,
and change the user experience as described later on.
The area consists of a fence of 3 x 4 metres built of small load carriers in a research
facility of 1000 square metres. Inside one finds an obstacle of the dimensions 0.8 x 0.4
x 0.8 m (L x W x H) and an ATV of the dimension 0.6 x 0.4 x 0.3 m (L x W x H) at two
different locations (see Fig. 1; locations a1 and a2). Outside the fence a human, wearing
a Microsoft HoloLens, is positioned at various locations looking inside the area from
different angles (see Fig. 1; locations b1 to b5).
Fig. 1. Experimental Setup
4 T. Kirks et al.
The ATV is designed in a way that the safety concept allows to omit expensive
safety sensors - meaning the weight and maximum velocity do not cause damage to
humans in case of collision. Due to that fact, the control system of the ATV has to be
implemented in such a manner that collision avoidance has to be integrated for drive
control. The ATV is controlled by an implementation of ROS. All low-level control of
sensors and actuators is handled by dedicated nodes. For navigation it uses the cartog-
rapher node enabling it to navigate in a taught map. This allows autonomous behaviour
so that it can drive autonomously to positions to load or unload goods. There are in-
terfaces implemented to command the ATV to drive to goals. On the one hand web
services accessing the ROS topics are used to gain a relatively platform independent
access to the ATV’s functionality. Further, rosbridge [5] is used to communicate via a
rosbridge client using websockets. For the purpose of this experiment the pose of the
ATV (x, y, φ ), the data of the laser rangefinders (one in the front and one in the back
for navigation, each field of view of about 160 degree) and the status information like
battery voltage or errors status are published on specific topics and available through
the mentioned interfaces.
To communicate with the human agent the proposed design of the multi-agent sys-
tem of [10] is implemented. The agent for the ATV is running on the ATV’s control
board and implements the rosbridge client via a hardware abstraction layer. The com-
munication between ATV agent and human agent on the HoloLens is service based,
where the provided services can be announced and discovered by a zeroconf imple-
mentation [4] to allow ad hoc integration of new users in the network.
The human uses the Microsoft HoloLens to be part of the decentralized control
system and communicate with the ATV. Similar to the agent implementation on the
ATV, the human agent is implemented on the HoloLens enabling the subscription to
topics of the position and the laser rangefinder data. Additionally, the human agent
keeps properties of the user like the proxemics distances which will differ from user
to user. The agent is programmed in a way that it uses the native sensors and tracking
feature of the HoloLens to locate the user in the environment. To locally incorporate
both participants in the setup, a QR code is used to match the ATV’s coordinate system
Fig. 2. A distant view inside the public distance
Title Suppressed Due to Excessive Length 5
Fig. 3. A radar view informs the human
Fig. 4. The line of sight (LoS) between hu-
about ATVs nearby - inside the social dis-
man and ATV is partially blocked through
tance.
an obstacle.
with the one of the HoloLens. The actual pose of the human is also published, but
not used in this experiment. Looking through the HoloLens, the human sees either no
ATV specific information, or a radar indicating there is an ATV close by and raising
awareness or the virtual outline (e.g. the 3D model) of the ATV, if it is in close range
but occluded by an obstacle. The decision on what to show depends on the individual
proxemics distances. Furthermore, the sensor data is visualized around the real ATV for
debug purposes (e.g. the autonomous behavior may not be comprehensive to the user
and might conclude from false data, in that case we can get more detailed visual data
for analysis of this behavior).
3.2 Realization
Depending on the different distances analyzed in proxemics we use different views for
the information. In case an ATV is inside the public distance of the individual (individ-
ual at position b1) in Fig. 1) no further information is given in the view field (see Fig. 2;
live capture of the human’s view through the HoloLens). If an ATV is entering the so-
cial zone of the individual (individual at position b2) in Fig. 1) the position of the ATV
will be displayed on a radar view (see Fig. 3, remark: live capture of HoloLens). The
radar can ensure a safe feeling while not disturbing the human at work. ATVs which are
closer e.g. in the intimate zone (individual at position b3) in Fig. 1) and are not in the
line of sight (see Fig. 5) or are partially blocked by a rack or another obstacle (see Fig.
4) are displayed through an x-ray view on the AR glasses. Through these methods the
awareness of the human can be increased and a safe feeling can be ensured.
Since there are various types of workers e.g picker, service technician or warehouse
manager different information about the ATV is needed. The picker is interested in the
load of the ATV as well as its goal position. On the other hand the service technician
needs to be notified about upcoming errors (e.g. predictive maintenance) as well as the
functionality of the sensors (see sensor signals in Fig. 6). For an overview of common
information of the different roles refer to Table 1.
This information will be requested by the human agent by subscribing to the needed
services of the ATVs. Further, the human agent saves the worker’s properties. Depend-
ing on the experiences with ATVs the proxemic zones are varying. This information is
used to adjust the different types of views. Robotic experienced workers who are not
6 T. Kirks et al.
Fig. 5. The LoS between human and ATV is
Fig. 6. The LoS between human and ATV is
blocked through an obstacle.
given.
Table 1. Information requested by different roles
picker service technician warehouse manager
maximum load errors uptime
velocity sensor data efficiency
goal position error rate error rate
idle time uptime costs per job
afraid of ATV do not need to be notified so early since the ATV can move closer. Dif-
ferent distances used ensure that the information flood is kept small and one can focus
on his own task.
4 Conclusion
The implementation and investigation in this experimental setup demonstrate an intu-
itive method to raise awareness for the human worker in robotic environments. It is a
flexible way to include the human in a multi-agent system and thus making him able to
virtually see robots or potential dangers close by. In this case AR is a feasible technol-
ogy to present context-based information at the right location for the human to compre-
hend the ATVs’ behavior. Methods from the field of proxemics are used to respond to
the users’ needs and each specific human gains the intended features and behaviors in
the interaction human-robot systems.
5 Outlook
The next steps in our evaluation is the analysis of reaction of the humans. The x-ray
view and the displayed information of the automated transport vehicles depending on
the proxemics are used to raise the awareness of the human and therefore increase the
acceptance of the human towards the ATVs. The different distances can vary depending
on the individual humans in example because of his experience with the systems or with
robots and complex technologies in general. Hence, the proxemics will be examined in
more detail. The user might still move away from the ATV because he has seen the
information of the ATV too late and is surprised by it. Furthermore, the awareness of
Title Suppressed Due to Excessive Length 7
each individual human might vary throughout the day or working week depending on
personal situations e.g. tired, stressed, hungry, etc. To raise the awareness depending
on the humans conditions we will examine the usage of eye-tracking as a feedback for
the distraction of the user. Eye-tracking has already been used in other user applica-
tions to measure stress and awareness so that we believe it will influence the system
positively when used during online computation. Also, we would like to evaluate user
experience with respect to the way robots should be highlighted for raising the atten-
tion in an adequate manner. Finally, we intent to extend the setup on a larger scale in
the whole research facility, where multiple ATVs roam around humans, to investigate
interdependencies between the actors.
References
1. Albu-Schaeffer, A., Bicchi, A., Chatila, R., Luca De, A., Giralt, G., Hirzinger, G., Mattone,
R., Oriolo, G., Schiavi, R., Siciliano, B., Tonietti, G., Vendittelli, M., Luigi, V.: Physical
human-robot interaction in anthropic domains: Safety and dependability (01 2005)
2. Baum, L.F.: The Master Key: An Electrical Fairy Tale Founded Upon the Mysteries of Elec-
tricity and the Optimisms of Its Devotees. Bowen-Merrill Company (1901)
3. Butler, J.T., Agah, A.: Psychological effects of behavior patterns of a mobile personal robot.
Auton. Robots 10(2), 185–202 (Mar 2001). https://doi.org/10.1023/A:1008986004181
4. Datatracker, I.: Zero configuration networking (zeroconf) (2019), https://datatracker.
ietf.org/wg/zeroconf/charter/
5. Foundation, O.S.R.: rosbridge_suite (2017), http://wiki.ros.org/rosbridge_suite
6. Günther, W.A., Blomeyer, N., Reif, R., Schedlbaur, M.: Pick-by-vision: Augmented reality
unterstützte kommissionierung. Tech. rep., Universität München, Lehrstuhl für Fördertech-
nik Materialfluss Logistik (2009)
7. Hall, E.T.: The hidden dimension. Doubleday Anchor Books, Doubleday (1966)
8. Jost, J., Kirks, T., Mättig, B.: Multi-agent systems for decentralized control and adaptive
interaction between humans and machines for industrial environments. In: 7th IEEE Interna-
tional Conference on System Engineering and Technology (ICSET). pp. 95–100. IEEE (Oct
2017). https://doi.org/10.1109/ICSEngT.2017.8123427
9. Khalid, A., Kirisci, P., Ghrairi, Z., Pannek, J., Thoben, K.D.: Safety requirements in collab-
orative human robot cyber physical systems (02 2016)
10. Kirks, T., Jost, J., Uhlott, T., Jakobs, M.: Towards complex adaptive control systems in intral-
ogistics. In: The 21st IEEE International Conference on Intelligent Transportation Systems.
IEEE (2018). https://doi.org/10.1109/itsc.2018.8569949
11. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Infor-
mation Systems vol. E77-D, no. 12, 1321–1329 (12 1994)
12. Otsuka, R.I.H., Inooka, H.: Study on emotional evaluation of robot motions based
on galvanic skin reflex. The Japanese journal of ergonomics 31(5), 355–358 (1995).
https://doi.org/10.5100/jje.31.355
13. Reeves, B., Nass, C.: The Media Equation: How People Treat Computers, Television, and
New Media Like Real People and Places. Cambridge University Press (1996)
14. Sutherland, I.E.: A head-mounted three dimensional display. In: AFIPS ’68 (Fall, part I)
Proceedings of December 9-11, 1968, fall joint computer conference. vol. 1, pp. 757–764
(1968)