3D Tactile Obstacle Awareness System for Drones using a Tactile Interface around the Head

Abstract

We propose a 3D obstacle awareness system for drone pilots, implemented as a tactile user interface around the head. The concept of this system is presented alongside a variety of use cases and recommendations for future work.

-

This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution. Interdisciplinary Workshop on Human-Drone Interaction (iHDI 2020) CHI ’20 Extended Abstracts, 26 April 2020, Honolulu, HI, US © Creative Commons CC-BY 4.0 License.

Author Keywords

Drones; tactile obstacle awareness; drone navigation; wearables.

CCS Concepts

•Human-centered computing ! Haptic devices; Interaction techniques; Ubiquitous and mobile computing systems and tools;

Introduction and Related Work

Drone pilots face obstacle awareness challenges in case of bad lighting conditions, distractions, or when flying in any direction that is not in the camera view. Possible obstacles include static and dynamic obstacles such as other drones, humans, animals, or even brick walls within buildings. We propose a tactile system to indicate obstacles, including their distance from the drone, in the 3D space around the user (see Figure 1).

Earlier work and concepts on human-drone interaction was neatly summarized in [ 4 ] and explained in further detail by Baytas et al. [ 1 ]. Our obstacle awareness concept presented in this paper extends the idea of augmenting spatial awareness for humans [ 2 ] and aims to instead increase spatial awareness of a human controlling a remote drone. Earlier approaches to this challenge were able to show promising results for a 2D navigation task using ultrasound sensors attached to a drone and a vibrotactile belt [ 13 ]. We aim to extend Spiss et al.’s obstacle awareness system to 3D use cases, which cannot be displayed properly by the tactile belt used in [ 13 ].

In our previous work, we presented HapticHead [ 6, 7, 5 ], a vibrotactile display around the head consisting of a bathing cap with a chin strap and a total of 24 vibrotactile actuators (see Fig. 2). We were able to show that our prototype can be used in 3D guidance and localization scenarios for people with normal vision in both virtual (VR) and augmented reality (AR) scenarios. The system can indicate directions all around the user and guide the user to look at a defined point in space with a median deviation of 2.3° to the actual target. This precise guidance capability may also be used to make users aware of obstacles in the space around them. The previous work further included a scenario in which blindfolded users were able to feel the presence of real physical objects in the 3D space around them and subsequently were able to find and touch the objects (see [ 7 ] and Fig. 3).

Input system: Suitable 360 degree obstacle detection for drones

A suitable 360°obstacle detection system for drones is needed as an input for our proposed system. There are a variety of systems and technologies that could serve as an obstacle detection system, such as multiple stereo cameras working together [ 12, 10 ], 3D LIDARs [9], or even a system using, e.g., HyperOmni Visions (HOVIs) [ 11 ]. These input systems would need to filter and extrapolate static and dynamic obstacles, including their distance and 3D viewing angle from the drone camera perspective. The detected obstacles should further be filtered so that obstacles further away than a threshold distance would be excluded from the results, as these can be deemed harmless at the given moment.

Output system: Indicating obstacles around the drone by HapticHead

In our prior work, we introduced a 3D guidance algorithm for arbitrary actuator configurations such as HapticHead [ 7 ]. This guidance algorithm proved to be quite efficient and fast in guiding study participants to look in the indicated direction in 3D, including elevation. The same algorithm can be used in obstacle awareness scenarios as well. Just like in [ 7 ], the depth to obstacles may also be indicated by a vibrotactile pulse-pattern and intensity modulation which gets faster and stronger, the closer an object is.

The spatial mapping of the vibrotactile feedback is drone centric: The output occurs relative to the drone that the user is controlling and is mapped in a one-to-one fashion to the HapticHead. The front of the drone is mapped to the front of the head. Obstacles appearing in front of the drone are haptically displayed on the forehead. Obstacles that appear to the right of the drone appear on the right side of HapticHead, and so on. This yields in natural mapping of the drone coordinate system to the head coordinate system. To the user it feels as if he or she is flying as a pilot inside the drone, intuitively feeling obstacles along its way. When indicating multiple obstacles at the same time with the proposed tactile interface, a user will likely suffer from a loss of localization accuracy. For one, if two obstacles are close together, the user will only be able to perceive one of them, as the vibrotactile pulse-pattern would become confusing if two obstacles overlap from the perspective of the drone and thus allocate the same actuators on the HapticHead. Arguably, this limitation is no deal breaker, as the user can still feel the distance of the closer of the two (or more) objects.

Furthermore, if more than two obstacles are indicated at the same time, even if they are not allocating the same actuators, a loss of accuracy is still likely. This results from sensory congestion/overload or funneling illusion effects if too many actuators are active at the same time [ 3, 8 ]. As a solution to these issues, we suggest to only indicate the closest two or three obstacles at the same time and merge obstacles that are close together, only indicating the closer obstacle.

Use Cases

As indicated in the introduction, the proposed system may be used in a variety of use cases related to drone operation and handling. These include: 1. Flying into any direction that is not in the camera view (e.g., side-, back-, down-, or upwards); 2. operating a drone at night or in bad lighting conditions; 3. operating a drone around areas with many static or dynamic obstacles such as other drones, humans, animals, plants, or walls within buildings; 4. operating a drone while being distracted (e.g., by other humans).

In the first three cases, the system would provide tactile guidance to the closest two or three obstacles, so that the user can intuitively navigate his drone out of a dangerous situation. In the fourth case, the system would provide tactile warnings in case an obstacle is close which reminds the user to redirect his attention back to the drone.

Another use case would be accessibility: visually impaired drone operators should have a much easier time avoiding obstacles due to the additional tactile feedback channel.

Conclusion and future work

In conclusion, we propose a tactile obstacle awareness system for drone operators, which may be used in a large variety of use cases. Future work may implement the proposed system and test the assumed benefits in a real environment. In Haptic Interaction, Shoichi Hasegawa, Masashi Konyo, Ki-Uk Kyung, Takuya Nojima, and Hiroyuki Kajimoto (Eds.). Springer Singapore, Singapore, 393–399.

References

[1] Mehmet Aydin Baytas , Damla Çay, Yuchong Zhang, Mohammad Obaid, Asim Evren Yantaç, and Morten Fjeld . 2019 . The Design of Social Drones . In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19 , Vol. 4 . ACM Press, New York, New York, USA, 1 - 13 . DOI: http://dx.doi.org/10.1145/3290605.3300480

[2] Alvaro Cassinelli , Carson Reynolds, and Masatoshi Ishikawa . 2007 . Augmenting spatial awareness with haptic radar . Proceedings - International Symposium on Wearable Computers , ISWC ( 2007 ), 61 - 64 . DOI: http://dx.doi.org/10.1109/ISWC. 2006 .286344

[3] Michal Karol Dobrzynski , Seifeddine Mejri, Steffen Wischmann , and Dario Floreano . 2012 . Quantifying Information Transfer Through a Head-Attached Vibrotactile Display: Principles for Design and Control . IEEE Transactions on Biomedical Engineering 59 , 7 (jul 2012 ), 2011 - 2018 . DOI: http://dx.doi.org/10.1109/TBME. 2012 .2196433

[4] Markus Funk . 2018 . Human-drone interaction: Let's get ready for flying user interfaces ! Interactions 25 , 3 ( 2018 ), 78 - 81 . DOI: http://dx.doi.org/10.1145/3194317

[5] Oliver Beren Kaul , Kevin Meier, and Michael Rohs . 2017 . Increasing Presence in Virtual Reality with a Vibrotactile Grid Around the Head . Springer International Publishing, Cham, 289 - 298 . DOI: http://dx.doi.org/10.1007/978-3- 319 -68059-0_ 19

[6] Oliver Beren Kaul and Michael Rohs . 2016 . HapticHead: 3D Guidance and Target Acquisition through a Vibrotactile Grid . In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '16 . ACM Press, New York, New York, USA, 2533 - 2539 . DOI: http://dx.doi.org/10.1145/2851581.2892355

[7] Oliver Beren Kaul and Michael Rohs . 2017 . HapticHead: A Spherical Vibrotactile Grid around the Head for 3D Guidance in Virtual and Augmented Reality . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI '17 . ACM Press, New York, New York, USA, 3729 - 3740 . DOI:http://dx.doi.org/10.1145/3025453.3025684

[8] Oliver Beren Kaul , Michael Rohs, Benjamin Simon, Kerem Can Demir, and Kamillo Ferry . 2020 . Vibrotactile Funneling Illusion and Localization Performance on the Head . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '20 (CHI '20) . ACM, New York, NY, USA, 13. DOI: http://dx.doi.org/10.1145/3313831.3376335

[10] Deukhyeon Kim , Jinwook Choi, Hunjae Yoo, Ukil Yang , and Kwanghoon Sohn . 2015 . Rear obstacle detection system with fisheye stereo camera using HCT . Expert Systems with Applications 42 , 17 - 18 ( 2015 ), 6295 - 6305 . DOI: http://dx.doi.org/10.1016/j.eswa. 2015 . 04 .035

[11] Hiroshi Koyasu , Jun Miura, and Yoshiaki Shirai . 2001 . Realtime omnidirectional stereo for obstacle detection and tracking in dynamic environments . IEEE International Conference on Intelligent Robots and Systems 1 ( 2001 ), 31 - 36 . DOI: http://dx.doi.org/10.1109/iros. 2001 .973332

[12] Sergiu Nedevschi , Radu Danescu, Dan Frentiu, Tiberiu Marita, Florin Oniga, Ciprian Pocol, Rolf Schmidt, and Thorsten Graf . 2004 . High accuracy stereo vision system for far distance obstacle detection . IEEE Intelligent Vehicles Symposium , Proceedings ( 2004 ), 292 - 297 . DOI: http://dx.doi.org/10.1109/ivs. 2004 .1336397

[13] Stefan Spiss , Yeongmi Kim, Simon Haller, and Matthias Harders . 2018 . Comparison of Tactile Signals for Collision Avoidance on Unmanned Aerial Vehicles .